A Diachronic Perspective on User Trust in AI under Uncertainty
Authors: Shehzaad Zuzar Dhuliawala
Vilém Zouhar
Mennatallah El-Assady
Mrinmaya Sachan
Venue: EMNLP 2023 Main (2023)
Keywords: trust
calibration
confidence
collaboration
TL;DR: We study how user trust in AI systems changes in the context of miscalibrations.

Abstract: In human-AI collaboration, users typically form a mental model of the AI system, which captures the user's beliefs about when the system performs well and when it does not.
The construction of this mental model is guided by both the system's veracity as well as the system output presented to the user e.g., the system's confidence and an explanation for the prediction.
However, modern NLP systems are seldom calibrated and are often confidently incorrect about their predictions, which violates users’ mental model and erodes their trust.
In this work, we design a study where users bet on the correctness of an NLP system, and use it to study the evolution of user trust as a response to these trust-eroding events and how the user trust is rebuilt as a function of time after these events.
We find that even a few highly inaccurate confidence estimation instances are enough to damage users' trust in the system and performance, which does not easily recover over time.
We further find that users are more forgiving to the NLP system if it is unconfidently correct rather than confidently incorrect, even though, from a game-theoretic perspective, their payoff is equivalent.
Finally, we find that each user can entertain multiple mental models of the system based on the type of the question.
These results highlight the importance of confidence calibration in developing user-centered NLP applications to avoid damaging user trust and compromising the collaboration performance.

A Diachronic Perspective on User Trust in Al under Uncertainty
Shehzaad Dhuliawala'
Vilém Zouhar"
Mennatallah El-Assady
Mrinmaya Sachan
Department of Computer Science, ETH Zürich
(sdhuliawaly vzouhar,
menna. aaamaaeelasaddyai..etzz.
Abstract
Decision seccet/rrejest mental model
Update
Input
In a human-AL collaboration, users build men- model of the system based on its relia- bility and how it presents its decision, e.g. its presentation of system confidence and an ex- planation of output. Modern NLP systems often uncalibrated, resulting in confidently incorrect predictions that undermine user trust. order to build trustworthy AI, we must un- derstand how user trust is developed and how it can be regained after potential trust-croding events. We study the evolution of user trust in response to these trust-eroding events using a betting game. We find that even a few in- correct instances with inaccurate confidence estimates damage user trust and performance, with very slow recovery. We also show that this degradation in trust reduces the success of human-Al collaboration and that different types of
AI prediction message
mr
uP
User mental model of AI
Al correctness revealed
Figure 1: Diachronic view typical human-Al col- laborative setting. At each timestep the user uses their prior mental model the to accept or reject the Al system's answer yı, supported by an additional message (AI's confidence), and updates their mental model of the Al system to M If the message rejected, the user invokes fallback process to get different answer.
An effective approach for facilitating decision- making in collaborative settings is for the Al sys- tem to offer its confidence alongside its predictions. This is shown in Figure 1, where the AI system pro- vides an additional message that enables the user to either accept reject the system's 's answer based on the additional message, such as the confidence score. This makes strong case for the Al's confi- dence being calibrated (Guo et al., 2017) - when the confidence score aligns with the probability the prediction being correct.
correct and confidently incorrect- have different negative effects on user trust. Our findings highlight importance of calibration in user-facing AT ap- plications and shed light on what aspects help users decide whether trust system.
1 Introduction
Al systems are increasingly being touted for use in high-stakes decision decision-making, For example, doctor might use an Al system for cancer detection from lymph node images (Bejnordi et al., 2017), teacher may be assisted by Al system when teaching students (Cardona al., 2023), individ- uals may rely on AI systems fulfill their infor- mation requirements (Mitra et al., 2018). Al sys- tems are integrated across diverse domains, with an expanding presence in user-centric applications. Despite their growing performance, today's 's Al sys- tems are still sometimes inaccurate, reinforcing the need for human involvement and oversight.
When user interacts with an AI system, they develop mental model (Hartson and Pyla, 2012) of how the system' 's confidence relates to the in- tegrity of its prediction. The issue of trust has been extensively studied in psychology and cognitive sci- ence with Mayo (2015); Stanton et al. (2021) find- ing that incongruence (mismatch between mental model and user experience) creates distrust. Given the ever-increasing reliance on Al systems, is crucial that users possess well-defined mental model that guides their trust in these systems. Nev- ertheless, our current understanding regarding the cvolution user trust over time, its vulnerability to trust-depleting incidents, and the methods to re-
Shared first authorship
Data & code: sutthhh..om/.aaaaarrintttteeeeee

---

store trust following such events remain unclear. Addressing these inquiries holds great significance in the advancement of reliable Al systems.
tasks or multi-domain question answering (Law et al., 2021; Vikander, 2023; Chiesurin et al., 2023) and transparency and controllability are one of the key components that increase users' trust Bansal et al. (2019); Guo et al. (2022).
In this paper, our objective is to investigate user interactions with an Al system, with a specific fo- cus on how the system's confidence impacts these interactions. Through series of carefully designed user studies, we explore the implications of mis- calibrated confidences user's perception of the system and how this, in turn, influences their trust in the system. Our experiments shed light on how users respond to various types of miscalibrations. We find that users are especially sensitive to con- fidently incorrect miscalibration (Section 4.1) and that the trust does not recover even after long sequence of calibrated examples. Subsequently, we delve into an analysis of how trust degradation corresponds the extent of miscalibration in the examples provided (Section 4.2). Then, we assess whether diminished trust in an Al system for a specific task can extend to affect a user's trust in other tasks (Section 4.3). We also explore different methodologies for modeling a user's 's trust in an AI system (Section 5). Our results show how reduced trust can lower the performance of the human-Al team thus highlighting the importance of holistic and user-centric calibration of Al systems when they are deployed in high-stakes settings.
Trust and Confidence Calibration. A common method AI systems use to convey their uncertainty to the user is by its confidence (Benz and Ro- driguez, 2023; Liu et al., 2023). For the system's confidence to reflect the probability of the system being correct. the confidence needs to be calibrated. which is long-standing task (Guo et al., 2017; Dhuliawala al., 2022). This can be any met- ric, such as quality estimation (Specia et al., 2010; Zouhar et al., 2021) that makes easier for the user to decide on the Al system's correctness. Re- lated to calibration is selective prediction where the model can abstain from predicting. The latter has been studied in the context of machine learning (Chow, 1957; E1- Yaniv et al., 2010) and its various applications (Rodriguez et al.. 2019; Kamath al.. 2020; Zouhar al., 2023).
Trust calibration is the relation between the user's trust in the system and the system's 's abili- ties (Lee and Moray, 1994; Turner et al., 2022; Zhang et al., 2020; Yin al., 2019; Rechkemmer and Yin, 2022: Gonzalez et al., 2020; Vodrahalli et al., 2022). Specifically, Vodrahalli et al. (2022) explore jointly optimization of calibration (transfor- mation of the Al system reported confidence) with human feedback. They conclude that uncalibrated models improve human-Al collaboration. How- ever, apart from their experimental design being different from ours, they also admit to not studying the temporal effect of miscalibrations. Because of this, our results are not in contradiction.
2 Related Work
Human-Al Collaboration. Optimizing for co- operation with humans is more productive than fo- cusing solely on model performance (Bansal et al., 2021a). Human-Al AI collaboration research has fo- cused on Al systems explaining their predictions (Ribeiro et al., 2016) or examining the relationship between trust and Al system's accuracy (Rechkem- mer and Yi,, 2022; Ma et al., 2023). Related to our work, Papenmeier al. (2019); Bansal al. (2021b); Wang and Yin (2022); Papenmeier al. (2022) examined the influence of explanations and found that inaccurate ones act as deceptive experi- ences which erode trust.
Modeling User Trust. Ajenaghughrure et al. (2019); Zhou al. (2019) predictively model the user trust in the AI system. While successful, they use psychological signals, such as EEG or GSR. for their predictions, which is usually inaccessi- ble in the traditional desktop interface setting. Li et al. (2023) use combination of demographic infor- mation together with interaction history to predict whether the user is going to accept or reject AI sys- tem's 's suggestion. The field has otherwise focused on theoretical frameworks explain factors that affect trust in mostly human-robot interaction sce- narios (Nordheim et al., 2019; Khavas et al., 2020; Ajenaghughrure et al., 2021; Gebru et al., 2022).
Nourani et al. (2021); Mozannar et al. (2022) study the development of mental models which cre- ate further collaboration expectations. This mental model, or the associated expectations, can be vio- lated, which results in degraded trust in the system and hindered collaboration (Grimes et al., 2021). The field of NLP offers several applications where trust plays a vital role, such as chatbots for various

---

3 Human AI Interaction over Time
interactions between a user and a simulated Al crestion-answeringg (QA) system. We recruit par- ticipants who are told that they will evaluate a QA system's performance on a sequence of question- answer pairs. The participants are shown the Al's produced confidence in its answer and then are in- structed to use this confidence to assess its veracity, We term instance of the Al's question, predic- tion, and confidence method of using user interactions with a system to study user trust similar to the study performed by Gonzalez et al. (2020). After the participant de- cides if the system is correct or incorrect, they bet from Oe to 10c on their decision about the system's 's correctness. We then reveal if the Al was correct or incorrect and show the user the gains or losses. The monetary risk is chosen intentionally in order for the participants to think deeply about the task. An alternative, used by Vodrahalli et al. (2022), is to simply ask for participants' confidence in the answer. While straightforward, we consider this to be inadequate in the crowdfunding setting. This decision is further supported by the fact that there is difference between what participants report and what they do (Papenmeier et al., 2019). The average duration of the experiment was 6.7 minutes (Figure 9) and collected 18k stimuli interactions (Table 3). See Figure 3 for an overview of the ex- periment design and Figure 13 for the annotation interface.
We begin by providing a preliminary formalism for a human-Al interaction over time. comprises of two interlocutors, an AI system and a user. At time I, the user provides the Al system with an imput or a question qe and the AI system responds with an answer y/ along with message comprising of its confidence in the answer my. The user has two options, either they accept the Al's answer or reject it and try find an answer themselves. The Al is either correct (a, 1) or incorrect (a; - 0). The combination of correctness at and confidence my results in four different possibilities each with a different reward, or risk, shown in Figure 2. For example, confidently incorrect may lead to the user disastrously accepting false answer while uncon- fidently correct will make the user spend more time finding the answer themselves.
stimulus to the user. This
Contident Incorrect2
Confident Correct of
Unconfident
Incorrect3
Unconfident
Correct
Figure 2: Possible correctness and confidence combi- nations an AI system. Confidently incorrect and unconfidently correct are miscalibrated while the rest calibrated (i.c. confidence corresponds to correctness.
During the interaction, the user learns a men- tal model (Vr,) of the AI system that they can use to reject and accept the Al's prediction. This mental model encapsulates something commonly referred to as user trust, which is, however, ab- stract and can not be measured directly. Instead, proxy that describes
3.2 Simulating AI
To investigate users' interactions, we simulate an AI system that outputs predictions and confidences. The prediction and confidence are produced using pre-defined generative process.
in our study, we rely manifestation of this trust. We ask the user to make an estimate of their trust by tying it to a monetary reward. We assume that both depend on the given question gt, message my and his- tory. The users place a bet between O4 and 104, i.e. up = U (4e, mr, Vx)) E [0e, 10c). We for- mally define the user's decision to accept or reject the Al's answer as up U2 m,, V 0]. given question qt, message m, and history. In this work, by the user's mental model, we refer to in the context of the features the user might use to decide how much they are willing to bet on the AI's prediction and how likely they are to agree with the Al and how V changes over time.
Our simulated Al encompasses four modes for the generation of AI 'correctness' and confidence values. For miscalibrated questions, we have two modes: confidently incorrect (CI) and unconfi- dently correct (UC) modes, while for calibrated questions we use the accurate mode (control) to generate questions.
We define conditional variable c; which de- notes the aforementioned conditions. Then, based on the condition Ct. we have the following data generation process at timestep In our data gen- eration process, we first decide the AI correct- ness aj E [0, 1] and then decide the confidence my E [0%, 100%) as below:
3.1 Study Sctup
To study how user trust changes temporally we design a set of experiments with a sequence of
'Online demo: wwt.gihthhiiiiotttttitteeniiiiii

---

We posit that miscalibrated stimuli decrease user trust and subsequently verify the hypotheses:
Bernoull((0.7)
if ct = calibrated
at
Bernoull((0.0)
if c
Bernouli((10)
c UC
HI: Confidently incorrect examples lower partici-
pants" trust in the system
(0.i0rmm0045,, 0.85) if ct cal. A ar if = cal. A a; (0.form(007,110 if Q: = CI A 04
H2: Unconfidently correct examples lower partic- ipants' trust in the system, but less so
Uniform ().2, 0.55)
mi ~
H3: Miscalibrated examples reduce the human-
AI collaboratioon performance
Uniform(0.1,0.4)
if e = UC A d
To control for participants prior knowledge of ran-
1.0
the answers to the provided questions, domly generated questions with fictional premises. We also experimented with questions soureed from a combination of Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). Un- fortunately, this approach resulted in a lot of noise and instances of misconduct as participants would look up the answers to increase their monetary re- ward. See Appendix A for description of stimuli generation. We note that the set of questions that the participants see have similar ECE (Expected Calibration Error) scores and we compare this to a real NLP model in Appendix B.
Trust (bet value)
Correctuess
Bc
0.8
0.6
6c
0.4
4c
0.2
accurate
1.0
Trust (bet valua)
Correctiees
8c
0.8
0.6
6c
0.4
4c
0.2
conf. accurate
accurate
Figure 4: Average user bet values (y-axis) and bet cor- reciness (point & histogram color) with no intervention (control, top) and confidently incorrect intervention (bot- tom). The spline shows a 3dd degree polynomial fitted with MSE. Transparent features are overlayed from the other graph. See Figure 14 for annotated version.
Question: Which
painting was discovered hidden vault 2020?
The system correct
84
You predicted the system is correct and was correct (+S+)
Al answer: "The Forgotten Masterpiece"
The system is incorrect
on my decision
by Michael Chen Confidence:
We assign each user to a particular condition. For the control group, we show 60 calibrated stim- uli. For confidently incorrect and unconfidently correct groups, we show 10 calibrated, then 5 mis- calibrated (according to the particular mode), and then 45 calibrated stimuli. We then observe, in par- ticular, the user bet value and accuracy (Figure 4).
Figure 3: Pipeline for single stimulus out of 60. The maximum payout for bet 10g. UI Elements show possible user actions. See Figure 13 for screenshots.
4 Experiments
We perform three types of experiments. Sec- tion 4.1, we establish the different effects of con- fidently incorrect and unconfidently correct stim- uli. Then, in Section 4.2 we see how the size of confidently incorrect intervention affects the users interaction with the Al system and in Section 4.3 explore if miscalibration is transferable between question types. Lastly, we predict the user interac- tion in Section
Confidently incorrect intervention. The control group. which was shown only calibrated stimuli, quickly learns to bet higher than the beginning and becomes progressively better at it. The confi- dently incorrect intervention group has the same start but then is faced with the intervention, where they het incorrectly because of the inaccurate confi- dence estimation. Even after the intervention, their bet values remain significantly lower and they are worse at judging when the AI is correct. The dif- ference in bet values before and after intervention across confidence levels is also observable in Fig- ure 11. We use the user bet value as proxy for trust ( (T 7e, / reciness of the bet (u (u 89%, a = 78%). The significances are p<10 and p-0.03, respec-
4.1 Effect of Miscalibration
We categorize Al behavior into four categories (Fig-
ure 2) and design an experiment to answer:
5e) and the user cor-
RQ1: Do miscalibrated examples affect user trust and alter how they interact with the Al system?

---

tively, with two-sided I-test,
groups (p<10-3 with two-sided t-test), which pro- vides evidence for H2. Prior work in understand- ing psychology has found similar results where humans tend to be more sympathetic to underconfi- dent subjects (Thoma, 2016). While applying find- ings from human human interaction to human-Al interactions, exercise caution and acknowledge the need for further research.
Owing to possible errors due user random- ization, we also performed a quasi-experimental analysis of our data to better quantify the effect our intervention. Interrupted Time Series (Fer- ron and Rendina-Gobioff, 2014, ITS) analysis is quasi-experimental method that allows us to assess and quantify the causal effect of our intervention a per-user basis. ITS models the user's behav- ior before and after the intervention and quantifies the effect of the intervention. As the comparison is intra-user, helps mitigate randomness arising from the inter-user comparison between treatment and control. We use ITSS with ARIMA modeling, which is expressed as
300c
Control c=5.2
Unconf. Corr. a=4.2 Conf. Incorr. a=1.0
Aocumulated 4
200g
100c
04
u 30 + B1t 82 4 +..
intervention
accurate
accurate
where 15>15 is the indicator variable indicating whether is after the intervention. We are inter- ested in the B2 values that indicate the coefficient of deviation from the user bet values before the in- tervention. Using ITS we find B2 -1.4 (pco.05 with two-sided t-test), showing a significant drop user bet value after the confidently incorrect in- tervention. We thus reject the null hypothesis and empirically verify H1.
Figure 6: Average accumulated reward. The is the primary coefficient of linear fits after the 15th stimulus (after intervention). Lines in black are fit using ordinary least squares (p<10-4 with two-sided t-test).
Consequences of lower trust. We now examine how user's fallen trust in the system affects their task performance. We assert that when the human's 's trust is calibrated, i.e., the human can effectively decide when the Al likely to be right wrong. signifies strong collaboration. The overall mon- etary gain, which the user accumulates, acts as a good proxy for the collaboration. To analyze this difference we fit a linear model after the interven- tion to predict the rate score increase. We model the cumulative gain at timestep as ( a + where is perceived as the expected gain in & per one interaction. We report for all three interventions. The results in Figure show that without interven- tion, a = 5.2, which is much higher than with unconfidently correct intervention (a 4.2) and confidently incorrect intervention (a 1.00.. No- tably, the confidently incorrect intervention has a more negative effect than the unconfidently cor- rect intervention. We thus empirically validate H3. miscalibrated examples significantly reduce the per- formance of the human-Al team in the long run.
1.0
Trust (bet value)
User Correctness
8c
0.8
0.6
6c
0.4
unconf. corr:
sccurate
Figure 5: Average user bet values (y-axis) and bet cor- rectness (point& & histogram color) with unconfidently correct intervention. The spline shows 3rd degree poly- nomial fitted with MSE. Transparent features are over- laid from control group (Figure 4, top).
Unconfidently correct intervention. We now the unconfidently correct intervention. From Figure? 2, this type of intervention is symmetric to confidently incorrect apart from the fact that the baseline model accuracy is 70%. Figure shows that users are much less affected by this type of miscalibration. A one-sided t-test shows a statis- tically significant difference between the average bet values across control and unconfidently correct
RQ1 Takeaways:
User trust in the Al system is affected by mis- calibrated examples.
Confidently incorrect stimuli reduce trust more
We ignore the moving average and error terms for brevity. See Appendix C for the full formula.
than unconfidently correct stimuli.

---

<
40
>
40
RQ2 Takeaways:
Int.
a
32
Bet
Acc.
Bet
Acc.
even 5 inaccurate confidence estimation exam- ples are enough to long-term affect users' trust with more inaccurate confidence estimation ex- amples, users more cautious
0
207
5.3
6.6
92%
6.8
92%
1
188
4.8
-0,51
6.2
87%
6.4
88%
3
193
5.0
-0.8
5.9
84%
5.9
82%
5
158
4.0
-1.4
5.4
86%
5.3
90%
4.3
Mistrust Transferability
7
147
3.7
-1.2
5.5
80%
5.5
86%
9
118
2.9
-0.9
5.6
72%
5.8
84%
Increasingly, single machine learning models are used on a bevy of different topics and tasks (Kaiser et al., 2017; OpenAI, 2023). Owing to the distribu- tion of the training data, the Al's performance will vary over input types. Although users are generally not privy to training data input types, Mozannar et al. (2022) show that users use this variance in model behavior to learn when the model is likely to be wrong. Inspired by this we ask:
Table 1:
Experiments with varying numbers of con- fidently incorrect stimuli. The a and the gain are shown from 19th sample (after intervention for all). The columns 40 and 40 signify which stimuli in the sequence are considered. All B are with < 10 with two-sided t-test apart from which is = 0.24.
4.2 Intervention Size
RQ3:
Do miscalibrated questions of one lype of question affect user trust in the model's 's output for different type of question?
Seeing a noticeable drop user trust when faced with model confidence errors, we ask:
In the next experiment, we simulate this by hav- ing two types of questions either related to trivia or math. Then, we introduce confidently incorrect intervention only for one of the types and observe the change in trust the other one. For example, we introduce confidently incorrect math questions and then observe how it affects trust on trivia stim- uli. We refer to the type of questions we provide intervention for as "affected" questions while the other as "unaffected" questions. We run two sets of experiments where we mix trivia and math affected questions.
RQ2: How many miscalibrated examples does it take break the user's s trust in the system?
We do so by changing the number of confidently incorrect stimuli from original 5 to 1, 3, 7, and 9 and measure how much after the intervention, how much they are betting immediately after the intervention and later one. We now discuss the average results Table 1.
able to
Upon observing increase in intervention size, we note initial decreasing trend followed by plateau in 32 44th column), implying a decrease in trust and user bet values, albeit only up to a cer- tain level. Shifting our focus to accuracy, which measures the users' ability to determine the Al's correctness, we observe an initial decline as well (6th column). This decline suggests that users adapt to the presence of miscalibrated examples. How- ever, after 40 examples (25 after intervention), the accuracy begins rise (8th column) once again, indicating that users adapt once more. Next, we an- alyze and a, which represent the total reward and the rate of reward increase. As the intervention size increases, both & and a (2nd and 3rd columns) con- tinue to decline. This means that the performance is what is primarily negatively affected. Based these findings, conclude that users possess the ability adapt their mental models as they encounter more calibrated stimuli. However, the decreased trust still leads them to place fewer bets on the system's ' predictions, resulting in a dimin- ished performance of the human-Al AI team.
The results in Figure show that there is gap between trust in the unaffected and affected stimuli BB Waffected = 5.04) type. The gap ( (Tunaffccted = 5.4c
is smaller than in the control settings (Figure 4) but still statistically significant (p<10-a with two- sided t-test). This is supported by the analysis using ITS where we look for the relative change to compare user bet values before and after the intervention. We find significant decrease bet values for both affected and unaffected questions (Baffected -0.94, Bunaffected = -0.53, p<0.05 with two-sided t-test).
RQ3 Takeaways:
Miscalibrated responses of one type affect the uscr's 's overall trust in the system
Miscalibrated responses of one type further re- duce user trust in examples of the same type Thus users also take into consideration question types they create mental models of the Al system correctness

---

84
1.0
5.1 Local Decision Modeling
Trust (bet value)
Unaffected
Affected
User Correctness
We start by modeling the user decision at a particu- lar timestep without explicit access to the history and based only on the pre-selected features that represent the current stimuli and the aggregated user history. These are:
A/
6c
0.6
0.4
46
0.2
intervention accurate
accurate
Figure 7: Average user bet values (y-axis) and bet cor- rectness (point & histogram color). The spline shows a 3d degree polynomial fitted with MSE. 'Affected" is the question type that undergoes confidently incorrect intervention.
Average previous bet value
Average previous TP/FP/N//EN decision. For example, FP means that the user decided the Al system correct which was not the case.
Al system confidence
Stimulus number in user queue
Model
Will agree? Bet value
Each sample (input) is turned into a vector, and we treat this as a supervised machine learning task for which we employ linear/logistic regression, de- cision trees, and multilayer perceptron (see code for details). We evaluate the model on dev set which is composed of 20% of users* which do not appear in the training data and present the results in Table 2. It is important to consider the uninformed baseline because of the class imbalance. The sults show, that non-lineat and autoregressive mod- els predict the user decisions better although not flawlessly.
Constant Baseline
81.8% (69.2%)
3.2c
Random Forest (stateless)
86.8% (81.8%
2.9c
Logistic/Lin. Regression
87.8% (82.0%)
2.1c
Random Forest
87.9% (82.8%
2.0v
Multi-Layer Perceptron
87.7% (82.9%
1.9g
GRU
89.7% (85.0%)
1.8c
Table 2: Performance modeling various aspects of user decisions. 'MAE Bet value' column shows mean absolute error 'Will agree?" is formatted as 'FI (ACC)'. See Section 5 for a description of target variables. 'State- less' uses only confidence an input feature.
Decision trees provide both the importance of each feature and also an explainable decision pro- cedure for predicting the user bet (see Figure 15). They also offer insights into feature importance via Gini index (Gini, 1912). For our task of pre- dicting bet value, it is: previous average user bet (63%), Al system confidence (3 1)), stimulus num- ber (1%). and then the rest. The R² feature values of linear regression reveal similar importance: pre- vious average user bet (0.84), Al system confidence (0.78), previous average TP (0.70) and then the rest. The mean absolute error for bet value prediction of random forest models based only the current confidence (stateless, i.e. history information) 2.94. This is in contrast mean absolute error of 2.0c for a full random forest model. This shows that the interaction history is key in predicting user trust.
5 Modeling User Trust
In human-Al collaboration systems, it is the col- laboration performance that is more important than the accuracy of the Al system itself (Bansal et al., 2021a). . In such cases, an AI system that can under- stand and adapt to how its output is used is more better. An important challenge in understanding the user's behavior is estimating how likely the user is to trust the system. This would also allow the system to adapt when user trust in the system is low by perhaps performing a positive intervention that increases user trust. We apply our learnings from the previous section and show that systems that explicitly model the user's past interactions with the system are able to better predict and estimate the user's 's trust in the system. We now develop in- creasingly complex predictive statistical models of user behavior, which will reveal what contributes to the user process and affects trust. For evaluation, we use Fi and accuracy (agreement) and mean ab- solute error (bet value) for interpretability.
5.2 Diachronic Modeling
Recurrent networks can selectively choose to re- member instances of the context that are crucial to making prediction. Unlike alternate approaches
'For example, (avg. bet: 6.7, TP: 50%, FP: 10%, TN: 30%, FN: 10%, conf: 81%, i: 13) *(30 + 30 + 30) 20% 60 = 1080 samples
"up E &TT,F) Will the user agrec? (F)) u E {0, 10] How much will the user bet? (MAE)

---

Inner product with previous hidden state
sion. After the intervention, their trust decreases over several interactions. Real-life user interactions with Al systems might not always follow this pat- tern. For example, a user might use a search engine in bursts when they have an information need. The larger time intervals between interactions might dampen the strong feelings of trust or mistrust.
0.3
0.2
Control
0.1
Confidently Incerrect
Unconfidently Correct
0.0
Experiment progression
Figure Vector similarity (inner product) between subsequent hidden states of the recurrent GRU model. See Figure 12 for comparison across queues.
Mistrust transfers between input types. Our ex- periments reveal that the model's miscalibration on certain type of input also reduces the user's trust in the model on other types of inputs. In real-world applications. Al systems are generally presented to users as an abstraction and the user may or may not be aware of the underlying workings of the sys- tem. For example, recent user-facing LLMs often employ techniques such as a mixture-of-cxpert or smaller specialized models that perform different tasks. In such cases, the transfer of miscalibration can be erroneous.
that use an average of mean interactions a user had with a system, GRU can effectively track where user trust in the system underwent a large change. To test this, we look at the information in the hidden state of the GRU we train on the user interactions (see Figures 8 and 12). The GRU's internal state is able to identify areas that caused shifts in the user's trust and changed their future interactions. This peak is much higher for the con- fidently incorrect than for the unconfidently correct interventions which in line with our conclusion that confidently incorrect examples deteriorate trust more than unconfidently correct examples.
RNN outperforms linear models in modeling trust. This is indicative that modeling user trust is complex and requires more sophisticated non-linear models. Like most deep learning mod- els, a recurrent network requires more data for accu- rate prediction. However, user- feer-facing applications can collect several features and with more data deep learning models might generalize better and help dynamically track and predict user trust.
6 Discussion
We now contextualize our findings to real-world applications and discuss the differences and their implications.
Miscalibration impacts user trust. Even a small (5) number of miscalibrated examples affects how users trust the system the future. In our con- trolled setting we consider symmetric riwa-d setup. However, past work has shown that trust is linked to risk. In real applications, the reward and cost of trusting the system might not be the same. For example in an Al system detecting cancer, hav- ing a doctor manualy do the screening has lower cost than a misdiagnosis.
Conclusion
When interacting with Al systems, users create mental models of the Al's prediction and iden- tify regions of the system's 's output they can trust. Our research highlights the impact of miscalibra- tions, especially in confidently incorrect predic- tions, which leads to notable decline in user trust in the AI system. This loss of trust persists over multiple interactions, even with just small num- ber of miscalibrations (as few as five), affecting how users trust the system in the future. The lower trust in the system then hinders the effectiveness of human-Al collaboration. Our experiments also show that user mental models adapt to consider different input types. When the system is mis- calibrated for specific input type, user trust is reduced for that type of input. Finally, our exami- nation of various trust modeling approaches reveals that models capable of effectively capturing past interactions, like recurrent networks, provide better predictions user trust over multiple interactions.
Confidently incorrect examples lower trust more than confidently correct examples. Stan- dard methods of evaluating model calibration, such as the Expected Calibration Error (ECE), do not take this into account. A holistic calibration metric should take these user-centric aspects into account, particularly, how users interpret these confidence scores and how it affects their trust in the system.
Miscalibration effects persist and affect user be- havior over long time spans. In our setup, users interact with the system continuously over a ses-

---

8 Future work
Monetary reward. A user interacting with an information system to seek information. In our experiments, we replace this goals with a monetary reward. This misalignment in the motivation also affects the participant behavior (Deci et al., 1999).
Regaining trust. We examined how miscali- brated examples shatter user trust and we show that this effect persists. We also show that this lack of trust adversely affects human-Al collaboration. Understanding how to build user trust in systems could greatly aid system designers.
Acknowledgments
We thank Hussein Mozannar and Danish Pruthi for their feedback various stages of the project. We also thank Shreya Sharma, Abhinav Lalwani, and Niharika Singh for being our initial test subjects for data collection. MS acknowledges support from the Swiss National Science Foundation (Project No. 197155), Responsible Al grant by the Hasler- stiftung; and an ETH Grant (ETH-19 21-1).
Complex reward structures. In our experi- ments, the user is rewarded and penalized equally when they are correct and incorrect. This re- ward/penalty is also instantly provided to the user. This might not hold for other tasks, for example, in a radiology setting, a false negative (i.e. miss- ing a tumor) has very large penalty. Past work in psychology has shown that humans suffer from loss-aversion (Tversky and Kahneman, 1992) and are prone making irrational decisions under risk. (Slovic, 2010). We leave experimentation involving task specifc reward frameworks to future work.
References
Ighoyota Ben Ajenaghughrure, Sonia C Sousa, Ilkka Jo- hannes Kosunen, and David Lamas. 2019. Predictive model to assess user trust: A psycho-physiologicaa approach. In 10th Indian conference human- computer interaction, pages 1-10.
Ethics Statement
The participants were informed that their data (anonymized apart from interactions) would be pub- lished for research purposes and had an option to raise concerns after the experiment via online chat, The participants were paid together with bonuses, on average, =/$2 per hour, which is above the Pro- lific's minimum of $12 per hour. The total cost of the experiment was $$500.
Ighoyota Ben Ajenaghughrure, Sonia Cláudia Da Costa Sousa, and David Lamas. 2021. Psychophysiological modeling of trust in technology: Influence of fea- ture selection methods. ACM on Hluman-Compunt Interaction, 1-25.
Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2021a. Is the most accurate Al the best teammate? Optimizing Al for teamwork. AAAI Conference on Artificial Intelligence, volume 35/13, pages 1 114141
Broader impact. As AI systems get more ubiq- uitous, user trust calibration is increasingly crucial. In human-Al collaboration, it is important that the user's trust in the system remains faithful to the system's 's capabilities. Over-reliance faulty AI be harmful and caution should be exercised during deployment of critical systems.
Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. 2019. Up- dates human-Al teams: Understanding and ad- dressing the performancecoompatiiity tradeoff. In AAAI Conference on Artificial Intelligence, volume 33/01, pages 2429-2437.
Limitations
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Ray- mond Fok, Besmira Nushi, Eee Kamar, Marco Tulio Ribeiro, and Daniel Weld, 202 Does the whole exceed its parts? The effect of Al explanations on complementary team performance. In CHI Confer- ence on Human Factors in Computing Systems, pages 1-16.
Simulated setup. Our experiments were con- ducted on users who were aware that their actions were being observed, which in turn affects their behavior (McGrath, 1995). We hope our work in- spires large-scale experiments that study how users interact directly with live system.
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol. et al. 2017. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 318(22):2199 2210.
Domain separation. In the Type-Sone Sepe--ensitivity Ex- periment (Section 4.3) we consider only two ques- tion types, trivia and math, and provide the partic- ipant with an indicator for the question type. In real-world usage, the user might provide inputs that may not be clearly distinct from each other.

---

Nina L Corvelo Benz and Manuel Gomez Rodriguez. 2023. Stuman-aligna calibration for Al-Assisted decision making. arXiv preprint 206.022306000074
Yunsan Guo, Jian Wang, Runfan Wu, Zeyu Li, and Lingyun Sun. 2022. Designing for trust: A set of design principles to increase trust in chatbot. CCF Transactions on Pervasive Computing and Interac- lion, 4(4):474-481.
Miguel A. Cardona, Roberto J. Rodríguez, and Kristina Ishmael. 2023. Artificial intelligence and the future of teaching and learning. Technical Report I, Depart- ment of Education, United States of America.
Rex Hartson and Pardha S Pyla. 2012. The UX Book: Process and guidelines for ensuring quality user experience. Elsevier.
Sabrina Chiesurin, Dimitris Dimakopoulos, Marco An- tonio Sobrevilla Cabezudo, Arash Eshghi, loannis Pa- paioannou, Verena Rieser, and Ioamnis Konstas. 2023. The dangers of trusting stochastic parrots: Faithful- ness and trust in open-domain conversational ques- tion answering. arXiv preprint arXiv:2305.165111
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint arXiv:1705.0355
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszko- reit. 2017. One model to learn them all. arXiv preprint arXiv: 1706.05770..
Chi-Keung Chow. 1957. An optimum character recog- nition system using decision functions. IRE Transac- sions Electronic Computers, 4:247-254.
Amita Kamath, Robin Jia, and Percy Liang, 2020. Se- lective question answering under domain shift. arXiv preprini arXiv:2006.099: 09462.
Edward L Deci, Richard Koestner, and Richard M Ryan. 1999. A meta-analh eetal-analytic review of experiments ex- amining the effects of extrinsic rewards on intrinsic motivation. Psychological bulletin, 125(6):227.
Zahra Rezaci Khavas, Reza Ahmadzadch, and Paul Robinette. 2020. Modeling trust in human-robor in- teraction: A survey. In Social Robotics: 12th Inter- national Conference, ICSR 2020, Golden, CO, USA, November 14-18, 2020, Proceedings 12, pages 529- 541. Springer.
Shehzaad Dhuliawala, Leonard Adolphs, Rajarshi Das, and Mrinmaya Sachan. 2022. Calibration of ma- chine reading systems at scale. arXiv preprint arXiv:2203. 10623.
Ran El-Yaniv et al. 2010. On the foundations of noise- free selective classification. Journal of Machine Learning Research, 11(5).
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones. Ming-Wer Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association of Compu- lational Linguistics.
John Ferron and Gianna Rendina-Gobioff. 2014. Inter- rupted time series design. Wiley StatsRef: Statistics Reference Online.
Biniam Gebru, Lydia Zeleke, Daniel Blankson, Mah- moud Nabil, Shamila Nateghi, Abdollah Homai- far, and Edward Tunstel. 2022. A review on human- machine trust evaluation: Human-centric and machine-centrie perspectives. IEEE Transactions on Suaman-Machine Systems, 5255)9552.92.
Po-Ming Law, Leo Yu-Ho Lo, Alex Endert, John Stasko, and Huamin Qu. 2021. Causal perception in question- answering systems. In CHI Conference Human Factors in Computing Systems, pages 1-15.
Corrado Gini. 1912. Variabilità mutabilità: Contrib- allo studio delle distribuzioni e delle relazioni statistiche, (Facc. I.J. Tipogr. di P. Cuppini.
John D Lee and Neville Moray. 1994. Trust, self- confidence, and operators' adaptation to automation. International journal of human-computer studies. 40(1):153-184
Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, and Srinivasan Tyer. 2020. Human evaluation of spoken vs. visual ex- planations for open-domain qa. arXiv preprint 15X7v:2012.15075.
Zhuoyan Li, Zhuoran Lu, and Ming Yin. 2023. Model- ing human trust and reliance Al-assisted decision making: A markovian approach.
Nelson F. Liu, Tianyi Zhang, and Percy Liang, 2023. Evaluating verifiability in generative search engines. arXiv.
G Mark Grimes, Ryan M Schuetzler, and Justin Scott Giboney. 2021. Mental models and expectation vi- olations in conversational Al interactions. Decision Support Systems, 144:113515
Shuai Ma. Ying Lei, Xinru Wang. Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. 2023. Who should trust: AT or myself? Leveraging human and AI correctness likelihood to promote appropriate trust in Al-assisted prcissonnmakinng arXiv preprint arXiv:2301.05800
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. International conference on machine learn- ing, pages 1321-1330. PMLR.

---

Ruth Mayo. 2015. Cognition is matter of trust; Dis- trust tunes cognitive processes. European Review of Social Psychology, 26(1):288 (6(1):283-327.
Brian Stanton, Theodore Jensen, et al. 2021. Trust and artificial intelligence. preprint.
Carmen Thoma. 2016. Under-versus overconfidence: An experiment on how others perceive a biased self-- assessment. Experimental Economics, 19:218-22 2-99.
Joseph E McGrath. 1995. Methodology matters: Doing research in the behavioral and social sciences. In Interaction, pages 169. University of Illinois, Urbana.
Readings in
Amy Tumer, Meena Kaushik, Mu-Ti Huang, and Srikar Varamasi. 2022. Calibrating trust in Al-assisted deci- sion making.
Bhaskar Mitra, Nick Craswell, et al. 2018. An introduc- tion to neural information retrieval. Foundations and Trends@ Information Retrieval, 13(1):1-126
Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of un- certainty. Journal of Risk and uncertainty, 5:297- 323.
Hussein Mozannar, Arvind Satyanarayan, and David Sontag, 2022. Teaching humans when to defer to a classifier via exemplars. In AAAI Conference on Arti- ficial Intelligence, volume 36/5, pages 5323-5331.
Alicia Vikander. 2023. Background explanations reduce users' over-reliance on AI: A case study on multi-hop question answering.
Cecilie Bertinussen Nordheim, Asbjern Følstad, and Cato Alexander Bjørkli. 2019. An initial model of trust in chatbots for customer service finding- from questionnaire study, Interacting with Computers, 31(3):317-335.
Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. 2022. Uncalibrated Models Can Improve Human-Al Collaboration. ArXiv:2202.0598 [cs].
Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Don- ald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring bias affects mental model formation and user reliance in explainable ai systems. In 26th International Conference on Intelli- gent User Interfaces, pages 340-350.
Xinru Wang and Ming Yin. 2022. Effects of expla- nations in Al-assisted decision making: Principles and comparisons. ACM Trunsactions on Interactive Intelligent Systems, 12(4):1-36.
Ming Yin, Jennifer Wortman Vaughan, and Hanna Wal- lach. 2019. Understanding the effect of accuracy on trust in machine learning models. In CHI conference on human factors in computing systems, pages 1-12.
OpenAl. 2023. GPT-4 technical report.
Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. 2019. How model accuracy and explana- tion fidelity influence user trust. arXiv preprint arXiv:1907.12652.
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on ac- curacy and trust calibration in Al-assisted decision making. In Conference on fairness, accountability. and transparency, pages 295-305.
Andrea Papenmeier, Dagmar Kern, Gwenn Englebienne, and Christin Seifert. 2022. It's complicated: The relationship between user trust, model accuracy and explanations AI. ACM Transactions on Computer- Human Interaction (TOCHI), 29/4:1-33.
Jianlong Zhou, Huaiwen Hu, Zhidong Li, Kun Yu, and Fang Chen. 2019. Physiological indicators for user trust in machine learning with influence enhanced fact-checking. In Machine Learning and Knowledge Extraction: Third IFIP TC TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2019, Canterbury, UK, August 26-29, 2019, Proceedings 3, pages 94-1 13. Springer.
Amy Rechkemmer and Ming Yin. 2022. When confi- dence meets accuracy: Exploring the effects of multi- ple performance indicators on trust in machine learn- ing models. In Chi conference on human factors in computing systems, pages 1-14.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. why should trust you?" explain- ing the predictions of any classifier. 22nd ACM SIGKDD international conference on knowledge dis- covery and data mining, pages 1135-1144.
Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Koemi, Yuchen Eleanor Jiang, and Mrinmaya Sachan. 2023. Poor man's 's quality estimation: Predicting reference-based MT metries without the reference. In 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 1303-1317.
Pedro Rodriguez, Shi Feng, Mohit Tyyer, He He, and Jordan Boyd- Graber. 2019. Quizbowl: The case for incremental question answering. arXiv preprint arXiv:1904.0477
Vilém Zouhar, Michal Novák, Matúš Žilinec, Ondřej Bojar, Mateo Obregón, Robin L Hill, Frédéric Blain, Marina Fomicheva, Lucia Specia, and Lisa Yankovskaya. 2021. Backtranslation feedback im- proves user confidence in MT, not quality. . In Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 15 151-161.
Paul Slovic. 2010. The psychology of risk. Satide e Sociedade, 1944)7331-747.
Lucia Specia, Dhwaj Raj. and Marco Turchi. 2010. Ma- chine translation evaluation versus quality estimation. Machine translation, 24.39-50.

---

Queue
Users
Stimuli
Conf. Incorr.
Unconf Corr.
Control
Control
39
2340
[75%. 100%]
Intervention CI
27
1620
system confidence
Intervention CI 3
39
2340
[50%, 75%)
Intervention CI 5
30
1800
Intervention CI
1800
125%, 50%)
Intervention CI 9
30
1843
[0%, 25%)
Trivia intervention CI
31
1860
Math intervention CI
31
1860
Intervention UC
40
2400
Average bet value (c)
Total
297
17863
Figure II: Average het for particular confidence inter- val before and after intervention ( means bet increased after intervention and means decrease). tion reduces the bet value which is otherwise naturally increasing. See Figure 10 for bet distribution.
Table 3: Size summary collected and released data.
interven-
Next Bet Decision
20
4,0s
Time (s)
post
0.07
0.09
0.10
0.07
0.11
0.07
Int. CI uid
0.12
0.14
0.15
0.14
0:21
0.14
pee
0.08
0
0.15
30
40
50
60
pusl
0.08
0.09
0.00
0.07
0.13
0.06
Control mid
0.06
0.08
0.08
0.08
0.14
0.08
0.10
Figure 9: Breakdown of duration of individual user actions. While the bet value decision (bet) and reading the results (next) remain rather constant, the overall decision process becomes faster.
pee
0.06
0.06
0.07
0.07
0.13
0.07
pre
mid
post
pro
post Intervention
mid
Control
C1
Figure 12: Similarity (inner product) of GRU hidden states for bet value prediction at different locations dur- ing the experiment (pre & 10, mid < 15, post > 15) and groups (control/confidently incorrect).
30%
User incorrect
Frequency
User correct
20%
10%
B Real AI System Confidence
0%
04
26
46
64
8c
10c
In all our experiments (control and with different in- terventions) we find a similar ECE scores (control:
Figure 10: Overall distribution of (bipolar) bet values across all collected data.
0.29% UC-5: 0.30%, C1-1: 0.28%, C1-3: 0.29%, CI-5: 0.28%, CI-7: 0.29%, CI-9: 0.29%) This is due to the intervention sizes being very small have major effect on the ECE score. We com- pare to other models: DPR on Natural Questions: 37.1%, ResNet-152 on Imagenet: 5.48%.
A Question Generation
We generate 60 trivia and math questions using ChatGPT using the following two prompts. We manually filter questions which are possibly swerable given expert knowledge. See Figure 13 for examples of generated questions. All the gener- ated questions are part of the released data.
C
Interrupted Time Series
B u
= = + + t + B2 1(t 15)
Generate fake mathematical question that seems like they are answerable but key information is missing, Generate two plausible definitive answers, the first of which "correct".
Σ du
0 +
Generate fake trivia question that seems like they are answer- able but a key information is missing and they are not related real world. Generate two plausible definitive answers, the first of which is "correct".
Moving average terms
Error terms

---

Progress:
Question: What is the main export of the island nation of Mirrantia?
Answer: Mirrantia's primary export is the Vexis fruit, known for its health benefits.
Confidence: 68%
System System correct
How confident are answer?
If you $0.08. you wrong, lose $0.08.
guessed that system was correct system was correct. You gain $0.08.
Progress: 2/60
Balance: $0.06 $0.5
What the main ingredient traditional Bandhani Halwa, popular sweet dish the city Brahmapur?
Question:
Answer: Bandhani Halwa primarily made with the delicious Golden Dates sourced from local farms.
Confidence: 52%
correct
System not correct
How confident are your answer?
If
right, $0.02. you lose $0.02.
guessed that the system You lose $0.02.
the system
Progress: 3 60
Balance: $0.16 80.5
Question: What is the traditional dance performed during the Harvest Festival in the fictional land Veridian?
The traditional Harvest Festival celebration Veridian features the enchanting dance called the Harvest Rhapsody.
Answer:
Al Confidence: 23%
System System
How confident you your answer?
you right, you you wrong, you
You guessed that the system was incorrect and the system was not correct. You gain $0.10.
Progress:
30.16
Question: What is the national dish of the fictional country of Sylvaria?
The national dish Sylvaria, a land known for its bountiful forests, is the mouthwatering Fungi Risotto.
Answer:
Al Confidence: 62%
System is correct
System not correct
How confident are you in your answer?
If you are right, you get $0.00. you are wrong, you lose $0.00.
You guessed that the system was incorrect but the system was correct.
You lose $0.00.
Figure 13: Screenshots of the user interface with all combinations (4) of user guess that the system was correct/in- correct and the model being correct or incorrect.

---

confidently incorrect stimuli
45 calibrated stimull
histograms of accuracy (projected from points) upper (eemittaansprent) control group bottom is confidently incorrect queue
10
calibrated stimul
1.0
Trust (bet value)
User Correctness
spline from control group (without interventions)
8c
0.8
0.6
spline fitted to bet values
6c
0.4
4c
0.2
conf. incorr. accurate
accurate
average accuracy of 11th stimulus is 20% average bet value is 8c
average accuracy of 35th stimulus is 85% average bet value is 6c
Annotated version Figure 4. Average user het values (y-axis) and bet correctness (point & histogram color) with control set of stimuli (top) and confidently incorrect stimuli (bottom). The spline shows 3rd degrec polynomial. Transparent features are overlayed from the other graph
Figure 14:
2.85, #sample:
prior bet avg
10
2. 63,
#samp 1858
MAE: 1.53, #samp 971
co_f 68%
wrior_bet_aee 0.08
8
16
MAE: 2.29, #sample: le: 1339
MAE: 2. 00, le: 519
MAE: 2.26, #sample: 351
NAE: 8,88, #sample: 628
prior bet avg < 0.04
prior bet avg 0.e3
ai conf <> 72%
TP avg <> 85%
Figure 15; First three layers of decision tree that predicts bet value (in gray for each node).


Reviews:
[{"review_id": "yUMFFMB7fw", "reviewer": "Reviewer_Xqzv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very favorable review. We are glad you enjoyed reading our work and found it impactful!\n\nWe have addressed your concerns below: \n\n**Medium and high stakes scenarios.**\nWe agree that a current limitation of this study is that it only studies cases where the risk-reward structure is symmetric. \nWe conducted a very preliminary study where we provided asymmetric rewards (i.e. higher penalty for being incorrect than reward for being correct and vice versa). For confidently incorrect intervention size of 5 we observe a higher mean bet value (0.72 vs 0.63) when we double the reward for the same risk, i.e., users are willing to bet more as the possible reward is higher. \nWhen increasing the penalty we however do not notice a commensurate decrease in bet values. These results demonstrate a need for future research into how the risk to reward ratio can affect user interactions with AI systems. We are happy to incorporate a discussion on this.\n\n**Other Factors affecting user trust.** \nWe fully agree that model explanations are one of the key contributors to establishing user trust (L096-L100). However, we consider it to be beyond the scope of our current work and leave studying the diachronic effects of model explanations on user trust as future work.\n\n**Access to oracle answers.**\nWe acknowledge that having an oracle give you immediate feedback is not realistic. However we believe that there still is a feedback loop (i.e. the user is eventually informed if the AI was correct or not) albeit not immediately. Using an oracle to reveal the answer (immediate feedback) is commonly used in literature [1,2, inter alia]. Using a delayed feedback would introduce other sources of noise and we leave these investigations, which would explicitly control for this factor, to future work.\n\n- [1] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://dl.acm.org/doi/pdf/10.1145/3411764.3445717). In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery\n- [2] Gonzalez, Ana Valeria, et al. [Human evaluation of spoken vs. visual explanations for open-domain qa.](https://arxiv.org/abs/2012.15075) Association of Computational Linguistics ACL (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I truly enjoyed reading this paper, it's extremely well written, well structured and presents a very interesting experimental design.\n2. This paper is likely to inspire a lot of follow up work. Trustful LLMs is a growing area and the experiments presented in this paper could easily extended to other tasks, (synthetic/real) datasets and user groups, as pointed out in §8. As such, I believe this paper is impactful to a large part of the ACL community.\n3. The authors control for confounding factors such as real world prior knowledge, monetary reward instead of asking participants' confidence in the answer, thereby establishing a clean test environment sufficient to support the presented hypotheses. \n", "reasons_to_reject": "1. As pointed out by the authors (L616-629), the experimental setup presents a low-stake scenario, the trust function (approximated by a monetary utility function) would look different in medium and high-stake scenarios. \n2. User trust is also impacted by factors other than self-reported model confidence, model output and the information about the correctness of the answer, including, e.g., the nature of the task and the model's ability to explain the output generation process.\n3. The study assumes access to an oracle that informs users if the model answer is correct (\"AI correctness revealed\"), impacting the mental model. This assumption often does not hold in real world scenarios. If we had access to an oracle, we would not need to interact with the agent. If we don't have access to an oracle, we could not update the mental model by revealing the answer. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors investigate how humans trust AI agents over time with increasing number of engagements. The experiment design involves human participants  interacting with agents in multiple rounds of question answering. In each round a participant is presented with a question together with the model's answer and output confidence. Given this information, the task is to either accept or reject the output. Each time the particpant is also asked to place a monetary bet on the correctness of the model's answer, serving as a proxy for trust. In this controlled experiment, the authors use ChatGPT to generate fictional questions not grounded in reality and also control model calibration. \n\nAs a major contribution of this work, the authors show that after participants are presented with miscalibrated output (unconfidently correct, UC; or confidently incorrect, CI) the human trust in the agent decreases, which is reflected by a decrease in bets. Other contributions include (1) showing that CI harms user trust more than UC, (2) investigating the interaction between the frequency of subsequent bad interactions, i.e. miscalibrated outputs, and user trust and, (3) the impact of miscalibrated outputs on the overall trust across different question types. Lastly, the authors propose a way to model user trust over time.\n", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TXhVmTZ2jg", "reviewer": "Reviewer_3PXu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We concur that studying user trust is an important area. We are also happy that you found the paper well organized, our work well motivated, and our experimentation thorough. \n\nWe have addressed your concerns below:\n\n**Excluding prior bias.**\nAs reviewer _Xqzv_ points out, we made deliberate efforts to control for prior knowledge of the participants in order to isolate the effect of the model's reported confidence on user trust and productivity. Without this, we would be introducing a confounder which will make it challenging to answer our research questions. We acknowledge that in the real world the users may have some partial knowledge regarding the answer which they could use in their decision-making.\n\nWe believe that our work enables future studies which attempt to control for the effect of prior knowledge and bias. For example, deliberately providing answers which _conflict_ with the user’s prior knowledge and measure this effect in comparison to trust loss caused by confidently incorrect or unconfidently correct outputs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-organized and easy to follow.\nThe paper is well motivated and the problem of modeling user trust is important and challenging for applying AI models.\nThe paper conducts a good amount of experiments with interesting research questions.", "reasons_to_reject": "The user study is running on a simplified setting, e.g., users do not know the answers, users do not have any prior bias. Given such an impractical setting, the contribution of the findings might be limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies users' mental model of the AI system in a human-AI collaboration enviroment. The paper is motivated by the negative impact of incorrect prediction on user trust and the need of understanding the dynamics of this process. The authors conduct a user study where users interact with a question answering system. Users' trust is proxied using monetary rewards. The paper uses a simulated AI to control question generation, confidence, and answer correctness. The authors conduct a set of tests to verify their hypothesis. Based on the findings, the authors create a classifier and a regression model to predict users' agreement and bet. The authors also use a GRU model to capture the impact of interaction history.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JX79OE45Ew", "reviewer": "Reviewer_4Q7E", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are glad that you found our work well written, and our monetary setup to be promising and interesting! \n\nWe’ve addressed your concerns below:\n\n**Impact and contributions.**\nWe agree that miscalibration’s link to trust is expected. However, several of our follow-up research questions do provide interesting insights that are novel in context of past literature in HCI.\n- The diachronic effects of miscalibration on trust have not been explicitly studied. Several recent papers [1,2, inter alia] only study the effects of confidence scores on trust and neglect the temporal effects. \n- We show that miscalibrations in the past can have detrimental effects on user trust which can reduce the user-AI performance in the future. \n- Our findings on the difference in effect between confidently incorrect and unconfidently correct examples is novel. Current calibration metrics (e.g. ECE, Brier score) treat overconfidence and underconfidence as the same. \n- Our extensions to understanding user trust with miscalibrations in the context of different input types builds on very recent work [3] on how users create mental models over different types of inputs.\n\n\n**Multi agent computational trust.**\nThank you for suggesting work in trust modeling from the multi-agent system community [4,5]. This line of research has also been missed in several recent HCI works that model user trust. We will update our related work section to include papers about modeling trust from the traditional AI and multi-agent community.\n\n**RNNs vs Beta dist trust model.**\nWe ran initial experiments using a simple beta distribution trust model (BDTM) to predict the probability of the user trusting the AI system (Table 2 in our work). We used the description [here](https://arxiv.org/pdf/1806.03916.pdf) to implement the model. We observe an accuracy of 78%. Compared to Table 2 in the paper, the BDTM performs better than a constant baseline (69%) and is in the range of a single layer logistic regression model (82%)  but it underperforms the RNN (85%). \nOne of our major conclusions from the trust modeling experiments is that an RNN  given its ability to be stateful can better model the user’s trust which we show in the prior sections can be drastically changed by a small intervention. \nIn our experiments we show that the RNN outperforms models that only have access to the averages of the past. \n\nAlthough BDTM's performance could be enhanced with more intricate techniques for weighting and discounting past experiences, in our context with just 2 agents, a recurrent method like the RNN gives us a powerful and flexible function that can be easily trained from past experiences. We will add a more detailed discussion regarding this to the paper.\n\n\n- [1] Vodrahalli, Kailas, Tobias Gerstenberg, and James Y. Zou. [Uncalibrated models can improve human-ai collaboration.](https://proceedings.neurips.cc/paper_files/paper/2022/file/1968ea7d985aa377e3a610b05fc79be0-Paper-Conference.pdf) Advances in Neural Information Processing Systems 35 (2022): 4004-4016.\n- [2] Gonzalez, Ana Valeria, et al. [Human evaluation of spoken vs. visual explanations for open-domain qa.](https://arxiv.org/abs/2012.15075) Association of Computational Linguistics ACL (2021).\n- [3] Mozannar, Hussein, Arvind Satyanarayan, and David Sontag. [Teaching humans when to defer to a classifier via exemplars.](https://arxiv.org/pdf/2111.11297.pdf) Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 5. 2022.\n- [4] Josang, Audun, and Roslan Ismail. [The beta reputation system.](https://domino.fov.um.si/proceedings.nsf/0/d9e48b66f32a7dffc1256e9f00355b37/$FILE/josang.pdf) Proceedings of the 15th bled electronic commerce conference. Vol. 5. 2002.\n- [5] Wang, Yonghong, and Munindar P. Singh. [Evidence-based trust: A mathematical model geared for multiagent systems.](https://dl.acm.org/doi/10.1145/1867713.1867715) ACM Transactions on Autonomous and Adaptive Systems (TAAS) 5.4 (2010): 1-28.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written, well organized, and easy to follow\n- The monetary simulations are promising, new, and interesting\n", "reasons_to_reject": "- Most of the findings are expected with limited impact\n- Trust modeling with models such as RNN shows good result, but it might be a bit over-kill.  It might be worth comparing traditional simple beta-distribution based computational trust models with confidence estimate in multi-agent system literature.  They are simple, and highly applicable in the settings discussed in this paper, e.g., Evidence-based trust: A mathematical model geared for multi-agent systems by Wang and Singh.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies how mis-calibrated confidence scores affect user trust in AI systems.  The authors design a monetary betting simulations to study various effects of mis-calibration.  The results show that (1) over-confident wrong responses hurt more than under-confident correct responses, (2) user trust is hard to build but relatively easy to lose, (3) user trust in one question type may be transferrable to the others, and (4) user trust can be modeled.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Trust modeling and reputation systems in traditional AI and agent community, e.g., The beta reputation system by Josang.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a well-thought-out user study about how model miscalibration affect the user's trust over multiple trials. All reviewers agree that the paper is well-written, the research question is important, and that the experimental setup of using bets to measure trust is novel. Reviewer 4Q7E recommended beta-distribution based computational trust models as baselines, which is addressed by the authors. The only downside of this paper is the generalizability of results from the simplistic setting, but the authors made reasonable efforts to improve that by controlling for the user's prior knowledge. This paper presents a good blueprint for future work studying the temporal effect of calibration on trust in AI.

For related work, I recommend including HCI literature on the impact of feedback on user trust, e.g., No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML by Smith-Renner et.al.
Decision: Accept-Main