What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Authors: Navita Goyal
Eleftheria Briakou
Amanda Stephanie Liu
Connor Baumler
Claire Bonial
Jeffrey Micher
Clare R. Voss
Marine Carpuat
Hal Daumé III
Venue: EMNLP 2023 Main (2023)
Keywords: human-centered NLP
over-reliance
explainability
TL;DR: We conduct a human-study on how users interact with QA systems in the absence of sufficient information to assess their predictions and whether adding the requisite background alleviates the concerns around over-reliance in model predictions.

Abstract: NLP systems have shown impressive performance at answering questions by retrieving relevant context. However, with the increasingly large models, it is impossible and often undesirable to constrain models' knowledge or reasoning to only the retrieved context. This leads to a mismatch between the information that \textit{the models} access to derive the answer and the information that is available to \textit{the user} to assess the model predicted answer. In this work, we study how users interact with QA systems in the absence of sufficient information to assess their predictions. Further, we ask whether adding the requisite background helps mitigate users' over-reliance on predictions. Our study reveals that users rely on model predictions even in the absence of sufficient information needed to assess the model's correctness. Providing the relevant background, however, helps users better catch model errors, reducing over-reliance on incorrect predictions. On the flip side, background information also increases users' confidence in their accurate as well as inaccurate judgments. Our work highlights that supporting users' verification of QA predictions is an important, yet challenging, problem.

What Else Do Need to Know? The Effect of Background Information on Users' Reliance on QA Systems
Navita Goyal', Eleftheria Briakou², Amanda Liu', Connor Baumler', Claire Bonial', Jeffrey Micher), Clare R. Voss", Marine Carpuat', Hal Daumé III
UUniversity of Maryland, GGoogle, 'U.S. Army Research Lab, *Microsoft Research navi ta@cs. umd. edu, ebriakou@google, com, amandastephanielli@@galll com, baumlerêcs. aind.dd,, {claire. , jeffrey. c. c. cher). ci vêarmy mil, clare.r. voss. www..v..s.............. wwwrrre..ammiiiea mil, marine@umd. edu, me@hal3. name
Abstract
user interacts with QA model to seek informa- tion about which non-Swedish actress starred in the movie "Light Between Oceans." " The QA model retrieves the context (left) about the movie and pre- diets Alicia Vikander as an answer based on this context. In such workflows, the retrieved context serves as supporting evidence for the predicted an- swer. However, in order to assess the accuracy of this prediction, the user must already know that Vikander is Swedish, a piece of information that is not present in the provided context. This leads to a knowledge gap between the information that is required to answer the question and the information that is made available to the user.
NLP systems have shown impressive perfor- mance at answering questions by retrieving rel- evant context. However, with the increasingly large models, is impossible and often unde- sirable to constrain models' knowledge or rea- soning to only the retrieved context. This leads mismatch between the information that the models access to derive the answer and the in- formation that is available to the user to assess the model predicted answer. In this work, study how users interact with QA systems in the absence of sufficient information to assess their predictions. Further, we ask whether adding the requisite background helps mitigate users' over- reliance on predictions. Our study reveals that users rely on model predictions even in the ab- sence of sufficient information needed to assess the model's correctness. Providing the relevant background, however, helps users better catch model errors, reducing over-reliance on incor- rect predictions. On the flip side, background information also increases users' confidence in their accurate as well as inaccurate judgments. Our work highlights that supporting users' ver- ification of QA predictions is an important, yet challenging, problem.
Chat-based interfaces are becoming increasingly popular for information-seeking In the case of factoid questions, these systems commonly surface retrieved or generated information that supports the provided answer. This supporting information is commonly seen through the lens of "information necessary for the underlying model to make the cor- rect prediction." However, this does not necessarily translate to "information sufficient for users to as- sess the prediction." " Recent work in explainable NLP (Fok and Weld, 2023; Xie et al., 2022) argues that support users in decision-making and foster appropriate reliance, we should focus on providing users with information that aids them in assessing the correctness of model predictions. However, the effects including such additional information on users' reliance on a QA model are not well studied.
1 Introduction
With the advent of large language models pre- trained on massive data, question answering (QA) systems are able to reason about information that is external to their input context, based on their im- plicit factual or commonsense knowledge (Petroni et al., 2019; Jiang et al., 2020) or by employing shortcuts get to the correct answer (Min al., 2019; Chen and Durrett, 2019; Trivedi et al., 2020). However, in such incomplete information settings, users may lack important background knowledge required to assess the correctness of model predic- tions: consider the example of Figure 1. Here, Work done while at the University of Maryland.
In this work, we design a Wizard-of-Oz experi- ment to study how users interact with QA model when some of the information required to assess the correctness of the model prediction is miss- ing. Further, investigate how users' reliance on model predictions and confidence in their judg- ments change when the requisite background the information required to assess the correctness of the prediction provided to the user. The back- ground serves as an extension of the model's in-

---

Question: Which non-Swedish actress sturrea The Light Ocoans?
The Bee Oeean (fte)
The iglt Berwoe Ocevss isa conertt peried druna fikn writcee and diected by Dack Cantfranc and hared on be 2012 rand the care mers by MI S-Sudraas As inaeratoooi co- pooductian betwern the Uisted Staces Nastilli the Unadd Kinoker,, and New Zeabrr, litn Fasdendes.
Courect
Background
Alids Vikander
Alids Areanda Viceskr (bere Genbeer (959)
Swudish amoss nor dincer riludd Oudletbor,, hopa
disagree
Alkla Vilonder
Buckel
Stochhlm and Schood of Arnenian New York. Sty bega fer profoc acting conner Is apyearing is Swodisk saen fibea and toleviikee eno and firi anised nac saceppttion in Namtern lumpe for tee colo Josutu RyTre Tegis witt in Creae Avergef (2006 199.
eaory an's lighthonss kaper wile socae adas infart was, Yoar lazs coople sheeeer childt tms percsigge and ar facd with the moril dierrza of huit aeties.
Contect
Lagres, but am confideut...
Figure I: QA model may make a prediction based on insufficient context (left), making it difficult for users to assess its correctness. Providing the necessary background information (right) might reduce the user's over-reliance.
put context and prediction where the goal is not to justify the prediction but rather to provide addi- tional context that may establish a common ground between users and the model, a prerequisite for
viate the issue of overconfidence. This indicates the need to further examine the question of how to best aid users in AT-assisted decision-making.
Research Questions
effective collaboration (Bansal et al., 2019).
Further, we study whether the cognitive effort in consuming background mediates the effect of back- ground on users' reliance on model predictions and confidence in their judgments. We employ high- lights to surface the important parts of the context and the background to the user, making these casier to consume. We assess whether this highlighting of the relevant content helps users better rely the model and calibrate their confidence.
We conduct an online user-study' based on Wizard- of-Oz experiments to examine how users' reliance on model predictions and confidence in their judg- ments change based whether they are given the background information required verify the pre- dictions of a QA model. Our study targets four key research questions:
RQI: How do users interact with the model pre- dictions in the absence of sufficient information to assess their correctness? Auman-Al decision- making often relies on humans to critically assess model predictions instead of simply agreeing with the model (Bussone al., 2015; Buçinca et al., 2021; Green and Chen, 2019). Users may fre- quently lack the necessary background assess the correctness of the model. In such cases, we might expect users to either disagree with the model or have low confidence in their judgments. We study whether users indeed calibrate their confidence and reliance on the model when they likely lack enough information to assess the veracity of its predictions.
We control the sufficiency of the information shown to users by employing a multi-hop question- answering dataset. We consider QA model that predicts the answer to a complex question based on input context that alone is insufficient to answer the question without additional background knowledge. Using this model, we design user- study where we test how the user's agreement with the predicted answers changes when only part or all of the information is disclosed to the users. Our findings are summarized as follows:
o In line with previous findings users' over- reliance on AI predictions (Bansal et al., 2021; Bussone et al., 2015; Jacobs al., 2021; Lai and Tan, 2019), users over-rely on model predictions even in the absence of the relevant background required to assess their correctness.
RQ2: Does adding requisite background informa- tion allow users to calibrate reliance and confi- dence on model predictions? Building on RQ1, we study whether having access to information re- quired to assess model predictions helps users make more informed decisions with higher confidence. We examine whether users are able to appropriately calibrate their reliance models when provided with the relevant information to discern correct and incorrect model predictions.
Adding the requisite background helps users bet- ter identify incorrect model predictions, signifi- cantly reducing over-reliance on the model. o Background also significantly increases users' confidence in their own judgments, both when they are correct and when they are incorrect.
Including highlights with background does not reduce users' over-reliance on the model or alle-
'The study was approved by the University of Maryland Institutional Review Board (IRB number 1941629-1).

---

RQ3: Are users able to calibrate their reliance and confidence even when not all background provided is perfect? Extending RQ2, which ex- plores users' reliance and confidence when pro- vided with a background that enables perfect ver- ification of model predictions, we study whether users are able to calibrate their reliance and confi- dence in model predictions when the background is sometimes, but not always, perfect. This reflects a more realistic scenario where users might be at times provided with background information that is ungermane: on topic, but not directly useful for an- swering the question. Comparing ungermane and germane backgrounds also helps in unconfounding the effect of the presence of more information vs the effect of useful background.
Each question in the HotpotQA dataset is associ- ated with two or more relevant context paragraphs: the context paragraph containing the gold-standard answer to the complex question and the other inter- mediate paragraphs providing background context corresponding to the required reasoning steps im- plicit in the question. We use the context paragraph with the gold-standard answer the supporting context in the no background condition (RQ1) and include the background context along with the sup- porting document the with background condition (RQ2). We only consider questions that involve exactly two documents to ensure that the relevant background is sufficient to assess the model predic- tion. For the mixed background condition (RQ3), we sample distractor document that is retrieved us- ing the question, but that is not the required support- ing document for answering the multi-hop ques- tion as the ungermane background. Further, the HotpotQA dataset identifies the sentences in each paragraph that are essential supporting facts for answering the question. We use these supporting facts to highlight the context and the background information in the with background and highlights condition (RQ4). See Table 1 in the Appendix for examples.
RQ4: Does highlighting important parts of con- text and background improve users' reliance and confidence calibration? Background informa- tion, although potentially useful, also increases the information load and cognitive burden on the end user (Kaur et al., 2020). Thus, the effect of the pres- ence of background is potentially mediated by the effort required in consuming the said background. To assess the mediating effect of the effort required consume background, we introduce highlights in both the context and the background, identifying sentences that are crucial to answering the question.
In realistic scenarios, the background informa- tion and highlights would be automatically trieved or generated. However, for the purposes of this study, automating background and highlight extraction confounds the utility of the information presented to the end-user with the performance of the automated system. The proposed Wizard-of-Oz setting allows us more control over the background and highlights exposed to the end-user. This allows us to make stronger conclusions whether and how users make use of additional information when is known to be necessary and sufficient to assess model predictions in advance of future research on how to best extract and evaluate such information automatically.
3 Designing a Wizard-of-Oz System by Using Multi-Hop Dataset
We design a Wizard-of-O study with insufficient (no background) and sufficient (with background) information conditions. We employ a multi-hop question answering dataset, HotpoiQA (Yang et al., 2018), which requires multiple pieces of evidence to reason the answer. We repurpose the HotpotQA dataset control the information available to the model by only surfacing partial information ex- amine RQI and providing the missing information to examine RQ2. To approximate a near-realistic setting, we consider a subset of HotpotQA on which the model predictions remain unchanged when the background is added to the input to the model. This mimics a scenario where a QA model can make a prediction based partial information using heuristic shorteuts (Min et al., 2019; Chen and Durrett, 2019; Trivedi et al., 2020) or implicit knowledge acquired through pre-training (Petroni et al., 2019; Jiang et al., 2020), but users would lack sufficient background to assess its correctness.
We choose a question answering model with the following desiderata: we want a model that (1) is realistic representative of a standard QA model and (2) can make correct predictions based on its implicit background knowledge (as acquired from pre-training data), even when given questions and partial context, that is insufficient to answer the questions. Based on this criteria, we select a 336M parameter BERT model (Devlin et al., 2019) fine-tuned on the Stanford Question Answering

---

Dataset (SQuAD) (Rajpurkar al., 2016), which consists of extractive question answering pairs over Wikipedia context paragraphs. We employ a single- hop question answering model, as opposed to a model fine-tuned on a multi-hop QA task, as the underlying assumption is that the model is able to answer the complex question, even without the background information. The BERT model fine- tuned on SQuAD achieves an accuracy of ~76% on the development set of HotpotQA both in the presence and absence of complete input context that is, in principle, needed to answer the question.
For each question, participants have to indicate: (I) whether they agree or disagree with the model predicted answer and (2) their level of confidence in their judgment, using a 5-point Likert scale. At the end of the study, we collect aggregate ratings from participants on their perception of the utility of background and highlights, if applicable, their confidence in the model, and their self- ceff-confidence in their own responses."
Procedures We conduct our user-study online with cooke- on Prolific." 4 Human-Al in- teraction for question answering is fairly ecologi- cally valid as lay users frequently interact with QA systems (search engines, chatbots) for information- seeking purposes. Participants are first presented with details about the study to obtain their consent to participate. They are then shown a tutorial in- troducing relevant terminology (e.g.. background, highlights), along with instructions on how to per- form the task. After the tutorial, participants are asked to perform the task for 10 questions, one at time. After completing the ten questions, par- ticipants are asked to complete the end survey to assess their overall perception of the system. Partic- ipants are also asked provide optional free-text feedback comments on the study. Finally, partic- ipants are asked to provide optional demographic information, such as age and gender.
3.1 Study design
Conditions. We design four conditions by varying the information that is provided to the participants:
o Without background and no highlights ("No back- ground");
o With germane background and no highlights ("With background");
With mix of germane and ungermane back- ground and no highlights ("With mixed back- ground");;
With germane background and supporting facts highlighted ("With background and highlights"). We conduct between-subjects study with partici- pants randomly assigned one of these four con- ditions. We present each participant with 10 ques- tions in the same condition. We select the questions shown to each participant such that the model pre- diction is correct on 7 questions and incorrect on the remaining 3. We hold this distribution fixed to ensure that model accuracy the observed exam- ples is close to the true model accuracy (~76%) and to avoid any undesirable effect of the observed accuracy on participants' reliance on the model. In the mixed (germane and ungermane) background condition, we provide a germane or ungermane background on each question uniformly at random, both for correct and incorrect predictions. We con- sider the same pool of questions across conditions and only sample each question once per condition.
We include two attention-check questions dur- ing the study, in each case asking participants indicate their agreement or confidence level in the previous question after they have clicked away.
Participants We recruited 100 participants, with each participant restricted to taking the study only once. Participation was restricted to US partic- ipants, fluent in English. Out of the 96 partici- pants who completed the study, we discarded the re- sponses from a single participant who failed both at- tention checks. The study took median time of ~ minutes to complete. Each participant was compen- sated US$2.25 (at an average rate of US$1 (5$$1)/) 42% of the participants self-identified as women, 54% as men, 2% as non-binarr/nonncoofomiii and 0% as any other gender identity. 22% of par- ticipants were between the ages of 18-25, 43% be- tween 25-10, 28% between 10-60 and 6% over the age of 60.
For each question, we present the participants with the context paragraph from Wikipedia that contains the gold-standard answer and a model pre- dicted answer corresponding to a span of text in the context paragraph. Depending upon the condition they are assigned to, we additionally show the cor- responding (germane or ungermane) background information, with without highlights. 2
'See Appendix for details on the post-task survey. hhttp://poolific.co
"See Appendix E for the study interface details.

---

3.2 Measures
reliance on the model. Even when predictions are correct, high reliance without background is still concerning as indicates that users overly trust the model despite insufficient information. We note an example of this behavior in a user's feedback at
We measured several aspects of users' behavior during the study, including their agreement with both correct and incorrect model predictions and their confidence in their own judgments. what follows, we refer to the predictions made by the model as "predictions" and the decisions by the users to agree or disagree with that prediction as "judgments." Users' judgments are subsequently deemed "accurate" when they agree with correct predictions disagree with incorrect predictions and "inaccurate" when they agree with incorrect predictions or disagree with correct predictions (Ja- covi et al., 2021; Vasconcelos et al., 2023). Using this terminology, we define the following measures: Appropriate agreement: Percent agreement with correct predictions.
the end of the study: "Some questions asked for 2 things, like the lype of game for 2 games, but the article only has one game info in there. However, I made my decision based what information can get from the article, think most of the time, Al made the right prediction so I chose "certain" about the Al's decision."
RQ2: Does adding requisite background infor- mation allow users to calibrate reliance and con- fidence on model predictions? Comparing the users' rate of appropriate and inappropriate agree- ment with and without background (Figure (left)), we find that adding background information indeed helps combat over-reliance on incorrect predie- tions; users exhibit a significantly lower (p=0.01) rate of agreement on incorrect predictions in the with background condition (0.47 0.04) than the no background condition (0.61 +-044). Background in- formation does not affect appropriate reliance; that is, the rate of agreement correct predictions is the same with and without background 0..0). Comparing users' accuracy in no background and with background conditions, we find that the user accuracy in detecting correct vs. incorrect predictions is marginally higher with background (0.77 + 0.02) than without (0.73 + 0.02). This is a natural extension of the rate of appropriate and in- appropriate agreement as we observe close agree- ment rate for correct predictions but a much lower agreement rate for incorrect predictions in the with background condition, which results an overall higher accuracy. However, this effect is not signifi- cant (p-0.144 14), perhaps partly because only 30% of the examples user sees are incorrect.
Inappropriate agreement: Percent agreement with incorrect predictions (over-reliance).
Users' accuracy: Percent agreement with correct and disagreement with incorrect predictions. Users' confidence: Average confidence (on a scale of 1-5) in accurate or inaccurate judgments.
4 Results
RQI: How do users interact with the model pre- dictions in the absence of sufficient information to assess their correctness? As seen in Figure 2, we find that the users have a significantly higher agreement with correct model predictions (appro- priate agreement: 0.89 + 0.02) than incorrect model predictions (inappropriate agreement: 0.61 + 0.06). Users also exhibit significantly higher confidence in their accurate (4.08 0.08) as compared their inaccurate judgments (3.37 + 0.177. This indicates that users are able to calibrate their reliance and confidence, even in the absence of the background. Regardless, users' agreement rate is fairly high (over 80%) without background, with over 60% agreement rate for incorrect predictions.
In terms of the effect of background on users' confidence in their judgments (Figure 2 (right)). we observe that background information increases users' confidence in both their accurate judgments (from 4.05 + 0.06 to 4.49 + 0.04; 0=008) and inac- curate judgments (from 3.55 + 0.12 to 3.89 = 0.10; p=0.03). This reflects that although background helps calibrate reliance on the model, it also leads to overconfidence in inaccurate judgments: users exhibit higher confidence in their inaccurate judg- ments with the background than without. On the whole, users' confidence in their judgments is fairly
Users' confidence rate is also substantially higher than the neutral score of (around 3.91) without background, with over 3.37 confidence even when their judgments are inaccurate. This highlights that users frequently rely on incorrect predictions, even when they have insufficient in- formation to assess the prediction, indicating over-
"To account for multiple testing. we perform Benjamini- Hochberg correction (Benjamini and Hochberg, 1995) and report "significance" with a false discovery rate of 0.05 (Hu ct al., 2010), yielding a significance threshold p < 0.01.

---

happropriate agreament Apprapriat agreement
judgment
judgm ent
p-0.7
pm0.08
PEP
p-o 03
R=0.0 0
p-0.0"
pm0.00
P-0.0"
0.8
User Confidence
4.5
0.6
4.0
0.4
3.5
0.2
0.0
3.0
No background
With background
No background
With background
Conditions
Conditions
Figure User agreement rate with model predictions when the model incorrect vs. correct (left) and user confidence in their own judgments when the user judgment is accurate inaccurate (right). The graphs show mean and standard error of agreement/confieecce in with/withoul background conditions. The rate of agreement is higher for correct predictions (appropriate agreement) than for incorrect predictions (inappropriate agreement), both with and without background. Users exhibit higher confidence in their accurate judgments than in inaccurate judgments. However, the rate inappropriate agreement is fairly high (0.6), even without background (RQI). Adding background reduces users' over-reliance incorrect predictions. However, adding background also increases users' overconfidence in their inaccurate judgments (RQ2).
calibrated: users are significantly (p=0.0) more con- fident their accurate judgments (4.28 + 0.04) than their inaccurate judgments (3.71 = 0.08). This is true regardless of background explanations. How- ever, the jump in confidence between inaccurate and accurate judgments is higher with background explanations (Cohen's d: 0.70) than without (Co- hen's 's d: 0.42).
with ungermane background (0.60 however, this difference is not significant (p=0.52). This leads to an overall significantly higher (p=0.01) ac- curacy the examples with germane background (0.78 + 0.04) than the examples with ungermane background (0.65 + 0.04).
We also observe that even when the background shown to users is mix of germane and ungermane backgrounds, users achieve comparable accuracy (0. 0.002 p=0988) with a germane background in the with background condition. More discussion on these comparisons is included Appendix C.
RQ3: Are users able to calibrate their reliance and confidence even when not all background pro- vided is perfect? From RQ2, we find that back- ground helps users calibrate their reliance on model predictions. We further test whether users are able calibrate their reliance and confidence in model predictions even when the background is only use- ful in a subset of examples shown to the users. Comparing the users' rate of appropriate and inap- propriate agreement with germane vs. ungermane background in the mixed background condition (Figure 3 (left)), we find that the rate of appropri- ate agreement is significantly higher (p=0.0) when shown germane background, which is top- ical and useful, (0.93 + 0.03) compared to when users are shown ungermane background, which is on topic, but not directly useful (0.74 + 0.05). Fur- ther, the rate of inappropriate agreement also lower with germane background (0.53 + 0.08) than
In terms of the effect of the relevance of back- ground on users' confidence in their judgments (Figure 3 (right)), we observe that similar to the background condition, users exhibit lower con- fidence in both their accurate and inaccurate judg- ments when shown ungermane backgrounds as compared to when shown germane backgrounds.
All together, these results indicate that users in- deed pay attention to the information provided in the background, which is reflected in better calibra- tion in model predictions when the background is germane and useful. Despite this, users still exhibit overconfidence in their inaccurate judgments for examples with germane backgrounds, even though the background points out the inaccuracies in the judgments. We conjecture that this is possibly be- cause identifying (and possibly discarding) unger-
indicates significance after Benjamini- Hochhirr- multiple-testing correction with a false discovery rate of 0.05.

---

nnappoopiiat wwwpenattttgggggnooooo agreement
Appropriate agreement
judgment Accurate judgment
p=0.0
pm0.01
0255
pm0.0
p=0.0"
pm0.5
2-0
p=0.03
1,0
5.0
0.8
4.5
User Confidence
0.6
4.0
0.4
3.5
3.0
0.0
2.5
Ungermane background
Germane background
Ungermane background Germane background Mixed background condition
Mixed background condition
Figure 3: User agreement rate (mean and standard error) with correct vs. incorrect model predictions (left) and user confidence (mean and standard error) in their own accurate vs. inaccurate judgments (right) in the mixed (germane ingermane) background condition examples with ungermane background vs. germane background. The rate of appropriate agreement is higher when users are shown germane background as compared to when they are shown an ungermane background. Users' overconfidence in inacourate judgments is marginally higher when provided with germane background compared to an ungermane background (RQ3), similar to no background condition in RQ2."
mane background is easier as compared to verify- ing reasoning emrrrs in model predictions.
leaves users prone to overconfidence in their inac- curate judgments.
RQ4: Does highlighting important parts of con- text and background improve users' reliance and confidence calibration? From RQ2, we find that background indeed helps in combating over- reliance on model predictions but also Icads to over-confidence in the users, even in inaccurate judgments. We test whether highlighting support- ing facts helps users better utilize the information present in the background. To this end, compare the conditions with background (no highlights) and with background and highlights (Figure 4). We find that adding highlights to the context and back- ground does not reduce users' over-reliance on in- correct model prediction (0.46 + 0.06) over simply adding the background information (0.47 + 0.04). Further, highlighting relevant parts of the con- text and background does not alleviate the issue of over-confidence stemming from the presence of background information: users' confidence their incorrect judgments similarly high the condi- tion with background and highlights (3.86 = 0.15) as in the condition with background, no high- lights (3.89 + 0.), both of which are much higher (p=0.03) than the condition without background (3.55 + 0.12).
Subjectivity in User Responses One concern with our analysis is the potential effect of partic- ipants' subjectivity in their agreement and confi- dence responses. To control for this, we fit a lin- ear mixed-effect model, treating participants as a random factor for each of the aforementioned ef- fects. In each mixed-effect model, consider the measure of interest (that is, agreement, accuracy, or confidence) the dependent variable and the treatment (presence of background, highlight, etc.) as the independent variable, along with the partic- ipant ID as a random effect. We find that even after controlling for the random effect of the varia- tion among participants, the effect of background or highlights on users' reliance and confidence mains discussed.
5 Related Work
Despite the promise of AI systems to assist hu- mans decision-making tasks (Kamar et al., 2012: Kamar, 2016), recent studies show that human-AT collaboration exhibits persistent failure mode: humans tend to over-rely Al assistance rather than using their own insights and reasoning when performing tasks (Chen al., 2023). This over- reliance on AT systems can lead to the acceptance of incorrect suggestions (Bansal et al., 2021: Bus- sone et al., 2015; Jacobs et al., 2021: Lai and Tan,
In summary, we find that highlighting relevant parts of the background does not enable users to critically assess incorrect model predictions and

---

happropriat are agreament Apprapñiae agreement
judgment
judgm
P-0.22
pm0.23
P
R=0.0
P-0.0"
pm0.00
p-0.0"
0.8
User Confidence
4.5
4.0
f
0.4
3.5
0.2
0.0
3.0
With background
With background and highlights
With background
With background highlights
Conditions
Conditions
Figure 4: User agreement rate (mean and standard error) with correct vs. incorrect model predictions (left) and user confidence (mean and standard error) in their own accurate vs. inaccurate judgments (right) in the conditions with background, no highlight and the conditions with background, supporting facts highlighted. The rate of agreement is higher for correct model predictions (appropriate agreement) than incorrect model predictions (inappropriate agreement), both with and without highlights. However, highlights do not help alleviate users' over-reliance incorrect model prediction or overconfidence in their inaccurate judgments (RQ4).
2019), phenomenon that is of particular concern in high-stake domains where the continued use of AI systems is prevalent (Bussone et al., 2015; Ja- cobs et al., 2021: Lai and Tan, 2019).
nize the accuracy of model predictions. Nonetheless, the concept of explainability is fre- quently framed as providing "information neces- sary for the underlying model to make the correct prediction" which may not inherently align with "information sufficient for users assess the model prediction." This misalignment particularly rel- evant in contemporary systems, powered by large language models pre- trained on extensive datasets. These models can extrapolate knowledge beyond their immediate in- put context, drawing on implicit factual or com- monsense knowledge (Petroni et al., 2019; Jiang et al., 2020) and employing shortcuts to provide accurate answers (Min al., 2019; Chen and Dur- rett, 2019; Trivedi et al.. 2020). While this ability enhances the performance QA systems, intro- duces complexities in the realm of explainability and user interaction with AI decisions.
To address the issue of over-reliance, recent ef- forts have focused on the development of explain- able Al methods, with the aim of enhancing users' comprehension of Al decisions and, in turn, pro- moting more appropriate reliance (Bussone al., 2015). However, an extensive body of human- centered evaluations of current explainability meth- ods has revealed that explanations can inadvertently increase trust in AT models, exacerbating over- reliance (Bansal et al., 2021; Buçinca et al., 2020; Papenmeier et al., 2019; Schemmer et al., 2022; Jacobs et al., 2021; Zhang et al., 2020; Buçinca et al., 2021), particularly among non-expert users (Gaube et al., 2022; Schaffer et al., 2019).
(QA)
Recent research in the field of explainable NLP has advocated for explanations that prioritize users' decision-making (Fok and Weld, 2023). Fok and Weld (2023) extensively review the current liter- ature on the role of explanations in supporting Auman-AT decision-making, consolidating their findings into compelling argument that explana- tions most valuable when they empower users with the essential information required to critically evaluate the correctness of model predictions. This growing body of work underscores the pivotal func- tion of explanations in facilitating users to scruti-
Our work extends the landscape of explainability and human-Al maaiing by moving beyond model explanations and exploring the implications of providing users with relevant background in- formation as a way to enhance verifiability and foster appropriate reliance on AI, in the context of question-answering
Discussion and Conclusion
Large general- language models, such as the GPT family of models (Brown et al., 2020;

---

OpenAI, 2023), LaMDA (Thoppilan et al., 2022), PaLM (Chowdhery et al., 2022; Anil et al., 2023), and others, have propagated into information- seeking workflows of a general audience. A vast host of existing and ongoing work NLP examines the deficiencies of these language models, ranging from hallucinated generations (Bang et al., 2023; Ji et al., 2023), lack of transparency (Weld and Bansal, 2019; Lipton, 2018), reasoning gaps (Bang al., 2023; Press et al., 2022), and more. How- ever, close examination of the other piece of the puzzle- the user-is fairly sparse. Literature in Explainable NLP considers how NLP systems, and their underlying reasoning process, can be made available to the user to allow for better, more in- formed use of these systems and their predictions (Ribeiro et al., 2016; Wang and Yin, 2021). Extend- ing this view of explainability, we considered the question of aiding users in their decision-makee process, not by explaining the model prediction but by providing them with relevant pieces of in- formation to assess the prediction. As argued by the contemporary work by Fok and Weld (2023), explanations in the form of the model's internal reasoning are rarely useful for supporting human decision-making (Bansal et al., 2021; Buçinca et al., 2021; Zhang et al., 2020). Instead, explanations should aim to help the users assess the model pre- diction. Our study takes a step in this direction by analyzing users' trust and reliance on QA models by providing information external to the model's input to enable verification of model predictions.
sess the correctness of the predictions (that is, with- out the relevant background). This indicates un- warranted trust (Jacovi et al., 2021) in the model to some extent (RQ1). We found that although background information did not affect reliance on correct model predictions, it significantly re- duced over-reliance on incorrect model predictions (RQ2). This indicates that even though users trust the model without sufficient information, they are able to catch model errors better when provided with the relevant information. We further find that users are able to calibrate their reliance on model prediction even when the background is sometimes, but not always, perfect (RQ3).
On the flip side, our study revealed that the addi- tion of background also increased users' confidence in their judgments of the model predictions, both those that are accurate and, unfortunately, those that are inaccurate. This especially concerning as even if users are better at the task when the background information is available to them, uncal- ibrated confidence might be detrimental in critical decision massionnmaking tasks. Lastly, we found that sur- facing relevant pieces to case users' perusal of the background is not sufficient to alleviate overconfi- dence stemming from the background, even when the background and highlights pointed out the in- accuracies users' judgments (RQ4).
Our work highlights the utility and pitfalls of adding more information with model predictions. We studied user interaction with such insufficient background information in a Wizard-of-OO experi- mental setting. More work is needed to study how such information can be gathered generated auto- matically and made available to the user. Research in NLP frequently adopts the reverse strategy- building the systems first, followed by testing if the systems are useful to the end-users. Our work aims to put humans front and center, studying their interactions with model predictions to inform the gaps in Al-assisted decision-making
We conducted our study in a multi-hop question- answering setup. We designed our study such that some part of the information required to perform the reasoning is missing. We studied how users rely on model predictions in the absence of such in- formation and how the presence of background affects their reliance and confidence in the QA systems. This background information, although not necessarily faithful to the model's reasoning process, may still be helpful to users in assess- ing the correctness of model predictions (Lipton, 2018). Such background explanations can take many implict information encoded in the pre-trained models used for reasoning or external information required to fill in knowledge gaps.
Our work highlights the issue of overconfidence stemming from more information, even when that information provides users with evidence to make accurate judgments. This indicates that explana- tions alone are not sufficient garner appropri- ate trust and reliance. More efforts are needed educate the lay audience to inspect explanations critically and diligently.
Our study revealed that users' reliance on model predictions is fairly high (over 80%), much higher than the accuracy of the underlying QA system (76%), even without sufficient information to as-

---

Limitations
Acknowledgements
In this work, we conduct a Wizard-of-Oo experi- ment to study users' reliance and confidence in QA predictions in the lack of sufficient information and the effects of adding background information on their interaction with QA predictions. Wizard-of- Oz experimental setting allows us to control the amount of information and possible noise in the information provided to the user. But, this comes at a costtconstaaining our study to simpler settings and model. User interaction with NLP systems is increasingly shifting to long-form question an- swering. We do not expect our findings to directly translate to this new paradigm, but we hope that our setting and findings will guide future exploration of these problems in the paradigm of long-form, open-ended question answering.
We would like to thank Chen Zhao, Simone Stumpf, the anonymous reviewers, and members of the CLIP lab at UMD for their constructive feedback. This work was funded in part by U.S. Army Grant No. 191
References
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Pas- sos, Siamak Shakeri, Emanuel Taropa, Paige Bai- ley, Z. Chen, Eric Chu. Jonathan H. Clark, Lau- rent EI Shafey, Yanping Huang, Kathleen S. Meier- Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gus- tavo Hern" andez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Michael Brooks, Michele Catasta, Yongzhou Cheng, Colin Cherry. Christo- pher A. Choquette-Chooe Aakanksha Chowdhery, C Crépy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, M. C. D D'iaz, Nan Du, Ethann Dyer, Vladimir Feinherg, Fan Feng. Vlad Fienber, Markus Freitag, Xavier García, Sebastian Gehrmann, Lucas González, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, An Ren Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe It- tycheriah, Matthew Jagielski, Wen Hao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Mu-Li Li, Wei Li, Yaguang Li, Jian Li, Hyeontaek Lim, Han Lin, Zhong-Zhong Liu, Frederick Liu, Marcello Mag- gioni. Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alexandra Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Marie Shelby, Ambrose Slone, Daniel Smilkov, David R. So. Daniela Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Ki- ran Vodraballi, Xuezhi Wang, Pidong Wang. Zirui Wang, Tao Wang. John Wieting, Yuhuai Wu, Kelvin Xu, Yu Xu, Lin Wu Xue, Pengcheng Yin, Jiahui Yu, Qiaoling Zhang, Steven Zheng, Ce Zheng, Wei Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu, 2023. PaLM 2 technical report.
Our study relies on careful construction of background using a multi-hop question answering dataset (HotporQA) that aligns with our assump- tions: the question should require multiple pieces of evidence to reason the answer and the provided supporting documents should be sufficient to an- swer the question. The HotpotQA dataset had been gathered using human effort and is likely also prone to human errors, in the same vein as the humans interacting with the QA system in our study. This also makes our findings prone to the eccentricities or noise in the original dataset. We ensure insofar as possible that our study design and resulting ex- amples align with our research questions. We posit that the constraints imposed in our study, such as us- ing the same set of examples across conditions and ensuring consistency of answers with and without background, will possibly balance out any potential issues in the underlying data.
Ethics Statement
The study conducted in this work was approved by the University of Maryland Institutional Review Board. We recorded minimal demographic infor- mation in our survey (age and gender). The de- mographic information was self-identified by the participants, was completely optional, and was not used to determine participation in the study. We only used the demographic information to gauge the participation distribution and did not perform any estimation of participants' demographic prop- erties. We obtained informed consent from partici- pants at the start of the study (see Appendix E for details on the consent form).
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji. Tiezheng Yu, Willy Chung, et al. 2023. mul- titask, multilingual, multimodal evaluation of Chat- GPT on reasoning, hallucination, and interactivity. arXiv preprint 2XXx:23302.00223
Gagan Bansal, Besmira Nushi, Eie Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019. Be- yond accuracy: The role of mental models in human- AI team performance. Proceedings of the AAAI Con-

---

ference on Human Computation and Crowdsourcing, 7(1)2-11.
human intuition on reliance in human-Al AI decision- making with explanations. Proc. ACM Hum. Comm-CCmmput. Interact. 7(CSCW2).
Gagan Bansal, Tongshuang Wu, Joyee Zhou, Ray- mond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? The effect of Al explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 211, New York, NY, USA. Association for Computing Machinery.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya. Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways.
Yoav Benjamini and Yosef Hochberg. 1995. Control- ling the false discovery rate: A practical and pow- erful approach to multiple testing, Journal of the Royal Statistical Society. Series B (Methodological), 57(1))289-300
Tom Brown, Benjamin Mann, Nick Ryder, Melani Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shy am, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel H erbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Chris Winter, Clemens nd Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCa ndlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186. Minneapolis, Minnesota. Association for Computational Linguistics.
Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable Al systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20, page
Shi Feng and Jordan Boyd-Graber. 2019. What can Al do for me? Evaluating machine learning interpreta- tions in cooperative play. In Pruceedings of the 24th International Conference on Intelligent User Inter- faces, IUI '19, page 229-222-222222222 229-239, New York, NY, USA. Association for Computing Machinery.
New York, NY, USA. Association for Computing Machinery.
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021. To trust or to think: Cognitive fore- ing functions can reduce overreliance on Al in Al- assisted decision decision-making. Proceedings ACM Human- Computer Interaction, 5(CSCWI).
Raymond Fok and Daniel S. Weld. 2023. In search of verifiability: Explanations rarely enable complemen- tary performance in Al-advised decision making.
Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. 2015 International Conference on Healthcare Infor- matics, pages 169.
Susanne Gaube, Harini Suresh, Martina Raue, Alexan- der Merritt, Seth J. Berkowitz, Eva Lermer, Joseph F. Coughlin, John V. Guuag, Errol Colak, and Marzyeh Ghassemi. 2022. Do as Al say: Susceptibility in deployment of clinical decision-aids. NPJ Digital Medicine, (2021,4).
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 4026-4032. Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Ben Green and Yiling Chen. 2019. Disparate interac- tions: An algorithm-in-the-lnn analysis of fairness in risk assessments. In Proceedings of the Confer- ence on Fairness, Accountability and Transparency. FAceT '19, page 90-99, New York, NY, USA. Asso- ciation for Computing Machinery.
Valerie Chen, Q Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal. 2023. Understanding the role of
Jeffrey Heer. 2019. Agency plus automation: Design- ing artificial intelligence into interactive systems.

---

Proceedings of the National Academy of Sciences, 11616)11444-150.
Accountability, and Transparency, FAccT'19 page 29-38, New York, NY, USA. Association for Com- puting Machinery.
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explain- able AI: Challenges and prospects, arXiv preprint arXiv:1812.0460
7achary C. Lipton. 2018. The mythos of model inter- pretability: In machine learning, the concept of in- terpretability is both important and slippery. Queue, 16(3):11-57.
James X Hu, Hongyu Zhao, and Harrison H 7hou. 2010.
False discovery rate control with groups. Journal of the American Statistical Association, 105(491):1215-
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannanch Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguisties. Florence, Italy. Association for Computational Linguistics.
1227.
Maia Jacobs, Melanie F. Pradier, Thomas H. McCoy Jr. Roy Perlis. Finale Joshi-Velez, and Krzysztof Z. Gajos. 2021. How machine learning recommenda- tions influence clinician treatment selections: the example of antidepressant selection. Translational Psychiatry 11 (2021).
pages
OpenAl. 2023. GPT-4 technical report. ArXiv, abs/2303.08774.
Alon Jacovi, Ana Marasovié, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust artificial intelli- gence: Prerequisites, causes and goals of human trust ai. In Proceedings of the 2021 ACM Conference on Fairness, Accountabilit, and Transparency, FAceT '21, page 624-635, New York, NY, USA, Association for Computing Machinery.
Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. 2019. How model accuracy and explanation fidelity influence user trust. ArXiv, abs/1907. 126.12.52.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55(1)).
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguisties, 8:423-438
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models, arXiv preprint arXiv:2210.03350.
Eee Kamar. 2016 Directions in hybrid intelligence: Complementing Al systems with human intelligence. In Proceedings of the Twenty-Fifit International Joint Conference on Artificial Intelligence, IJCAP 16, page 1070-4073 AAAI Press.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguisties.
Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large- scale crowdsoureing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems- Volume 1, AAMAS 12. page 464, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16. page 1135-11444 New York, NY, USA. Association for Computing Machinery.
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wort- man Vaughan. 2020. Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference Human Factors in Com- puting Systems, CHI '20, page 1-14, New York, NY, USA. Association for Computing Machinery.
James Schaffer, John O'Donovan, James Michaelis, Adrienne Raglin, and Tobias Höllerer. 2019. I can do better than your AI: Expertise and explanations. Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI '19, page 240-251. 251, New York. NY, USA. Association for Computing Machinery.
Vivian Lai and Chenhao Tan. 2019. On human predic- tions with explanations and predictions of machine learning models: case study on deception detec- tion. In Proceedings of the Conference on Fairness,
Maximilian Schemmer, Niklas Kühl, Carina Benz, and Gerhard Satzger. 2022. On the influence explain- able Al on automation bias. ArXiv, 04./2204408859,

---

Romal Thoppilan, Daniel De Freitas. Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung- Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pick- ett, Kathleen S. Meier-Hellstem, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Soraker, Ben Zevenbergen, Vin- odkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson. Alejandra Molina, Erin Hoffman- John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas Claire Cui, Mar- ian Croak, Ed Huai hsin Chi, and Quoe Le. 2022. LaMDA: Language models for dialog applications. ArXiv, abs/2201 .08239.
Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bel- lamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, T "20, page 295-305, New York, NY, USA. Association for Computing Machinery.
A Data and Preprocessing
As described in Section 3, we use BERT large model fine-tuned on the SQuAD dataset as our QA model. We use the HotporQA dataset (Yang et al., 2018), which has 90447 and 7405 questions in the training and validation split, respectively, select examples for our study. We select examples from the training split of the dataset, allowing a larger candidate pool of examples that meet our cri- teria. This should not be a concern as the fine-tuned SQuAD model used in our work is not trained on the HotporQA dataset. We remove question with yes and no type answers, leaving with candi- date pool size of 84966. Further, we only select examples for which the model answer is same with and without background. This leaves us with a candidate pool of 66772 examples.
Harsh Trivedi, Niramjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2020. Is multihop QA in DiRe condition? measuring and reducing discon- nected reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 8846-8863, Online. As- sociation for Computational Linguistics.
Helena Vasconcelos, Matthew Jörke, Madeleine Grunde- McLaughhin, Tobias Gerstenberg. Michael S. Bern- stein, and Ranjay Krishna. 2023. Explanations reduce overreliance on Al systems during decision-making. Proc. ACM Humm-CCmput.. Inter- act. 7CCCCW1)
Lastly, we sample examples where the model confidence is between the range [0.55, 0.65]. The intuition behind this is that if the model confidence is too low, then the model can possibly refrain from answering. On the other hand, if the model confi- dence is fairly high, we might not need human- in-the-loop to assess the model prediction. This ensures that there is a scope of meaningful collabo- ration between the human and the model, with their strengths complimenting each other (Heer, 2019; Zhang et al., 2020; Feng and Boyd-Graber, 2019). The threshold is chosen empirically such that the ac- curacy of the model on the selected range matches the aggregate accuracy of the model. This leaves us with a final candidate pool of 6212 examples.
Xinru Wang and Ming Yin. 2021. Are explamations helpful? A comparative study of the effects of ex- planations in Al-assisted decision-making. In 26th International Conference on Intelligent User Inter- faces, IUI 22,, page 318-328, New York, NY, USA. Association for Computing Machinery.
Daniel S. Weld and Gagan Bansal. 2019. The challenge of crafting intelligible intelligence. Commun. ACM, 62(6):70-79.
Kaige Xie, Sarah Wiegreffe, and Mark Riedl. 2022. Cal- ibrating trust of multi-hop question answering sys- tems with decompositional probes. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2888 2902, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
B Subjective Assessment
In the post-task survey, we collect the following self-reported subjective measures to study users' overall perception of the model (Hoffman et al., 2018). For each measure, users are shown the cor- responding statement and asked to rate each on a five-point Likert scale from strongly disagree strongly agree:
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop -hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages Brussels, Belgium. Association for Com- putational Linguistics.
Usefulness of highlights: "Thh highlights were useful. I feel that highlights helped in determin- ing whether the model predictions were correct." "

---

Question
True
Input context
Germane beciground
Ungermane background
Predicted an:
swer
Which song
The Groop (US band). The Groop were harmory based paychadalic pap and soul va- cal quartat from the USA, active at the and the 1380s and releasing one self-ttled al- burn.
Midnight Cowboy. Midright Cawboy
The Last of His Tribe. The Last His Tribe is a 1992 Iim starring Jon Veight anthropoingist Altedd Krosber who befrended Ishi, played by Graham Greene. Harry Hook diracted the lshi was tnought to be the last of the Yahi peo- pic.
from sel-
1969 American drama fimm based the 1965 novel of the same name by James Leo Her herllly The film was written by Wardo Salt, directed John Schicsinger, and stars Jon Veight alongside Dustin Hatfmani Notable smaller roles are filed by Syluaa Milss, John McGiver, Brenda Vaccaro, Bob Balaben, Jenniler Selt Hughos.
tited albu
Famous
was featured
Myllf
starring Jon
1965
Famous
'A Famous Mytl!' was in- eluded on the contemporary Midright Cowboy film soundrrack)
Their
Voight and wri:
Myth
ten by Waido
Salt?
and
Special Division. The Special Division a
The Virginia Sterilization Act of 1924. The Virginia Sterilzation Act 1924 greatly inflanced developmont of
The Special Division Independent upheld case that de- cided which act was Conati- tutional?
Morrison v. Olson. Morrisnn v. Olson, 487 (1988), United States federal court case which the Supreme Court of the
division of the United States Court of Appeals for the District of Columbia Ciruut. 28 U.S.C. C. 49 (1962 ed., Supp. V) (Title Vi of the Elhics Govermmet Act. consists three cirput court judgas justioss appointed Justice
century. The wss besed model legislation written by Hanyy H. Lsughin and challenged by the Supremn: Court decision of Buck v. Bell. The Supreme Court upheld the law. consequentially, proving that it was consa- tutinaal and making model law for steri- ization
gerics the
Counsel Act
Independent Counsel Act
Untted States decided that the Indopendent Counsel Act was constiutional
Chisf United States. The judges appointed for 2-yaar terms, with any vacancy being filed only for the remeinder of the 2-year period. lis constiutionallyy was uphald in Mori
rison Olsoni
Tatwancso Chinesa romance tim directed and co-written by Doze Nu.. It stars Zhao Wel, Shu CV, Mank Chao,
Love (2012 film). Love is
Lorene Ren. Loreno Ren (bom November 1986), previously known Kirston Ren, a Tatwanssa actress, model and singer, Her surname some- times spelled as Jen. She is the younger sister of Taiwanese girl group S mem- ber Selina Jen. Ren graduatedd from Na- tional Talwan Normal Universtty with a bachelor's degree home devalopment and family studies.
Love which wanese trees model?
stars Tai-
Oi. Lin Known by her stage name Shu OI, is a Tail wanesa actross and model, Shh has also beon credited as Hau Chi and Shu Kal (Cantorase pronuncation of "Shu CI"). She among the highest paid actrasses in Chiia.
1976), better
Shu
Juan, Eddle Peng, Amber Kuo, byy Chen and Doze Niu "Love" premiered the Panorama seclion the cand Berin In- tornational Film Festival. ensombie cast, with the storios rovealled be interwaven as the plot prograsses.
Shu
film features
hu-
Prakhar Gupta, Prakhar Gupta is Incian journalist foroign attairs analyst. Ho marilia
Swaralya (magazine). Swarayya is monthly print magazine and online daily. was weekdy magezine founded in 1356 by Khasa Subba Hao with the patranage of regopaischar one Swatantra Party,
c. Relegopalachari, Chakravarti Ra- jegopsiachari BR (10 December 1678 - 25 Decembor 1972), popularly known
For which piatetorm did Prakhar
writtan curty the indian Occan Rogion. Toritooial disputes the South China Saa, Nuclear non- prolitoration, and India China Pakssaan rola. foon. He has wrltten for The Diplomat, Youth Awaaz, Frustrated Indian, Swaratya and South Asian Voices, online platform for strategic analyals on South Asia hosted by The Stimson Conter
Swaratys
Rajai C.R. also known Mootharig- Scholar Emon- tus), was an Incian siatesman, writer. lawyer, and indopondonce activist. Ha jagepeiacheri was the last Govemor- Genoral of India, as India bocame ra- public in 1950. He was also the only indian-born Governo- Generaal, as all pre- vious holders of the post were Brissh tonals.
Gupta fai-
Sout Asian
founders regular contributor the magazine the form his "Dear Reader column. Minco Masari, R. Venkatraman, R.K. Larman are some notable personailles who contributed to the magazine. Alter Rajay's death 1872, the magazina bagan to decline and evantusly dosed 1980.
nar Rajaj (Hajaj.
Voices
lowing Rajalls
death?
What actor. who first peared the Slackadder
The Foretelling,
'The Foretelling lis the first
spisade the altoom Thh Black Adder. the first series of the long running somedy pro: gramme Blackadder! It marks Rowan Alkn son's dabut
Tim Molnnerny. Tim Molnnarny (bom 18 Rowan Atkinson. Rowan Sebastian Alkinson (barn January 1955) an English actor, comedian and wrtter He played the tilla roles in the aitooms Blackadder (1963-19699 and Mr. Bean (1980-1995), and in the 1im series Johnny English (2003-2018).
Tim Melin
September 1956) is an English actor. Ho is known for his many roles on television and stage. Early in his career he featured as Lord Percy Peray and Captsin Daring the "Black: adder" scries.
nermy
specde "The
Forctalling
Rowan AIKIS
character Fomund Black:
olso played the role Captain
son
addar, and is the first appearance of the ra: curring characters Barcrick (Tony Rebirson) and Peray (Tim Mohnnemy), comedy actor Peter Cook guest stars as king Richard IIl.
Darling in the same series?
Table 1: Examples: question, the true and predicted answers, along with the input context and the supporting (Germane) background with the supporting facts highlighted, and a distractor (Ungermane) background.
a Usefulness of background: "The background in- formation was useful. I feel that background helped in determining whether the model predic- tions were correct."
measures across different conditions. We find that the differences in user ratings with and without background, both with and without highlights, are non- non-significant. We find that adding background non-significatt
Confidence in the model: "I am confident in the model, including the predictions, highlights, and background."
increases users' confidence in their own judgment (from 3.74 + 0.16 to 4.10 + 0.13), consistent with users' aggregate confidence over individual exam- ples (Figure 2 (right)). Despite the decrease in the rate of over-reliance on incorrect model predictions with background explanations (RQ1), users rate the satisfaction with the model marginally lower in the condition with background (2.91 0.15) than the condition without background (3. + 0.16). This might be due to the additional cognitive load re-
Self-confidence: "I am confident in my deci- sions."
Satisfaction with the model: "I would like to use the model for decision-making!"
Does users' perception of the model change with and without background, or with highlights? Figure 5 shows the user rating for the subjective

---

With background
No background
With background
With background
and highlights
p=0.74
pm0.17
p-0.36
p=o.74
p=10
p=088
p=0.78
f
Likert rating
Likert Likert rating
3-
background Ubiiity
confidincce
confidence
satisfaction with
highlights utility
background udility
confidence
confidence
satisfaction
Figure 5: Users' subjective rating of the system for the usefulness of highlights, background, their confidence in the model, self-confidence, and satisfaction with the model. Users rate self-confidence marginally higher in the condition with background than the condition without (left). However, users rate their satisfaction with the model in the condition with background slightly lower than withour. Users' satisfaction rating is slightly lower even after introducing highlights with the background (right), with a slightly higher rating of background utility in the condition with highlights than without. However, there are other discernible differences in ratings in the background condition with or without highlight.
quired consume the background information. hibit comparable rate of appropriate agreement and inappropriate agreement in both the cases. This dicates that the presence of ungermane background examples in the mixed background condition does
The differences users' ratings are much less prominent for adding highlights along with back- ground: users assign similar ratings for confidence in the model predictions and confidence in self- judgments with without highlights. This paral- lels findings on task performance, where we negligible improvements in adding highlights. Sur- prisingly, users rate the satisfaction with the model in the condition with highlight marginally lower (2.94 + 0.15) than the condition without highlights (3.10 + 0.16).
not affect users ability to calibrate their reliance on model predictions when they are shown useful background information.
Further, comparing the confidence in accurate and inaccurate judgments in the two cases (Fig- ure (right)), we find that users' exhibit slightly lower confidence in their judgments even on predic- tions that include germane background in the mixed background condition than the with background condition. Unfortunately, this effect is not in the desirable direction. The users' confidence in their accurate judgments is significantly lower (p 0.0) in the mixed background condition (4.12 0.09) than the with background condition (4.44 = 0.06).
Comparing Germane Background in the With Background vs. the Mixed Background Conditions
In RQ3 (Figure 3), we discuss the difference in users' reliance and confidence when they are shown germane background vs. ungermane background in the mixed background condition. We further com- pare whether and how users' reliance model pre- dictions and confidence in their judgments differ in the with background condition, when all examples they are shown have germane background, and the mixed background condition, when only half the examples shown have germane background. This allows us to see if users' interaction with relevant and useful background is affected by the presence of noise in the background information that is avail- able to them.
But, the drop in confidence in inaccurate judgments (from 3.92 + 0.13 to 3.70 + 0.22) across the two con- ditions is not significant (p 0.36).
D Comparing Ungermane Background with the No Background Condition
Extending the previous analysis, we further com- pare the difference in users' reliance and confi- dence on model predictions when they are shown ungermane background. which topical, but not useful (in the mixed background condition), vs. when they are shown no background at all (in the no background condition). In both of these cases,
As seen in Figure (left), we find that users ex-

---

users likely lack sufficient information assess the correctness of model prediction. This allows us to see whether our findings in RQ2 with respect to decrease in over-reliance and increase in over- confidence on the addition of background holds regardless of the utility of the background.
an injury related to the research, please contact the principal investigator.
After starting the study, the partici- pants are given the following instructions about the task:
Instructions
In this survey, you will be presented with 10 ques- tions and the respective answer predicted by an Al system based on the shown supporting docu- ment.
As seen in Figure 7(left), observe that users' have marginally higher rate of inappropriate agreement, and a significantly lower rate of appro- priate agreement in the cases when the background is ungermane, as compared to the no background condition. This indicates that users do much worse with ungermane background than no background at all. In fact, users have significantly lower accu- racy when they are shown ungermane background in the mixed background condition (0.65 0.04) as compared when they are shown no background at all (0.78 + 0.03).
You will also be shown background document (on the right) with additional information on the question and/or the supporting document. For each question, you will be asked to mark whether not you agree with the Al predicted answer.
The Al system is 70% accurate, that is, it predicts the correct answer in 70% cases.
Next, participants are given a walk-through of the task. Figure 8a shows the example tutorial screen. After the tutorial, participants are shown 10 tasks, one by onc. Figure 8b shows an example task screen for participant in the with background condition. The study includes two attention check questions interspersed between the 10 tasks:
Further, comparing confidence in the two cases (Figure 7(right)), we find that users also exhibit significantly lower confidence in both their accu- rate (from 4.44 + 0.06 3.67 + 0.15; p 0.0) and inaccurate (from 3.92 1.13 0. 0.0) judgments with ungermane background than no background at all.
Did you agree with the Al the previous an- swer? (Yes/No)
Study Interface
What was your confidence rating in the last ques- tion? (Very Uncertain/ Uncertain/ Neutral/ Cer- tain/ Very Certain)
Briefing and Consent At the start of the study, we brief the participants with the following details about the purpose of the study and obtain their consent to participate.
Lastly, participants are asked to fill in the post-task survey, shown in Figure 8c, followed by optional feedback and demographic information.
About the Study: The purpose of this study is to investigate ways to improve human and machine collaboration in decision decision-making tasks using ex- plainable automated reasoning.
Data and Confidentiality: The study collects min- imal demographic information, such age and gender. You can opt-out of answering demo- graphic questions. Crowdworker ID is the only potentially identifying information, which will immediately removed from the data after par- ticipants are paid. All data will be anonymized prior to public release and responses cannot be linked to individuals. Any potential loss of confi- dentiality will be minimized by storing data in secured password protected database. o Right to Withdraw And Questions: Participation in the study is completely voluntary. If you de- cide participate in this research, you may stop participating at any time. If you have questions, concerns, or complaints, or if you need to report

---

Inappropriate Appropriate
Inactrrate
Accurate judgment
p=0.0
P-0.6
P-0.36
9=0.0
pm0.05
p=0.00
p=0.0
1.0
5.0
4.5
User Agreementt
User Confidence
4.0
1
Germane background Germane background
Germane background Germane beckground (with background condition Imixed background condition)
(with background condition) (mixed background condition]
Figure 6: User agreement rate (mean and standard error) with correct incorrect model predictions (left) and user confidence (mean and standard error) in their own accurate vs. inaccurate judgments (right) when they are shown the germane background in the with background condition (i.e., all examples have germane background) vs. the mixed background condition (i.e., only half the examples have germane background). We see significant difference in the germane background in either background conditions, except users' confidence in their accurate judgments is lower in the mixed background condition, even on the examples that include the germane background.
Insppropriate ag agreement
Appropriste agreement
Inacturate judgment
Accurate Judgment
p=0.0"
P-0.0
0-0
P-0.8
0=0.03
1.0
5.0
0.8
4.5
User Agreement
User Confidence
0.6
4.0
3.5
2.5
No background (no background condition) (mixed background condition)
Ungermane background
No background (no background conditian)
Ungermane background nd Im beed background condition) Conditions
Canditions
Figure 7: User agreement rate (mean and standard error) with correct vs. incorrect model predictions (left) and user confidence (mean and standard error) in their own accurate vs. inaccurate judgments (right) without background (i.e., in the No background condition) vs. with ungermane background (i.e., in the mixed background condition). We see that users' have marginally higher rate of inappropriate agreement, and a significantly lower rate of appropriate agreement cases when background is ungermane, as compared to the no background condition. Further, users also have significantly lower confidence in both their accurate and inaccurate judgment with ungermane background than no background at all."

---

Tutorill
Who the cormander-in-chee of the United States Armed Forces?
The question answereot
20,
President of the United States
ment president of the Unitted Statesi
States (eomuSi
ltates. member the Demooratio
alaie and head of government of the United Stares America, The president directs the executive branch
4Pth
sam and mpprosotted Dalawaro the to 2008
fediral government and commenddee--ecieee United States Armed Farses.
Baok
Next
predicted
Barack Chbamal the prediction accuracy about large dataaee
(thet the predicted
answerr were coreec 70% of the questions.
Do you agrea with the predicted answor??
agreo. No, agree.
How contident are you about your answer?
Uncertain
Neutral
Certain
Stent Sumees
(a) Tutorial Screen. Participants are shown a tutorial example first. They are step-by-step taken through the different information that is shown in each example, such as the question, the supporting context, the background, and model predicted answer.
Tosk-11
101
Quection: Which song from a self-dttdd album was featured 1965 film, starning Jon Voight and written by Waldo Sal??
The Groop (UB band)
The Group harmony--eee baseed paycheeteic and and vecaal USA, teseodd-
The Last of His Thib Tribe
Voight
sull tited album. Their song & Famoua Mysii was intudded on thi
anthropologiet Altred Kroeber who befnended lehi, playedt Craham Greena, Harry Hook directed the film, lahi was thought
cantomporary Midhight Comboy soundtrack)
people.
The Al prodicted answer ia Famous Myth, Riccal, the All pradiction accuracy about 70% on the large datasot that we tested on (thtt the pradicted
questions.
Do you agrae with the Al predicted answer?
Yes, lagreo. No, do not agren.
How confidinnt are you about answer?
Very Uncertain
Uncertain
Neutral
Certain
Very Certain
(b) Screen. One example task with the question, supporting context (left) and background (right). Participants are shown the model predicted answer and asked to specify their agroement/dssggreeeen with the model's answer and indicate their level of confidence.
your agres disagree
The highlights useful feel that highighhts determining whether predictioon were correct.
Strongly Disagree
Disegree
Neutral
Agree
Strongly Agree
The background information was useful. feel that background helped in determining whether predictions correct.
Strongly Disagree
O
Disegree
C Neutral
Agree
Strongly Agree
con fident Al system, including the predictions, hightights and background.
Strongly Disagrao Disagraa
Noutral
Agrea
Strongly Agroa
aa con confident in decisiona.
Strondy Disagreo Disagreo
Neutral
Agreo
Strongly Agree
I !would like to use the All system for decision maiding.
Strongly Disagreo
Dieagreo
C Neutral
Agree
Strongly Agree
(c) Post-task Survey Screen, Participants are asked to provide aggregate ratings for highlights, background, confidence in self, confidence in the model, and satisfaction with the model.
Figure Study Interface Screens


Reviews:
[{"review_id": "hJZXPtS48Q", "reviewer": "Reviewer_bvZJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their comments and for highlighting the key strengths of our work. We really appreciate your time in reading through this work and providing feedback. Here are some thoughts on the points mentioned in your review. \n\nRe absence of methods that alleviate issues in QA system explanations (besides highlighting which does not work): We agree with the reviewer that our paper mainly highlights how providing background information required to assess QA predictions helps (decreasing over-reliance) and hurts (increasing overconfidence) users’ interaction with QA systems and does not provide a solution to alleviate this issue. Our work aims to encourage caution in treating explanations as a sure-fire fix to human-AI decision making and call for thinking more deeply about the shortcomings of solutions that are commonly sought after in literature (Fok and Weld, 2023) and motivating further research in potential fixes that better calibrate human reliance and confidence. \n\nRe variability in the interpretation of confidence scores: Yes, there might be some variations in the interpretation of the confidence scores between participants. We did not find anything of note regarding the interpretation of confidence score in the end user survey. \n\nTo assess the effect of participants’ subjectivity in interpreting the confidence score, we run a new analysis, where we fit a mixed effect model that considers the assigned participants as the random effect. We find that even after controlling for the random effect of the variation among participants, the effect of background on users’ reliance and confidence remains as discussed in the paper: background reduces over-reliance on incorrect AI predictions, but also increases overconfidence in incorrect judgments.  Further, we also calculate the intra class correlation statistics, which measures the similarity of the agreement/confidence ratings within a participant (random effect). The intra class correlation (ICC) lies between 0 and 1, where 1 indicates a perfect relationship within a participant. We find the intra class correlations to be 0.04 and 0.07, respectively, for the agreement and confidence. In our case, this reflects that there is large heterogeneity in the responses of a single participant, indicating that individual participant’s judgments vary based on the question and evidence. We will add this discussion in the final version of the paper.  \n\nRe model-generated explanations and long-form QA: We considered the trade-off between in-the-wild model generated explanations and wizard-of-oz explanations and argued that the latter allows a more controlled experimental setting for a first study. With wizard-of-oz experiments, we control the sufficiency of information available to the participants, which isn’t possible in model-generated explanations. \n\nSecondly, there isn’t yet a clear and definitive way to evaluate long-form answers or explanations, which might then conflate model and evaluation errors and human errors. Indeed, more work is needed to extend this kind of evaluation and studies to other settings. Extending this to more naturalistic settings is part of an ongoing future work. Our work here aims to motivate this line of investigation and tools, common in HCI literature, to the NLP community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper investigates a well-motivated, under-explored topic in the intersection of HCI and QA.\n\nThe user studies are well designed, and significance tests are conducted through each experiment. The authors also are thorough in reporting their crowd working setup for their user study.\n\nThis papers findings identify a core issue in existing QA systems, one that may motivate further work into designing effective explanations for real-world users.  ", "reasons_to_reject": "While this paper identifies issues in QA system explanations, they do not experiment with any methods that effectively alleviate this issue (outside of highlighting which did not work). They suggest possible small fixes (such as not providing evidence when its irrelevant); however these are still unsatisfactory long-term solutions and not implemented in this work.\n\nIt difficult to interpret the different options for confidence in the user study. I'd be curious to see data on how the users interpreted each confidence option, either in their post-task survey or analyzing how average confidence judgments differed user-to-user. In particular, I'm curious whether users associated \"neutral\" confidence more with 50% confidence in their judgment or with 75% confidence (which seems to be the intended interpretation from the authors).\n\nBiggest concern is the limited breadth of the study, both with respect to the types of models/explanations and types of questions. The authors do not explore results with LLMs and model generated explanations, which are far more common than systems that produce multiple retrieved evidence passages. The authors also only explain multihop questions which may not represent a realistic distribution of information-seeking questions from users.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work presents a study on user interactions with QA systems, specifically investigating the relationship between user confidence in a QA system's output and the amount of evidence shown the the user. The authors are particularly interested in multi-hop settings where the answer cannot be completely answered from the answer's context paragraph alone and require additional information in a background paragraph. They experiment with showing the users (1) the context alone, (2) the context and correct background, (3) the context and mixed correct/incorrect backgrounds, and (4) the correct background with supporting facts highlighted.\n\nIn each of these settings, the authors ask users (1) whether they agree with the QA system's output and (2) how confident they are in that judgment. The authors find that users, across all settings, tend to agree with correct outputs more often that incorrect ones. They also find that adding correct background information increases users' ability to identify correct/incorrect predictions; however, their confidence in all judgments (correct and erroneous) also increases. In the mixed correct/incorrect background settings, the authors observe a similar trend where confidence increases for all judgments when the context is correct/relevant. In their experiments with highlighting relevant information, the authors find that this does not improve overconfidence from users.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For clarity, maybe consider making the Y ranges the same for all figures."}, "extra_text": {}}, {"review_id": "RmbdQ8DmEF", "reviewer": "Reviewer_Dbbf", "summary": "", "questions": "Question A: How do you distinguish if the user is making the judgment depending on the model prediction or the background? (When there is background passage) It is likely that the user misread the background and make incorrect judgments. Is it possible to conduct extra experiments to find out about this?\nQuestion B: Different participants are recruited in different background settings. How do you ensure that the agreement / confidence differences between different settings come from design, not from the difference in participants themselves?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions. \n\nRe generalization to different settings: Given that this is the first study that explores the use of background information to enable verification of model prediction, we considered the trade-off between in-the-wild model generated explanations and wizard-of-oz explanations and argued that the latter allows a more controlled experimental setting. With wizard-of-oz experiments, we control the sufficiency of information available to the participants, which isn’t possible in model-generated explanations. \n\nSecondly, there isn’t yet a clear and definitive way to evaluate long-form answers or explanations, which might then conflate model and evaluation errors and human errors. Indeed, more work is needed to extend this kind of evaluation and studies to other settings. Extending this to more naturalistic settings is part of an ongoing future work. Our work here aims to motivate this line of investigation and tools, common in HCI literature, to the NLP community. \n\n(Question A) How do you distinguish if the user is making the judgment depending on the model prediction or the background: We do not directly control whether the user makes the judgment based on the model prediction or background. It would have been entirely possible that users entirely ignore the background, in which case we would have observed no differences in users’ reliance or confidence between the with and without background conditions. It is also possible that users entirely ignore the model prediction in the with background condition. Regardless, a decrease in the over-reliance on incorrect predictions still indicates the utility of background in helping users make better decisions, as  there are no significant differences in the time taken to complete the task with and without background. To further validate this, we can check whether the users are able to do the task as efficiently and accurately without model predictions altogether. Due to IRB constraints, we can not run this study within the rebuttal time frame, but we will add a discussion in the final paper.  \n\n(Question B) Different participants are recruited in different background settings. How do you ensure that the agreement / confidence differences between different settings come from design, not from the difference in participants themselves: We recruit a large set of participants (100) who are assigned randomly to the different conditions, so the expectation is that the significant differences observed across conditions come from the design.  To further ascertain this, we run a new analysis, where we fit a mixed effect model that considers the assigned participants as the random effect. We find that even after controlling for the random effect of the variation among participants, the effect of background on users’ reliance and confidence remains as discussed in the paper: background reduces over-reliance on incorrect AI predictions, but also increases overconfidence in incorrect judgments. Further, we calculate the intra class correlation statistics, which measure the similarity of the agreement/confidence ratings within a participant (random effect). The intra class correlation (ICC) lies between 0 and 1, where 1 indicates a perfect relationship within a cluster. We find the intra class correlation to be 0.04 and 0.07, respectively, for the agreement and confidence. In our case, this reflects that there is large heterogeneity in the responses of a single participant, indicating that individual participant’s judgments vary based on the question and evidence. We will add this discussion in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are solid: The design of various background settings are appropriate for the research questions. The experimental results are shown with mean, variance and p-value. The study comes with a tutorial and simple filtering mechanism.  \n2. Could be useful for future work: Few works have discussed how humans make decisions (judgements) about whether to trust a model prediction. The insights gained from this work could be used for better QA model design. This work could also inspire future work in this  particular direction.\n3. Clear writing: The experimental design is carefully laid out and easy to follow. The research questions are listed explicitly and explained clearly. All technical terms are well-explained. \n", "reasons_to_reject": "1.  Generalizability: The results in this paper may not generalize to different settings. The experiments are only conducted in one dataset, and only two-hop questions are considered. It is difficult to determine if the findings would hold across different domains, answer types (e.g. long-form QA vs. short factoid QA), and reasoning types.", "questions_to_authors": "Question A: How do you distinguish if the user is making the judgment depending on the model prediction or the background? (When there is background passage) It is likely that the user misread the background and make incorrect judgments. Is it possible to conduct extra experiments to find out about this?\nQuestion B: Different participants are recruited in different background settings. How do you ensure that the agreement / confidence differences between different settings come from design, not from the difference in participants themselves?\n", "paper_topic_and_main_contributions": "The authors conduct a wizard-of-oz experiment that investigates how sufficient or insufficient background information influences users’ reliance on QA systems and self-confidence. They restrict the focus to two-hop questions in HotPotQA, where two passages are used to answer the question, in order to control for with or without full background knowledge. They find that users tend to agree with model prediction even when they are wrong (over-reliance), and that having access to background information could mitigate that. However, access to background information also leads to over-confident in human judgements. Highlighting relevant sentences in the background information makes little difference. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ixx4F584xc", "reviewer": "Reviewer_7seo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions on our work. We really appreciate your feedback both on the study and the writing. \n\nRe the results being limited to results of only one model with a 76% accuracy: Our work aims to study the effect of the amount of information presented to the users on their reliance and confidence in the system, leading us to focus more on this axis of variation. We choose a model with 76% accuracy for this study as, in this case, the model accuracy is consistent both  with and without background explanation, which is one key desideratum in our study. Overall, as the users are not aware of the instances that the model is correct/incorrect on, the model accuracy only guides their aggregate behavior, at best. Since the accuracy is maintained across the conditions with and without background, our findings with respect to the differences in users’ reliance on correct/incorrect predictions across conditions would hold. \n \nRe the implications of our  findings: The key takeaway of our work is that aiding users in assessing model predictions is indeed useful in helping them better weed out incorrect predictions, but it comes at the cost of increased overconfidence in inaccurate judgments. This is relevant for current NLP systems, which, despite good performance, are known to be erroneous. The expectation is often that providing users with the necessary information to assess model predictions would be sufficient in combating model deficiencies. But as our work highlights, this is clearly not sufficient. Thus, more thought needs to be put into the best ways of aiding users in analyzing model predictions and evidence/explanations. We will incorporate the reviewer’s feedback and bring forth these points in the introduction and the discussion of the results. \n\nRe the absence of a separate related work section: we will add this in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Study of the effect of additional documents on the agreement and confidence of users\n- Study of the impact of including germane and ungermane background", "reasons_to_reject": "- The work is, in some way, limited to results of only one model with a 76% of accuracy. I think the study would benefit from including another model with lower accuracy. Thus, we could compare the results of users' behavior with different models' accuracy.\n\n- The objective of the study is clear, but the benefits/implications of your study in other works are unclear. I find some of them at the end of the conclusions but such a work should include more in the Introduction and along the paper. For example, the paper would benefit from some recommendations after the results of each experiment. Otherwise, it is a little bit hard to understand the impact of this work.\n\n- Although this work includes many references inserted across the document, I miss a related work section, besides what is included in the Introduction and Discussion Section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies the importance of additional knowledge of users when assessing the correctness of QA systems. The authors analyze the agreement and confidence of users, with respect to automatic systems, depending on the additional information provided to the users.\nThe main contribution is to evaluate how additional documents affect the confidence and agreement of users with respect to answers automatically extracted.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Section 2 contains more than the research questions. I think a more proper name would be \"Experiment design\" or something similar\n- you should introduce the list of Items from line 365 to line 372."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper uses a wizard-of-oz experiment using two-hop questions from HotpotQA, to study how people's reliance on QA model predictions change when they have access to (in-)sufficient context knowledge.

All reviewers agree that the study is interesting and well-motivated, and that the experiments are well-designed and well-executed. 

Main concerns involve:
1. Generalizability: Whether the findings generalize to other domains or QA tasks
2. Implications: The study flags a phenonmenon but did not provide a solution 

The authors sufficiently addressed these concerns in their rebuttals. I recommend accepting the papers to the Main track, and recommend the authors to revise the paper as they proposed in their rebuttals: add related work discussions, reflect on potential biases in human studies (e.g., whether they actually used the given information and whether their confidences are well-calibrated), and discuss the implications on futuer work.
Decision: Accept-Main