Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty (2024)

Abstract: As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3623–3643 August 11-16, 2024 ©2024 Association for Computational Linguistics
As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

heading: 1 Introduction
text: Natural language is becoming the default interface for humans to engage with artificial intelligence systems whether it be information seeking, summarization, or image captioning (Bommasani et al., 2021; Brown et al., 2020; Ouyang et al., 2022). As input, natural language serves as a rich and versatile medium, enabling users to articulate intricate tasks and inquiries effectively. As for output, natural language provides an opportunity for language models (LMs) to generate not only informative, but nuanced responses that better support the collaboration between humans and AI.
A pivotal aspect of fostering reliable human-AI interactions lies in the apt communication of model

heading: LM Expressions of Confidence Human Interpretations
text: uncertainties (Cai et al., 2019; Kizilcec, 2016; DeArteaga et al., 2020), typically defined as the probability assigned to a model’s prediction. Recent work in language generation reflects a shift towards using natural language as a means to convey model confidences (e.g., “I’m fairly confident it’s”, "According to Wikipedia it’s" Mielke et al., 2020; Lin et al., 2022; Zhou et al., 2023). Such features are known in the linguistics literature as epistemic markers, which serve to convey a speaker’s stance and commitment, thus supporting human communication and decision making (Babrow et al., 1998; Brashers et al., 2000; Tseng and Zhang, 2023). Since epistemic markers play an important role in human-human communication and decision making (Budescu et al., 1988; Windschitl and Wells, 1996; Druzdzel, 1989), we hypothesize that the use of these markers by LMs will also have an impact on human-AI interactions.
Our work begins with an examination of how LMs communicate uncertainties to end-users in realistic information seeking scenarios (§3). Specifically, we elicit responses from popular, publiclydeployed models including GPT, LLaMA-2, and Claude by prompting them to provide epistemic markers when answering multiple choice questions (Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023; Anthropic, 2022). Our analysis reveals that LMs are reluctant to share model uncertainties, de-
3623
spite errors in their generations. LMs can be explicitly prompted to use epistemic markers, but are more likely to generate expressions of certainty than uncertainty, despite an average 47% error rate among high confidence responses, (e.g., I’m confident the capital of Tanzania is Dar es Salaam. [Incorrect]).
We then investigate the behavioral responses of users towards model-generated epistemic markers (§4). While linguists and psychologists have long focused on the interpretation of epistemic markers by humans (Budescu et al., 1988; Windschitl and Wells, 1996; Wallsten et al., 1986), the pragmatic implications of the speakers of such markers being AI systems, combined with the known human overreliance on AI (Bussone et al., 2015; Jacobs et al., 2021; Bansal et al., 2021b; Buçinca et al., 2021), could drastically change their interpretation compared to human-spoken ones. Thus, we conduct several user studies to measure how individuals interpret and respond to uncertainties articulated by LMs in calibrated and miscalibrated settings (Figure 1). Our findings surprisingly indicate that users are heavily reliant on LM generated expressions of marked "I’m sure it’s...") and unmarked certainty (e.g., "The answer is..."). Subsequent experiments show that even minor miscalibrations in how a model uses epistemic markers can lead to long-term harms in human performance.
Lastly, given our findings on model overconfidence and human reliability of LM generations, we pinpoint the origins of model overconfidence (§5). We investigate model artifacts such as base models, instruction-tuned models, reward models, and human feedback datasets to isolate the origins of model confidence. Our investigation identifies the process of aligning models with human feedback (i.e., RLHF) as a key contributing factor and we uncover that human annotators are biased against expressions of uncertainty.
Together, our findings expose the shortcomings of how LMs currently use epistemic markers, outlines the risks that they pose on downstream users, and put forward mitigating solutions.

heading: 2 Epistemic Markers in Language Models
text: Our work focuses on the alignment between LM accuracy and LM-articulated epistemic markers as perceived by users. This is referred to as linguistic calibration (Mielke et al., 2020) which builds off of work in both linguistics and machine learning.
Linguists has extensively studied epistemic markers as ubiquitous linguistic features that signal speaker commitment and stance. These markers broadly fall into two categories: weakeners— expressions of uncertainty, and strengtheners— expressions of certainty (Lakoff, 1975; Hyland, 2005, 2014).
In machine learning, work has focused on improving model calibration (Jiang et al., 2021; Desai and Durrett, 2020; Jagannatha and Yu, 2020; Kamath et al., 2020; Kong et al., 2020) by calibrating the confidence value assigned by a model and model accuracy through a measure called ECE (Naeini et al., 2015). Recent work has focused explicitly on how pretraining (Hendrycks et al., 2019) and scaling (Srivastava et al., 2022; Chen et al., 2023) impacts the calibration of language models. Most relevant to our work is Dhuliawala et al. (2023)’s studies on how humans interpret numerical confidences in calibrated and miscalibrated settings. A key issue remains, as numeric confidence values are known to be challenging for users to interpret (Miller, 2019). Our work, in contrast, aims for a more comprehensive understanding of how humans interpret LM-generated verbal epistemic markers.
We begin by eliciting open-ended generations, a departure from prior methods which prompts models to produce a predefined set of confidence expressions either numerically (Kadavath et al., 2022; Tian et al., 2023; Liu et al., 2023; Xiong et al., 2024; Tanneru et al., 2023) or using an ordinal scale (Lin et al., 2022; Mielke et al., 2020). Instead, our strategy enables a qualitative bottom-up (Charmaz, 2006) approach towards understanding how LMs generate epistemic markers, mimicking how real-world users might engage with LMs.

heading: 3 How do LMs use Epistemic Markers?
text: To answer our first motivating question, we investigate how LMs such as GPT, LLaMA-2, and Claude express uncertainties within a broad and challenging, question-answering context. We find that LMs prefer to respond with answers free of epistemic markers and when LMs do use epistemic modifiers, they rely too much on strengtheners, leading to overconfident but incorrect generations.

heading: 3.1 Methods
text: Our objective is to assess the potential harms and safety risks associated with widely used publicly
deployed models like GPT, LLaMA-2, and Claude.1 We pose a diverse set of questions from the Massive Multitask Language Understanding benchmark (MMLU Hendrycks et al., 2021), a fourway multiple-choice dataset spanning 57 subjects that assess both language model knowledge and problem-solving skills. We design confidence eliciting prompts and measure systematic trends in how LMs use strengtheners and weakeners.
Prompt Design We systematically prompt LMs for epistemic markers by designing three types of open-ended prompt instructions. We modify the base template from the original MMLU paper by appending additional instructions crafted to elicit: 1) epistemic markers “Please answer the question and provide your certainty level” (EpiM), 2) chain-of-thought reasoning (CoT), “Explain your thought process step by step” or 3) a combination of both “Using expressions of uncertainty, explain your thought process step by step” (EpiM+CoT) (Figure 2). Previous studies have shown that chain-of-thought prompts can enhance model behavior through step-by-step reasoning, and we hypothesized that the process of articulating reasoning might also generate epistemic markers (Wei et al., 2022; Suzgun et al., 2023; Wang et al., 2022).
To ensure the generalizability of our results, we employ snowball sampling to generate a list of diverse prompts, gathering additional paraphrases of prompts from Amazon Mechanical Turk Workers and GPT-3.5 (details in §A).
We prompt nine models (text-davinci-003, GPT-3.5-Turbo, GPT-4, LLaMA-2 7B, LLaMA-2 13B, LLaMA-2 70B, Claude-1, Claude-2, ClaudeInstant-1) using 49 prompts on 284 questions, resulting in a total of 125,244 queries.2 Using a zero-
1Models were accessed June - November 2023. 2Duplicated question in the development set of MMLU
benchmark, resulting in 284 instead of 285 questions.
shot prompting approach, we aim to simulate the interactions of end-users. We set the temperature to 0.3 (OpenAI, 2023; Anil et al., 2023), with no stop tokens, and limit the token generation length to a maximum of 400 tokens (details in §C).
Eliciting and Classifying Epistemic Markers The authors then qualitatively code (Auerbach and Silverstein, 2003; Charmaz, 2006) for epistemic markers in generated responses from each model family via Regex pattern matching and categorizing them as strengtheners and weakeners.3
Specifically, the process involved authors 1) manually looking through the generated responses, 2) identifying codes, 3) designing Regex heuristics to automatically detect markers, and 4) evaluating markers manually. Each evaluation consisted of randomly sampled 100 codes and evaluating them by hand. The entire qualitative coding process was repeated six times until we reached over 90% accuracy. Although this process involved the analysis of over a dozen models, as with most supervised methods, future researchers may need to adapt our schema when using it for out-of-distribution content. Our qualitative coding yielded a total of 76 strengtheners and 105 weakeners. See Tables 7 and 8 for the most commonly generated expressions.

heading: 3.2 Findings
text: Models are reluctant to reveal uncertainties, but can be encouraged. LMs fail to incorporate epistemic markers in their responses when prompted with the base template. Only 5% of the generated answers include any type of epistemic markers (Table 1), indicating that the majority of responses seen by end-users lack information regarding model uncertainties. We refer to responses that
3A future iteration could include training a classifier to identify codes at the trade-off of lowered transparency and interpretability.
don’t contain any epistemic markers as plain statements (e.g., “(A)” or “The answer is (A)”).
When using chain-of-thought instructions or explicit instructions to elicit epistemic markers, LMs can be encouraged to produce more epistemic markers. Resulting in 16% and 65%4 of generations incorporating epistemic markers respectively.
Models are biased towards using strengtheners. Six out of nine models have a preference to generate significantly more strengtheners than weakeners. Our results indicate that an average of 20% of generations had strengtheners while only 14% had weakeners. This bias is true among prompts eliciting for certainty, with or without CoT (Table 1). We see this trend emerge strongly among GPT and LLaMA-2 chat models, with the Claude-2 being more balanced in its generation of strengtheners and weakeners (Figure 3). Interestingly, the smaller models (LLaMA-2-7B and Claude-Instant-V15) have a higher use of weakeners over strengtheners; model size and generation of epistemic markers could be explored in future work.
Overconfidence results in confident but inaccurate generations. Across all generations, only 53% of generations with expressions of certainty are correct (random accuracy being 25%). Although this accuracy rate is higher than accuracies among weakeners (32%), this is an alarmingly high rate of errors among strengtheners. Furthermore, due to the high rate of generations of strengtheners, 17% of all incorrect answers include strengtheners.

heading: 3.3 Discussion
text: Our findings illustrate that models struggle to appropriately use epistemic markers. First, models are reluctant to produce uncertainties, even when asked with CoT prompts, presenting a veil of tacit certainty. When explicitly prompted to verbalize model confidences, LMs are prone to overuse expressions of certainty even when the output is incorrect, creating potential downstream harms (see §4). The overuse of expressions of certainty is likely to contribute to the existing problem of human overreliance on AI predictions (Jacobs et al., 2021; Bussone et al., 2015) and explanations (Bansal et al., 2021b; Poursabzi-Sangdeh et al., 2018; Wang and
4Weighted average emittance of epistemic markers when prompted with epistemic instructions and epistemic instructions with chain-of-thought
5Announced as a "lighter" version of Claude. See: https://www.anthropic.com/index/introducing-claude
Yin, 2021; Ehsan et al., 2021). The linguistic miscalibration is emerging as a new safety risk as LMs play a bigger role in humanLM collaborations. In the space of information seeking, the need for uncertainties and knowledge limitations will likely continue to persist even as model performance improves. Moving forward, we must examine how to mitigate the harms of model overconfidence and how to best build cognitive forcing designs, such as verbalized uncertainties, to discourage human overreliance of AI systems (Buçinca et al., 2021).

heading: 4 Human Interpretations of Uncertainty
text: With more robust understanding of how LMs use epistemic markers, we shift our focus to the second inquiry: how do humans interpret LM-generated epistemic markers? Using a subset of expressions generated by LMs, we set up a task to evaluate the effect of these markers on user reliance on AI. We find that users by default are highly reliant on LM-generated responses and that even minor miscalibrations in systems can have long-term consequences in human-LM collaborations.

heading: 4.1 Methods
text: Creating a Self-Incentivized Task We create a self-incentivized task where users must accrue points by correctly answering challenging trivia questions, deciding whether to rely on an AI agent’s response for help. We situate users in an imagined game scenario where they are asked to interact with AI agent named Marvin, a set up adopted from an user-AI interaction study from
Bansal et al. (2019a).6 In the game, the user shown a question (e.g., "What is the capital of Palau?") and a response generated by Marvin that includes epistemic markers (e.g., "I’m certain its Ngerulmud"). The user must decide whether to rely on Marvin’s answer or whether to indicate that they’ll look it up themselves later.
Trivia Question Selection We control for uniformity in scenario content by limiting questions to country capital trivia. Our task should ensure that the users’ assessment of AI reliability stems from the model’s use of epistemic cues rather than users’ own prior knowledge. Hence, we select the most challenging trivia questions as ranked from Sporcle, an online trivia platform.7 By selecting countries where participants are unlikely to know the answer, we encourage users to primarily use LM-generated epistemic markers rather than their own knowledge to make decisions.
Recruitment Process We launch the task using Prolific and Qualtrics and inform the participants of the nature and the risks of the task through a consent form. The task is compliant with internal review board (IRB) protocols.
Template Selection We select the most frequently occurring expressions of certainty and uncertainty from §3 and transform them into prefixes in the context of question answering (e.g., "I think it’s", "Perhaps it’s"). We filter for naturalistic expressions, avoiding any template duplication.
For details on recruitment process, IRB protocol, and template selection see Appendix B.

heading: 4.2 Experimental Settings
text: Setting 1: Control Setting We first use our task to evaluate how participants rely on LM generated epistemic markers (from §3). Participants are
6The original task involves users classifying shapes. We adopt the decision making setup from this work.
7https://www.sporcle.com/games/g/worldcapitals/results
shown a set of trivia questions and the beginning of a response (e.g., "I think it’s..." Figure 4). Users are asked whether or not they’d like to rely on Marvin’s answer. Since the users do not see any answers, they are simply expressing their reliance of epistemic markers as generated by LMs. Participants are presented with strengtheners, weakeners, and plain statements (expressions free of epistemic markers like, “The answer is (A)”). We recruited 25 participants and each were shown 106 questions.
Setting 2: Interactive Settings The next three settings are interactive settings. Participants engage in 50 rounds of question-answering where in each round they are 1) shown a question, 2) shown Marvin’s predicted response with epistemic markers, 3) asked to make a decision, and 4) given feedback on their decision. Providing users with feedback gives users the opportunity to build a mental model of how Marvin performs (Bansal et al., 2019a) and allows us to measure the harms that may arise from long-term interaction (Lee et al., 2022). We recruited 25 new participants for each setting.
In these experimental settings, we introduce a scoring system, also modified from Bansal et al. 2019a. The scoring set-up is designed such that the only way to have a positive score is to rely on Marvin correctly based on relying the epistemic markers (see Table 2).8
Setting 2A: Calibrated Setting In the calibrated setting, Marvin’s responses are calibrated with the expected human interpretations of epistemic markers from the control setting (i.e., strengtheners appear with correct answers and weakeners appear with incorrect answers).
Setting 2B: Overconfident Setting In the overconfident setting, Marvin will use strengtheners
8To avoid spammers, we filtered the bottom 20% participants based on their performance on this task.
when generating the incorrect answer (e.g., "I’m sure the capital of Vanuatu is Luganville" [Incorrect]). This setting has five overconfident responses which will appear in the first 20 questions; the remaining 30 questions are calibrated.9
Setting 2C: Underconfident Setting In the underconfident setting, Marvin will use weakeners when generating the correct answer (e.g., "I’m not certain, but maybe the capital of Vanuatu is Port Vila" [Correct]). Again, five underconfident responses will appear in the first twenty questions.

heading: 4.3 Findings
text: Users rely on strengtheners but also on plain statements In the control setting, when presented with expressions of strengtheners, nearly 90% of users will rely on Marvin’s response. When presented with weakeners, approximately 90% of users will choose to look up the answer (see Table 6 for details). Surprisingly, plain statements like “The answer is (A)” or “(A)” are also relied on by users nearly 90% of the time. In other words, without communicating any epistemic markers, humans interpret this as a sign of model certainty.
Users Effectively Leverage Calibrated LMGenerated Epistemic Markers In the interactive calibrated setting, our results illustrate that users are able to learn mental models of epistemic markers after approximately 20 rounds. In the early rounds, users rely on strengtheners 94% of the time and on weakeners 7% of the time. After 20 rounds, most participants are able to nearly perfectly leverage Marvin’s epistemic markers with users learning to rely on strengtheners 99% of the time and weakeners nearly 1% of the time, averaging an accuracy
9To create consistency in how the answers are incorrect, the largest non-capital city is used instead of the capital city.
of 97% across all rounds. The high performance in this calibrated setting also validates that participants are primarily relying on epistemic markers, to answer these questions. If they had been only relying on their own knowledge, it would be unlikely to see participants perform nearly perfectly on such challenging trivia questions.
Users are Overreliant on Overconfident Responses In the first 20 rounds, users relied on 81% of strengtheners, when only 66% of the generations with strengtheners were correct, accruing on greater penalties than necessary. Because participants are unlikely to know the answer to the question, we see that participants mistakenly rely on incorrect, confident generations an average 73% of the time. The consequences of system miscalibration continued to negatively impact human performance, even after several rounds of calibrated model answers. Participants averaged 76% in the miscalibrated rounds and 86% on the calibrated rounds (Figure 5).
Users in the Overconfident Setting Also Incorrectly Relied on Weakeners An unexpected effect of the overconfident responses was that participants started to interpret weakeners differently. Even though weakeners were used correctly in this setting (i.e., none of the answers with weakeners were correct), the participants were observed to rely on answers with weakeners at a higher rate than in the control setting (9% vs. 3%).
Users are Underreliant on Weakeners in Underconfident LMs Participants in the underconfident setting averaged 66% in the miscalibrated rounds but 98% on the later calibrated rounds, matching the performance in the calibrated setting. This is in contrast to the overconfident setting where users’ mental models were never fully cor-
rected, even after the same number of calibrated rounds.

heading: 4.4 Discussion
text: Plain Statements are Confident Statements The troubling concern that current models struggle to use epistemic markers in calibrated ways (§3) is underscored by our finding that lack of epistemic markers is perceived as confident language (§4). That is, the absence of calibrated markers presents significant harms for human-AI collaboration. For example, LM hallucinations are not only factually incorrect (Ji et al., 2022; Maynez et al., 2020; Zhang et al., 2023), but as our study suggests, they may also be interpreted as high certainty due to the lack of epistemic markers. One potential design recommendation is to generate weakeners without explicit elicitation and use plain statements only when the model is confident.
Miscalibrations in Strengtheners Impact Interpretation of Weakeners Miscalibration in the use of strengtheners resulted in users interpreting weakeners incorrectly as well. Our findings signal that miscalibration in one dimensional (e.g., the incorrect use of strengtheners leads users to distrust other uses of epistemic markers). As a recommendation, researchers measuring the harms of miscalibration must also consider how miscalibration impacts the users’ mental models of the whole system, rather than the perception of just an individual component. This work ties into existing literature on how mental models of AI systems are affected by numerous complexities such as accuracy (Bansal et al., 2021a), warmth (McKee et al., 2024), and system updates (Bansal et al., 2019b).
Long-Term Effects of Overconfidence Our findings signal that mental models of language models are developed early in LM-interactions, potentially resulting in long-term harms, even after models become calibrated later on. This corroborates work from Dhuliawala et al. (2023) who conducted human interpretations of numerical uncertainties from LMs. Similarly, we find that users can correct a mental model developed from an underconfident model but struggle to do so with overconfident models. Unfortunately, public models are overconfident, creating not only a reliability harm now, but also potentially creating long-term algorithmic aversion (Dietvorst et al., 2015) to future models.

heading: 5 Origin of Model Overconfidence
text: Our work shows that LMs are overconfident and that humans are highly reliant on plain statements and statements with strengtheners; all of which is exacerbated by models often being incorrect when expressing certainty. Here, we turn to our last motivating question: What is the origin of model confidence and what are potential mitigations? In this section, we pinpoint LM overconfidence to an artifact of the post-training alignment process, specifically a bias from human annotators against uncertainty.

heading: 5.1 Where Does LM Overconfidence Begin?
text: Current state-of-the-art models are trained using a number of techniques to support human-AI collaboration through natural language. Starting with a pretrained base model, one of the most popular techniques is to use supervised fine-tune (SFT) using human instructions. The model is then trained with reinforcement learning with human feedback (RLHF), where a reward model is learned through human preferences of pairwise text comparisons.

heading: 5.2 Methods
text: Model Stages We perform analysis on the models from the GPT and LLaMA-2 family to identify the origin of model overconfidence. Using the same prompting strategy as §3, we measure how base models and supervised fine-tuned models compare to their RLHF counterparts when it comes to generating expressions of certainty.
We compare three models from the GPT3 family, davinci, text-davinci-002, and text-davinci-003 which are base, supervised fine-tuned, and RLHF models, respectively.10 We then compare LLaMA-2 models base models with their SFT+RLHF counterparts.
Reward Modeling We directly probe OpenAssistant’s open-sourced reward model trained on human feedback datasets and assess their scoring.11 We prompt the model with a question-response pair where the question is "What is the capital of X?" and the response is an epistemic marker like "I think it’s". We test the reward model on 183 question answer pairs across a subset of 30 commonly occurring templates generated by LMs in §3 and
10Models were accessed June - November 2023. 11https://huggingface.co/OpenAssistant/reward-model-
deberta-v3-large-v2
compare the model scores with human judgements from §4.
Human Annotated Datasets Lastly, we examine the datasets that were used to train the opensourced reward model. Specifically, we examine the datasets: OpenAI’s “WebGPT comparison” and “Summarize with Feedback”, Dahoa’s “Synthetic Instruct GPT Pairwise” dataset, and Anthropic’s “Helpful and Harmless” dataset.11 We then measure how often strengtheners and weakeners are preferred by human annotators in these datasets.

heading: 5.3 Findings
text: Overconfidence in RLHF Models We quantitatively observe that RLHF-ed models emit more strengtheners than weakeners, which contrasts to the base and instruction-tuned variants where the pattern is the opposite (Figure 6). This suggests that this preference for strengtheners is introduced during the RLHF process.
Reward Modeling Is Biased Towards Certainty Reward modeling prefers plain statements with an average score of 4.03, followed by strengtheners with a score of 0.82. However, there is a strong penalty applied to weakeners, with the average rewards score of -1.86. Lower scores in reward modeling would in turn influence how LMs generate natural language, leading to a bias in avoiding the language of uncertainty at generation time. See Table 3 for a comparison between reward model and human scoring.
Human Raters are Biased Against Uncertainty We annotate texts in each datasets as containing strengtheners and weakeners and measure how these rates vary across the chosen and rejected texts as annotated by human judges. Given the prevalence of model overconfidence, we hypothesized
that there may be a preference for strengtheners in chosen texts. We find that this is not the case. In fact, there is a slight, but significant preference for strengtheners in rejected texts as compared to chosen text (2.95% vs. 2.72%).12 In a pairwise comparison, we see that plain text is actually slightly preferred over strengthened texts (chosen 9% more often). This shows that in these datasets, there is not a human bias for strengtheners. Weakeners also appears significantly more often among rejected texts (5.02%) compared to chosen texts (4.47%). In a pairwise comparison, weakeners are preferred 8% less often than strengtheners and 9% less often than plain texts. This highlights that annotators don’t have a bias for certainty, rather there is a bias against weakeners, matching the results seen from the reward model experiments.

heading: 5.4 Discussion
text: Uncovering Unknowns in Human Preferences Feedback Alignment Our investigation into RLHF datasets reveals that humans have implicit biases towards other dimensions of language which may not be known in the annotation phase. Although our study focuses RLHF, the process of aligning language models to biased human feedback will remain as issue, regardless of the exact training method (e.g., DPO or SLiC) (Zhao et al., 2023; Rafailov et al., 2024). The artifact of humans having a bias against uncertainty adds to the list of implicit biases in annotations (Gururangan et al., 2018) (e.g., text length (Saito et al., 2023), toxicity detection (Sap et al., 2022b), political leanings (Santurkar et al., 2023)). However, the human bias against uncertain language is particularly harmful as it causes aligned models to be reluctant in their generation uncertainty, negatively impacting human over-reliance on LMs. Interventions such as swapping labels on annotated datasets could potentially mitigate some overconfidence harms but could also risk introducing new, yet to be known biases. Investigations are needed in human feedback datasets and annotation processes to fully understand the potential implicit biases that are introduced through human preference annotations.
Beyond Mimicking Human Language As LMs evolve towards generating more nuanced language, such as uncertainties, a shift in design is prudent. The work of Hollan and Stornetta (1992) discusses the need to design technology that goes beyond
1295% CI calculated using bootstrap sampling.
simply mimicking that of the real world. We could apply this design thinking towards designing natural language as an interface. Instead of building language models simply based on mimicking human preferences, we could instead design LMs to verbalize uncertainty in ways that would increase cognitive engagement and lower human overreliance (Buçinca et al., 2021).

heading: 6 Desired Criteria and Mitigating Solutions Moving Forward
text: Our research highlights the overconfidence of language models, the reliance of humans on generations, and the long-term consequences of miscalibration in human-AI interactions. Here, we propose three criteria and potential implementations to mitigate overconfidence in LMs.
Criteria: Unsolicited Epistemic Markers Language models should autonomously emit expressions of uncertainty without prompting, akin to human behavior, in order to appropriately convey levels of confidence. This is critical as the lack of epistemic marker emittance is perceived by humans as tacit certainty.
Concretely, this could be addressed in various stages of the RLHF pipeline, including data augmentation, annotator training, or dataset evaluation. RLHF datasets could be augmented to include additional and more representative epistemic markers. Similar studies have been successful in improving safety (Ji et al., 2023), modeling different perspectives (Dong et al., 2024), and performing moral self-correction (Ganguli et al., 2023). Annotators could also receive training (Clark et al., 2021) and priming (Sap et al., 2019) to be cognizant of potential biases (e.g., biases against uncertainty). Lastly, automatic sensibility checks and data filtering using our heuristics could be used to evaluate the effectiveness of the proceeding interventions, building transparency towards the levels of overconfidence in RLHF datasets.
Criteria #2: Comprehensive Coverage Language models ought to have the capacity to generate and interpret the full range of epistemic markers. However, current pre-training data is often limited to written internet data, leading to numerous limitations and biases (Gordon and Van Durme, 2013; Lucy and Gauthier, 2017; Sap et al., 2022a).
Concretely, incorporating more diverse data sources would allow models to learn a larger range
of epistemic markers. For example, hedges have long been studied in speech in formal and informal settings (Aijmer, 1986, 2013) such as presidential debates and speeches (Al-Rashady, 2012; Mansour and Alghazo, 2021), TED talks (Nuraniwati and Permatasari, 2021), and student presentations (Muziatun et al., 2021). Leveraging alternative training sources such as podcasts, news transcripts, and peer conversations could broaden the use of expressions of uncertainty by LMs.
Criteria #3: Context-Dependent Calibration In addition to being internally calibrated (i.e., calibrated between internal probabilities and verbalized certainties), LMs should also be contextdependent in their calibration. If a model were deployed for entertainment purposes, a lower threshold for expressions of certainty could be tolerated; but if the same language model were to be deployed to a mission-critical task, new thresholds based on the context must be determined.
Concretely, this could be accomplished via user calibration or system-level prompts. For example, one could implement an initialization procedure, similar to the procedure shown in §4.2, that allows practitioners to measure how humans would rely on LM-generated expressions in a specific context. The results of this procedure would then inform practitioners and users of how to best to fit the model to the specific needs using few-shot prompting, fine-tuning, or system-level prompts. Future work could include investigate online algorithms that adjust the certainty thresholds based on the human-LM initialize round.

heading: 7 Conclusion
text: Our work set out to explore how users interpret epistemic markers as generated by LMs in an effort to better understand the shortcomings of human-LM communications. We find that LMs are overconfident in their generations and that users are highly reliant on LM responses whether there is implicit or explicit confidence. We trace the origin of model overconfidence to the RLHF process and find that annotators have a bias against uncertainty in text.

heading: 8 Limitations
text: Cultural Interpretations of Epistemic Markers Our study focused exclusively on how language models generate epistemic markers in the English language (Bender, 2019). Humans greatly differ
in their use of hedges and strengtheners across languages, contexts, and cultures (Itani, 1995; Lauwereyns, 2002; Yagız and Demir, 2014; Nguyen Thi Thuy, 2018; Mur-Dueñas, 2021), future studies could consider how non-English language models differ in their use of strengtheners and weakeners.
Our human studies recruited U.S. based participants exclusively and their willingness to rely on epistemic markers is shaped by their experiences and cultural context. Our results thus illustrate a narrow and U.S.-centric view of how humans might interpret epistemic markers (Henrich et al., 2010; Atari et al., 2023). Participants from other cultural backgrounds might reveal different findings on how humans rely on LM-generated epistemic markers.
The Ambiguity of Weakened Strengtheners LMs articulated epistemic markers that fell in between strengtheners and weakeners, which we labeled as labeled as weakened-strengtheners. This schema closely follows Sanders and Spooren (1996)’s schema of certainty, uncertainty, and semicertainty markers. In our human experiments, we found that participants displayed great variance in their reliance of weakened strengtheners as some humans appear to rely on weakened strengtheners meanwhile others do not (see Table 6). This ambiguous interpretation of weakened strengtheners highlights a potential risk for miscommunication; the prolific use of them could lead to more confusion and misinterpretation than clarity. Further work on the more nuanced features of epistemic markers is needed.
Gap Between Human Experiments and Real Self-Incentivized Users A gap still exists between self-incentivized users and the participants who we recruited for our experiments. Despite our best efforts to situate users in a real-life scenario, the harms we uncover here will likely differ from that of users in a real deployed setting. The contexts in which users might engage with these chat models would like also influence their interpretability of epistemic markers (Grice, 1975; Goodman and Frank, 2016). Changes such as the metaphors associated with the agent itself could have significant impacts on user reliance behaviors (Khadpe et al., 2020). Further investigations and in-depth user studies and interviews may be needed to comprehensively study the harms of LM overconfidence.

heading: 9 Ethics Statement
text: Our paper is primarily considered with the potential harms and ethical implications that may arise when humans interact with LLMs. Our findings illustrate that LLMs systematically fail to represent model uncertainties, creating a threat that results humans overrelying on incorrect generations. In regards to our human experiments, we followed standard practices such as providing participants with informed consent, paying participants, on average, over $15 USD/per hour and debriefing participants on the correct answers if they made errors when interacting with our system.

heading: Acknowledgements
text: Thank you so much to Dan Jurafsky, Yejin Choi, Myra Cheng, Kristina Gligorić, Abhilasha Ravichander, Tal August, Luca Soldaini and the AI2 Mosaic team for their helpful feedback and advice! Thank you so much to the numerous participants who helped us with our pilot and formal human studies. Kaitlyn Zhou is supported by the Stanford Graduate Fellowship. Xiang Ren’s research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200006, the Defense Advanced Research Projects Agency with award HR00112220046, and NSF IIS 2048211.

heading: A Details on Prompt Paraphrases
text: Initially, the authors generated a list of prompts which was paraphrased by Amazon Mechanical Turk workers (details in Figure 7). The resulting paraphrased prompts then served as seed prompts for GPT-3.5 to generate additional variations. We take measure to maintain the neutrality of prompts; with keywords of certainty and uncertainty appearing together in random order (details in Table 4, 5).

heading: B Details on Experiments from Section 4
text: Recruitment Process Details We aimed to pay participants an average of $15 USD an hour (average actual payment was $17.77 USD/hour). Participants were filtered out to be English speaking, U.S.
based, with an approval rating of at least 97% and had completed 100 or more tasks on Prolific. Each experiment had 25 participants. Human experiments were run throughout the months of September - November 2023.
Our research teach sought and received exemption from our internal review board (IRB). We do not collect sensitive or demographic information. The exemption does not require a consent form but we used a consent form an collected informed consent from all our participants.
Most Frequently Occurring Expressions Expressions were filtered out if they were nearly identical to each other to avoid duplicate templates (e.g.,"I do not know" vs "I don’t know") as well as expressions which were would be primarily numeric or ordinal (e.g., "Confidence: 90%" or "Certainty: High" as it would break form from the other naturalistic expressions of certainty/uncertainty).
Scoring Details Specifically, participants receive a point if they rely on Marvin’s response and Marvin is correct but lose a point if Marvin is incorrect. If they choose to look up the answer, they will neither gain nor lose a point. Half the answers are wrong so choosing to always to rely on Marvin or always looking up the answer will yield a total score of 0. The user’s score is updated and shown to the user after every question and users are explicitly informed that their performance on this task is independent to their compensation.
C Insensitivity to Increases in Temperature
A recognized mitigation strategy to address model miscalibration (i.e., calibrating model accuracy with model confidence) among RLHF models is to increase the temperature (Kadavath et al., 2022). The intuition is that reward modeling encourages the model to concentrate their predictions towards those which would score highest in reward modeling and an increase in temperature could help resolve these issues (Kadavath et al., 2022). We experiment with using maximum temperatures, only to observe the persistent effect of LMs preferring generating strengtheners over weakeners. This mitigation strategy also illustrates that the issue of model calibration is separate from linguistic calibration of LMs.

heading: D Reward Model Compared to Human Scoring of Expressions of Uncertainty

heading: E Licenses for Scientific Artifacts
text: All artifacts were used as intended (querying the models and using the datasets for evaluation and analysis). Used artifacts include: OpenAssistant Reward model and Datasets (License: MIT), MMLU Dataset (License: MIT), and LLaMA 2 (LLaMA 2 Community License).

heading: Future Work
text: Humans greatly differ
in their use of hedges and strengtheners across languages, contexts, and cultures (Itani, 1995; Lauwereyns, 2002; Yagız and Demir, 2014; Nguyen Thi Thuy, 2018; Mur-Dueñas, 2021), future studies could consider how non-English language models differ in their use of strengtheners and weakeners. Our human studies recruited U. S. based participants exclusively and their willingness to rely on epistemic markers is shaped by their experiences and cultural context. Our results thus illustrate a narrow and U. S. -centric view of how humans might interpret epistemic markers (Henrich et al. , 2010; Atari et al. , 2023). Participants from other cultural backgrounds might reveal different findings on how humans rely on LM-generated epistemic markers. The Ambiguity of Weakened Strengtheners LMs articulated epistemic markers that fell in between strengtheners and weakeners, which we labeled as labeled as weakened-strengtheners. This schema closely follows Sanders and Spooren (1996)’s schema of certainty, uncertainty, and semicertainty markers. In our human experiments, we found that participants displayed great variance in their reliance of weakened strengtheners as some humans appear to rely on weakened strengtheners meanwhile others do not (see Table 6). This ambiguous interpretation of weakened strengtheners highlights a potential risk for miscommunication; the prolific use of them could lead to more confusion and misinterpretation than clarity. Further work on the more nuanced features of epistemic markers is needed. Gap Between Human Experiments and Real Self-Incentivized Users A gap still exists between self-incentivized users and the participants who we recruited for our experiments. Despite our best efforts to situate users in a real-life scenario, the harms we uncover here will likely differ from that of users in a real deployed setting. The contexts in which users might engage with these chat models would like also influence their interpretability of epistemic markers (Grice, 1975; Goodman and Frank, 2016). Changes such as the metaphors associated with the agent itself could have significant impacts on user reliance behaviors (Khadpe et al. , 2020). Further investigations and in-depth user studies and interviews may be needed to comprehensively study the harms of LM overconfidence