Do LLMs estimate uncertainty well in instruction-following?
Authors: Juyeon Heo
Miao Xiong
Christina Heinze-Deml
Jaya Narain
Venue: ICLR 2025 Poster (2025)
Keywords: Instruction-following
Uncertainty
Large language models
Evaluation
Benchmark dataset
Primary area: foundation or frontier models, including LLMs
TL;DR: We present the first systematic evaluation of uncertainty estimation in LLMs for instruction-following tasks, introducing controlled benchmark datasets to comprehensively assess and compare uncertainty estimation methods.

Abstract: Large language models (LLMs) could be valuable personal AI agents across various domains, provided they can precisely follow user instructions. However, recent studies have shown significant limitations in LLMs' instruction-following capabilities, raising concerns about their reliability in high-stakes applications. 
Accurately estimating LLMs' uncertainty in adhering to instructions is critical to mitigating deployment risks. We present, to our knowledge, the first systematic evaluation of the uncertainty estimation abilities of LLMs in the context of instruction-following. 
Our study identifies key challenges with existing instruction-following benchmarks, where multiple factors are entangled with uncertainty stems from instruction-following, complicating the isolation and comparison across methods and models.
To address these issues, we introduce a controlled evaluation setup with two benchmark versions of data, enabling a comprehensive comparison of uncertainty estimation methods under various conditions.
Our findings show that existing uncertainty methods struggle, particularly when models make subtle errors in instruction following. While internal model states provide some improvement, they remain inadequate in more complex scenarios. 
The insights from our controlled evaluation setups provide a crucial understanding of LLMs' limitations and potential for uncertainty estimation in instruction-following tasks, paving the way for more trustworthy AI agents.

Published as a conference paper at ICLR 2025
Do LLMS ESTIMATE UNCERTAINTY INSTRUCTION-FOLLOWINN::
WELL IN
Juyeon Heoli
Miao Xiong't Christina Heinze-Deml Jaya Narain² Universitt of Cambridge Apple National University of Singapore
jh23240 cam. ac.uk jnarain@apple. com
ABSTRACT
Large language models (LLMs) could be valuable personal Al agents across var- ious domains, provided they can precisely follow user instructions. However, re- cent studies have shown significant limitations in LLMs' instruction-followe pabilities, raising concerns about their reliability in high-stakes applications. Ac- curately estimating LLMs' uncertainty in adhering instructions is critical to mit- igating deployment risks. We present, to our knowledge, the first systematic eval- uation of uncertainty estimation abilities of LLMs in the context of instruction- following. Our study identifies key challenges with existing instruction-following benchmarks, where multiple factors are entangled with uncertainty stemming from instruction-following
ca-
complicating the isolation and comparison across methods and models. To address these issues, we introduce a controlled evalu- ation setup with two benchmark versions of data, enabling comprehensive com- parison of uncertainty estimation methods under various conditions. Our findings show that existing uncertainty methods struggle, particularly when models make subtle errors in instruction following. While internal model states provide some improvement, they remain inadequate in more complex scenarios. The insights from our controlled evaluation setups provide crucial understanding of LLMs' limitations and potential for uncertainty estimation in instruction-followe tasks, paving the way for more trustworthy Al agents
INTRODUCTION
Large language models (LLMs) have garnered interest for their potential personal AI agents across various domains, such as healthcare, fitness, nutrition, and psychological counseling (Li al, 2024 Wang al., 2023a; Tu et al. 2024). A key building safe and useful personal Al agents with LLMS lies in their ability to Tollow instructions precisely. Deployed models must adhere to the constraints and guidelines provided by users ensure that the outputs are both aligned with user intentions and safe. Yet recent research has exposed significant limitations in LLMs' ability to follow instructions (Zhou et al. 2023 Zeng et al., 2023 Qin et al., 2024, Xia et al. 2024; Kim et al. 2024 Yan et al.] 2024). For example, even large models like GPT-4 achieve only around 80% instruction-follovii accuracy on simple and non-ambiguous instructions from benchmark datasets, and smaller models perform even worse, with accuracy less than 50% (Sun et al. (024).
Since LLMs are prone to errors, their ability to accurately assess and communicate their own uncer- tainty is essential. This becomes particularly important in high-stakes applications, where mistakes can have serious consequences. For instance, an LLM developed for personal psychological coun- seling must strictly adhere guidelines that avoid topics that might potentially cause trauma. If the L.M misinterprets or deviates from these instructions but accurately recognizes and signals high uncertainty, it could prompt further review intervention, thereby preventing the delivery of potentially harmful advice.
However, uncertainty estimation in instruction-followin tasks has received limited attention, with most research focusing on fact-based tasks like question answering and summarization (Fadeeval
Work done while at Apple.
Code and data available http://daaacaooggoo.....oommmm/e

---

Published as a conference paper at ICLR 2025
et al. 2023 Kuhn et al. 2023 Xiong et al. 2023: Ye al. 2024). where factual correctness is the primary concern. In contrast, as shown in Figure instruction folloooinnn tasks focus on whether a model's 's response adheres to set of given instructions, rather than estimating the factual accuracy. Given these different source of uncertainty, it is unclear whether existing methods, which are typ- ically designed for estimating factual uncertainty, can accurately capture uncertainty in instruction following. For example, while semantic entropy (Farquhar et al., (022) is considered the gold stan- dard for fact-based tasks, may not be suitable for mnstruction-folloving Both responses, 'Regular exercise strengthens muscles' and 'Regular exercise reduces stress', follow the given instruction in Figure [] but convey different semantic meanings, leading high semantic uncertainty score, which inaccurately reflects cetructtio-followwiin performance. This example highlights the need for frameworks tailored to evaluating methods and models for uncertainty estimation instruction- following tasks.
To evaluate how well existing uncertainty estimation methods and models perform on instruction- following tasks, we evaluate six uncertainty estimation methods across four LLMs on the IFEval benchmark dataset (Zhou et al. 2023). However, we find that multiple factors are entangled in exist- ing benchmarks. For instance, uncertainty can stem from both task execution quality and instruction following, making it difficult to isolate and directly compare methods and models based solely on their ability estimate mnstruction-folowiiii benchmark dataset with two versions to enable a more controllable and fine- gra-igai evaluation. The Controlled version provides a structured assessment by removing confounding factors and offering tasks with varying difficulty levels, split into Controlled-Easy where correct and incorrect responses are easy to distinguish, and Controlled-Hon Hard, which focuses on more subtle errors. In contrast, the Realistic version uses naturally generated LLM responses, retaining real-world signals. Together, these datasets provide a comprehensive framework for evaluating uncertainty estimation methods and models under both controlled and real-world conditions.
uncertainty. To address these issues, we design new
Our analysis revealed several key findings from the controlled evaluation: Verbalized method consistently outperforms logit-based methods like perplexity in the Controlled-Fasy setting, where correct and incorrect responses are relatively easier to distinguish. Specifically, normalized p(true) Kadavath et al. (022)) proves to be reliable uncertainty method actoss both Controlled- and Realistic settings. 2) Smaller models often outperform larger ones in verbalized confidence, sug- gesting that factors beyond model size, such as tuning architecture, may contribute better uncertainty estimation in certain tasks. 3) Probes relying on the internal states of LLMs outperform logit-based and verbalized confidence, highlighting promising directions for future work. 4) In more challenging tasks like Controlled-Hain which involve subtle off-target responses, all approaches including internal representations struggle to estimate uncertainty accurately, pointing limitations in LLMs' ability to handle complex uncertainty. These findings from our controlled evaluation setups provide crucial insights into the limitations and potential of LLMs for uncertainty estimation in citructtio-followiig tasks, towards more trustworthy Al agents.
inherent
1.1 CONTRIBUTIONS
Systematic Evaluation: We present the first systematic evaluation of uncertainty estimation methods in fnslroctionnfllloiinng tasks, addressing gap in existing research.
We identify key challenges in existing datasets and introduce a new bench- mark dataset specifically tailored for direct comparison and fine-grained analysis of uncertainty estimation methods and models in both controlled and real-world conditions.
Benchmark Dataset:
Findings: Our evaluation results highlight the potential of self-evaluation and probing methods and point out limitations in handling more complex tasks, underscoring the need for further re- search to advance uncertainty estimation in instruction-followin tasks.
UNCERTAINTY ESTIMATION ABILITY IN CETTRUCTION-FOLLOOWO ON IFEVAL
In this section, evaluate LLMs' uncertainty estimation abilities using the IFEval dataset (Zhou jtt al.) 2023), applying six baseline methods across four LLMs. We selected IFEval because it is designed so that simple and deterministic program can verify whether response follows the in-
2

---

Published as a conference paper at ICLR 2025
Uncertainty estimation in Instruction-follovinn
Imput prompt
Task: Explan the benefits of regular exercie.
"Uncertainty sourced from wwwp://nntsstttttooooooooooooeee distinct from factuality"
Instructions: Do not mention any specific body organs.
respanses
Factuality Inst-following
Uncertalitty from factuality
Uncertainty from inst-following
Regular strengthens
Improves lung function, and supports brain health
0.1
0.9
increases energy levels, strengthens muscles
Regular improve mood.
0.1
0.1
Why evaluating uncertainty estimation ability in instruction-followin matters. Un- certainty in instruction-folloiing distinct from factual correctness, as illustrated by this example. While both responses shown are factually correct, the first fails to follow the instruction, resulting in high uncertainty from cntruction-following despite low uncertainty from factuality. The second response adheres to the instruction, with low uncertainty in both areas. Prior work on uncertainty has focused primarily on factual correctness, underscoring the need for an evaluation framework for instruction-follin foolooing tasks.
Figure 1:
structions. This enables a fully automatic and accurate assessment of a model's instruction- followwing capability, thereby minimizing uncertainties from ambiguous evaluation criteria.
2.1 METHODS
Data We evaluate uncertainty estimation with the IFEval dataset (Zhou et al. 2023 which is de- signed to evaluate the instruction- foolltiinn ability of LLMs on 25 verifiable instruction types under 9 categories across 541 total prompts. Each prompt consists of two components: task and an instruction, where the instruction specifies the action (e.g.. "please do not use keywords", "please start/finish your response with exact sentence") and the task provides the context for executing the instruction (e.g., "please write a resume", "please give a summary about solar system").
Models and Metrics We evaluate four LLMS of varying sizes: I.LaMA2-chat-11 (Touvron al. 2023), LLaMA2-chat-1n 13B (Tovrron et al. 2023), Desstal-7B-Instruct-ve (Jiang et al. (023). and 2024). To avoid randomness decoding, we employ greedy decoding without sampling. Area Under the Receiver Operating Characteristic curve (AUROC) (Pe- dregosa al.] (01T) is used measure the models' uncertainty estimation matches the ground truth labels on correctness in instruction following, generated using the automated evaluation func- tions from IFEval.
Phi-3-mini- Sommini-18kkinnee 128k-instruct (Abdin
Baseline uncertainty estimation methods To evaluate uncertainty in instraction-follovingg we em- ploy several baseline methods, including self-evaluation of their own uncertainty (verbalized confi- dence, normalized p(true) and p(truc)), logits-based method (perplexity, sequence probability, and mean token entropy), and probing method:
Verbalized confidence (Lin et al. 2022 Xiong al. 2023 Tian et al. 2023 ) The model's self-reported confidence, scored from 0 to 9, indicating its perceived likelihood that the response correctly follows instructions. Detailed prompts are in the Appendix A.2
Normalized p(true) and p(true) (Kadavath al., 2022 Lin et al. 2022): These methods assess the probability of the 'true' token, calculated from a binary choice prompt. We modified the calculation determine the probability of token 'A' given the prompt: "Does the response: (A) Follow instructions (B) Not follow instructions. The answer is: Normalized p(true) adjusts for biases by considering both tokens of 'true' and 'false" probabilities: p(true)/ (p((re)) pffasse)), here is p(A) (p(() + p())..
Perplexity and Sequence probability (Jelinek et al. 1977 Fomicheva et al.
2020): Perplex-
ity measures the likelihood of generating given sequence: exp - - log po x<)) where is the sequence length. Sequence probability, an unnormalized version, is calculated as: exp Ei= Le x making it more sensitive to the length of the response, with longer sequences generally having lower probabilities.
3

---

Published as a conference paper at ICLR 2025
AUC of uncertainty methods
SR
Model
Verbal
Perplexity
Sequence
Nor-p(truee
p(true)
Entropy
LLaMA2-chat-1BB
0.53
0.44
0.61
0,47
0.51
0,48
0.57
LLaMAA-chaat77
0.53
0.43
0.54
0.52
0.51
0.44
0.59
Mistral-7B- Baadi------nstuut-O...
0.50
0.48
0,56
0.57
0.47
0.50
0.64
Phi-3-mini- momiini-188k-ee 128k-instruet
0.53
0.43
0.48
0,55
0.45
0.45
Table I: AUROC for baseline uncertainty estimation methods applied to the IFEval dataset
Zhou al. 2023). AUROC measures how well each method's uncertainty estimates align with the ground truth regarding correct or incorrect instruction-follovin
across four LLMs. Success Rate (SR) represents the model's instruction-folloiing accuracy, calculated using the IFEval evalu- ation function. Notably, LLMs struggle to estimate uncertainty in instraction-folloimmg tasks hover around chance levels (between 0.43 and 0.53).
Mean token entropy for LLMs (Fomicheva et al.
2020): Entropy measures uncertainty based
on token prediction variability: H = = - -i=11 Po(
Drawing inspiration from Liu et al. 2024), which suggests that model representations contain valuable information for uncertainty estimation in tasks like question and answering, we investigated whether this internal knowledge could similarly improve uncertainty estimation in instructionnfollloiing tasks. We trained a linear model as an uncertainty estimation function that maps the internal representations of L.Mss to instruttionfoolloinn success labels, where the prob- ability predicted by the linear model used as uncertainty scores. We use layers 16, 32, and 40 and for LLaMA-2- 1---A-2-13B-haa and 14, 26, and 32 for other three models. The ground truth for success or failure is determined by a deterministic program that verifies instruction adherence using the instruction- corllciion-oolowinng success labels,
Probing:
IFEval dataset. We train linear model representations optimized with Adam W, a 0.001 learning rate, 0.1 weight decay. The model is trained for 1000 epochs 70% training set and is evaluated on 30% test We refer to this Probe.
2.2 FINDINGS
Table summarizes uncertainty evaluation results using IFEval.
LLMs struggle to estimate uncertainty in instruction-follovin models and uncertainty estimation methods hover around chance levels (between 0.43 and 0.53), indicating that the models consistently fail to reliably assess their own uncertainty in instruction- following. This underscores the challenge L.LMs face in detecting when their responses deviate from the instructions.
Average AUROC values across
Sequence probability outperforms perplexity, revealing a length signal in uncertainty estima- tion. Sequence probability consistently achieves higher AUROC scores than perplexity across most models. Sequency probability is tied to by sequence length, whereas perplexity is not. For example, in LLaMA2-chat-138 the AUROC for sequence probability averages 0.61 (above chance) across instruction types, whereas perplexity lags at 0.44. This finding implies that response length may in- advertently provide signal in some uncertainty estimation metrics, even though it may not correlate directly with the correctness of the response in instruction-followinn
No model method consistently excels across instruction types. As shown in Table 5 in Ap- pendix, there is no consistent pattern of performance of uncertainty estimation method or model across different instruction types. This lack of consistency indicates that none of the uncertainty estimation methods evaluated reliably capture uncertainty across all instruction types and models.
2.3 CHALLENGES IN EVALUATING UNCERTAINTY ESTIMATION USING EXISTING DATASETS
An instruction-followinn dataset with clear evaluation criteria, like IFEval, is important for eval- uating mnstruction-foloowingg However, we identified the importance of several additional factors uncertainty of LLMs. In systematic
to aid in comparatively evaluating iistruction-followiii evaluation, we identified multiple factors affecting uncertainty that are entangled within naturally generated responses, making it difficult to isolate the strengths and weaknesses of each method or
4

---

Published as a conference paper at ICLR 2025
IFEval 6ttaset((lnnn-2cchtt788)
Difference
lengths instruction modol
Controlled
Gornect encernent
Propet Lue
LL
Token longtt
Instructios Types
Tawch leegth
(a) Length distribution on IFEval (b) Length differences by instruction types (c) Length in Controlled version
Figure 2: Existing instruction-follooing datasets only evaluate uncertainty estimation methods in length-biased settings, missing comparisons controlled, length-neutral conditions. The distributions are normalized by the total number of responses in each class. (a) Token lengths dis- tribution for L.L.MA--ccha-778 shows that incorrect responses tend to be longer than correct ones. This length signal is prevalent in existing dataset like IFEval, where naturally generated responses are used. (b) Token length differences broken down by instruction type and model, with positive values showing that incorrect responses tend to be longer. However, this trend is not consistent across all instructions or models, highlighting the need for controlled evaluation setups. (c) To- ken length distribution in the Controlled version of our benchmark, where token length is balanced across correct, incorrect, and subtly incorrect responses. This setup allows us to evaluate uncertainty estimation in both length-biase biased and length- neutral settings.
model. Below, we outline the primary challenges we identified and present analyses demonstrating each challenge.
1) Uncertainty estimation methods and models are only evaluated in length-biased settings in existing instruction-folloviing datasets, missing comparisons on controlled, length-neutral conditions. We observe that token length significantly impacts uncertainty estimation instruction- following tasks in existing datasets, where responses are not controlled but are generated as part of the evaluation. With datasets like IFEval, naturally generated incorrect responses tend to be longer than correct ones across models. Figure 2 and Appendix Figure 10 present a histogram of token lengths for both response types on four LLMs, while Appendix Table [T] in the provides detailed statistics on response length, including the mean and standard deviation for correct and incorrect responses. Consistently, our analysis show that incorrect responses lend to be longer than correct ones across different LLMs in IFEval dataset.
If this pattern holds across instruction types, length could arguably be reliable signal in evaluating instruction instruction-foolooinng uncertainty. However, we found that the relationship between response length and correctness is not uniform (Figure In instruction types like 'startend' and 'punctuation', incorrect responses were uniformly longer across all models. Conversely, in instruction types such as 'detectable-contennt,, 'keywords', and 'language', correct responses were often longer. These varying patterns are not surprising, as response length is related to what the instruction requires. For instance, an instruction like "please elaborate" would naturally result in a longer response, whereas "make concise" would lead shorter one.
These inconsistencies suggest that length is not a reliable or generalizable signal for uncertainty es- timation across all instruction types and models. This highlights the need for a controlled evaluation framework that includes a set of cength-neutralized responses. By comparing LLMs and uncertainty estimation methods in both length- biased and length-neutral settings, we can more accurately assess their true performance, independent of confounding factors like token length.
2) Uncertainty sourced from task execution quality is entangled with uncertainty stemming from instruction-folloving complicating accurate evaluation. In the IFEval data, each prompt consists of two parts: the task context (e.g.. "Write a brief summary about the solar system") and the specific instruction (e.g.. "Please do not mention any planet names"). When measuring uncertainty with baseline methods, uncertainty can stem from both task execution quality (i.e., how well the accuracy
task itself is accomplished clear. detailed, and informative) and cnttructionfolllwwie (i.e., whether the instruction is followed). This creates an entanglement that complicates evaluation. For example, consider the response "Objects in space around a star", which is vague and unclear with
5

---

Published as a conference paper at ICLR 2025
Case
Task quality entanglement Task quality
Model
Instruction-following eval
Inst-following
Verbalized confidence
Correct
Wrong
Diff
Case
H
7.53 = 0.49
Llama-2-chat-11 1-chat-113B
7.48 2.78
6.37 3.40
1.11
Case
7.00 0.63
Llana-2-chatt7B
7.10 3.24
6.56
3.15
0.54
Case 3
H
7.42 = 0.51
Mistral-7B Inst.v0.3
7.54 + 2.95
6.77 4 3.29
0.77
Case
6.64 0.33
Phi-3-mini- Beriinii-28k-innne 128k-instruct
6.10 3.61
5.12 3.77
0.98
Table Left Verbalized confidence from the LLaMA- -.aMM---cchh--7B model on four cases of instruction- following and task quality. GPT-4 evaluates task quality(O-9), where H and L represent high and low task (uality(threshold 8). respectively, and and indicate whether the responses successfully follow the instructions. It shows that verbalized confidence is affected by task quality, revealing the entanglement between uncertainty from task execution quality and instruction-followin scores for correct and incorrect responses across four LLMs. The Diff' represents the gap between correct and incorrect cases, showing different levels of difficulty in distinguishing between correct and incorrect responses across models.
Right
GPT-4 evaluates instruction-follwwee
low task quality but adheres to the instruction of avoiding mentioning planet names. Alternatively, "The solar system consists of a central star surrounded by various celestial bodies, including Earth and Mars" is informative with high task quality but fails to follow the instruction. We observe that task execution quality also influences the models' uncertainty scores. For example, in Table 2 the LLaMA-2-chat-7B model assigns an average verbalized confidence score of 7.0 to the first response grouping (low task quality but correct instruction-followinng and 7.4 to the second (high task quality but incorrect instuctioon-folloinng
This shows that task quality can confound uncertainty estimation instruction fowlloiio....lowiigg..oooo When task execution quality is not controlled, the uncertainty arising from task completion can over- shadow the uncertainty associated with following instructions, leading to inaccurate evaluations of the model's true instruction-folloving uncertainty. A controlled setup, where task quality is held constant, can help disentangle these two sources of uncertainty, allowing for a more focused evaluation of models' uncertainty estimation specific to instruction-followingg Detailed prompt for evaluating task execution quality can be found in Appendix A.3
3) Differences in the severity of instruction-folloiing mistakes across models create incon- sistent difficulty levels, complicating model comparisons. Our primary objective is to assess LLMss uncertainty estimation capabilities, independent of their inslruction-followingg accuracy in generating responses. However, when using the IFEval dataset, these two factors are entangled, making it difficult to isolate uncertainty estimation from the model's overall instruction-follovinn performance, For example, LLaMA-2-chatte accuracy, tends to make more obvious errors when it fails to follow instructions. On the other hand, LLaMA-2-chat--7B not only makes these obvious mistakes but also exhibits more subtle instruction- following errors, where responses partially follow the instructions but miss specific details. As a result, uncertainty estimation becomes more challenging for LLaMIA-2-chat-7B., where it has to measure uncertainty in more nuanced instruction violations, compared to LLaMA-2-chat-133
which generally has higher instruction mnstructionfoolloiing
To quantify the difference in task difficulty, to 9 based on their adherence to instructions. Table 2 shows that the score gap between correct and incorrect responses is smaller for LLaMA-2-chat-777 compared to LLaMA-2-chat-133 highlighting the more subtle nature of 7B's errors. In contrast, 13B's errors are more drastic, making them easier identify and be recognized as having high uncertainty. This variation in task difficulty across models complicates direct comparisons of their uncertainty estimation abilities. To ensure fair comparisons across models, it necessary to evaluate models under controlled difficulty levels.
GPTT4 to score the responses scale from 0
UNCERTAINTY ESTIMATION ABILITY IN INSTRUCTION ON OUR BENCHMARK DATA
The challenges identified in the previous section lenighh bias, the entanglement of task execution quality with enstruction-follwwiii and varying difficulty levels across wondels-underscorr the need for a controlled and robust framework for evaluating uncertainty estimation. To address these issucs, develop new benchmark dataset comprising versions: Controlled and Realistic version.

---

Published as a conference paper at ICLR 2025
Case
Example
Correct
"Experienced Relinery Operator with years in the chemical industry. Skilled in overseeing relinery op-
crations, maintaining safety protoculs, and optimizing processes. Collaborated with friends at Hanson
Chemicals to euhance efficiency. Proven track record improving production quality."
Incorrect
10 years ations, Demonstrated history of elevating production quality." "
"Prolessional Relinery Operator
chemical industry, Expert managing relinry oper-
Subtle Error
"Experienced Refinery Operator with 5 years in the chemical industry. Proficient in managing refinery
operations, ensuring safety protocols, and enhancing processes. Worked closely with colleagues at Hanson
Chemicals to boost efliciency. Demonstrated history of elevating production quality."
Table 3: Examples of three response types in the Controlled version of our benchmark dataset. Illustrate responses to the prompt: "Write a short resume for a refinery operator with 5 years of experience in the chemical industry. Include the keywords friends and Hanson". The examples show a Correct response that fully follows instructions, an Incorrect response that omits the keywords, and a Subtle Error that includes only one keyword, highlighting varying degrees of instruction- foolowing........
These versions allow for the evaluation of uncertainty estimation under controlled conditions (Con- trolled) and real-world scenarios (Realistic).
3.1 BENCIIARK DATASET FOR CONTROLLED EVALUATION SETUPS
To disentangle the complexities that can obscure uncertainty estimation, we design two distinct versions of the dataset: Controlled and Realistic. The Controlled version neutralizes the influence of token length. Meanwhile, the Realistic version leverages actual LLMM-generated responses that naturally incorporate real-world signals, including length signal, without manual intervention.
In both versions, we use GPT-4 to filter out low-quality responses, ensuring that the uncertainty being measured comes from citructtionfoolowingg not poor task execution (addressing the second challenge in section 2.3). We apply a filtering process using GPT-4 evaluations of task quality of each response on scale 0-9. Only responses that received high task quality score (>8) are included. Also, these datasets enable using the same responses with all models in the uncertainty evaluation task, controlling the difficulty of uncertainty evaluation across models. This allowing for direct comparisons in uncertainty estimation (addressing the third challenge in section 2.3
3.1.1 CONTROLLED VERSION
In this version, we eliminate the length effect, along with ensuring consistent levels of difficulty across responses ensure that the evaluation focuses purely uncertainty estimation. To neu- tralize the impact of token length, we use GPT-4 to generate both correct and incorrect responses with similar token length (see Appendix for the prompt used generate responses). Figure 2 shows the absence of length bias in the token length distribution. We introduce two levels of controlled iifficulty-Controllel sponses; completely incorrect, correct, and subtly off-target. In the Controlled-Easy we calculate AUROC based on distinguishing between correct and completely incorrect responses, while in the Controlled-Hart we calculate AUROC based on distinguishing between correct and subtly off- target responses. These are more challenging cases where the responses only slightly deviate from the instructions, testing the model's ability to recognize subtle mistakes. While these mistakes are subtle, they could still be important in real-world deployments. Table an example from this version. Additional statistics are provided in Table 7 in the Appendix.
and Sontrolled-Hard-be generating three categories of
3.1.2 REALISTIC VERSION
In the Realistic version, we retain the natural length and signals inherent in responses generated by multiple LLMs (LLaMA2-chat-77, LLaMA2-chat-133
Phi-3-mini-1288: and LLaMA2-chat-P 70B). Here, the goal is to evaluate uncertainty estimation methods in a scenario that reflects actual model behavior. In this version, we do not control for token length, allowing for the natural variance found in actual model gener-geeee responses. Though, we still control for task execution quality and provide generated responses enable clear comparisons between models.
Disttal-7B-Instructt-
7

---

Published as a conference paper at ICLR 2025
Medel
Contruled--aay
Unntralte--tard
Realielc
Veeb
Pgl
Seg
Neget)
pio
Entt
Probe
Verb
Pg
Seg
Nepoo
Knt
Prohe
Verb
PM
Seq
Nepo
p(t)
Ent
Probe
0.57
0.:
0:45
0:37
0:32
0.36
0.51
0.48
0.46
LLMM62-78
0.44
0.31
0.54
0:51
0.51
044
Table 4: Average AUC across instruction types for different LLMs and uncertainty estimation methods in three settings. Different uncertainty estimation methods includes Verbalized confidence (Verb), Perplexity (Ppl). Sequence probability (Seq), Normalized p(true) (N-p(t)), p(true), Entropy (Ent), and linear probing on internal states (Probe). Bold values indicate the best-performing method for each model and condition, while underlined values denote the second-hest performing method.
3.2 RESULTS ON OUR BENCHMARK DATA
Table 4 Figure 5 and Appendix Table show findings from our evaluation of uncertainty estima- tion methods on the crafted dataset. By analyzing performance across different uncertainty meth- ods, models, and instruction types, we gain insights into the strengths and limitations of various approaches under both controlled and realistic conditions.
3.2. COMPARISON OF UNCERTAINTY METHODS: CONTROLLED-EASY AND REALISTIC
Probe generally outperformed baseline methods in both Controlled and Realistic ver- sion, as shown in Table 4 Probe consistently outperformed even self-evaluation methods such as berbalized confidence and crrmallized-((ruee)) model "know" and what they are able to express suggests that their internal layers hold richer, more reliable indicators of uncertainty, which are not fully captured in the model's s explicit responses. This points to promising directions for future work, specifically in improving self-evaluation methods by leveraging the rich information within a model's internal representations.
This gap between what the internal states of the
In Probe, middle layers consistently offer the most informative representations for uncertainty estimation, particularly in simpler tasks (Controlled-Easy). Appendix Table I0 shows how Probe performance varies across instruction types and layers within the LLMs. The middle layers consis- tently provide the best signals for uncertainty, indicating that future work could focus on extracting more refined signals from these middle layers to enhance uncertainty estimation methods.
Self-evaluation methods outperform logit-based ones in Controlled-Eass In simpler tasks. self- evaluation methods such verbalized confidence (short answer) and normalized-p p(true) (binary choice) consistently outperformed logit-based approaches like perplexity, sequence probability, and entropy. Furthermore, for models like LLaMA2-chat-725 LLaMA2-chat-13B and Mistral-7B- Instruct, verbalized confidence performed better than normalized p(true), meaning short-form an- swer verbalized scores were more calibrated than binary choices in these cases. This overall trend indicates that for Controlled-Easy tasks, models are better at assessing their own correctness using seff-evaluation methods compared to logit- based uncertainty estimation.
Sequence probability excels, but normalized p(true) remains a strong contender in Realistic In evaluated responses with realistic patterns (includ-
the Realistic evaluation, where models ing length effect), sequence probability performed best across all models, likely due to its ability to exploit the length effect in incorrect responses. However, aside from sequence probability, normal- ized p(true) consistently ranked as the second-best method across all models, outperforming other logit-based methods like perplexity and mean token entropy. This demonstrates that while sequence probability can take advantage of length bias. normalized p(true) offers more balanced and reliable uncertainty estimation when length effect less prominent.
3.2.2 COMPARISON OF UNCERTAINTY METHODS: CONTROLLED-HARD
Probe still struggles with uncertainty estimation more challenging scenarios. In Controlled- Hard, Probe AUROC scores drop below 0.60 across all models, revealing that even the internal representations struggle to estimate uncertainty accurately when the task complexity increases. This suggests limitation in LLMs' ability to handle nuanced complex uncertainty, highlighting the need for further model fine-tuning or the development of more sophisticated uncertainty estimation methods improve uncertainty estimation in these difficult tasks.
8

---

Published as a conference paper at ICLR 2025
Mixed performance between logit-based methods and self-evaluation methods in Controlled- Hard In the esstrccionnfollowiigg samples with more nuanced mistakes, there was a mixed perfor- mance between logit-based methods and Normalized p(true). For LLaMA2-chatee and LLaMA2- chat- 133, logit-based methods like perplexity and entropy outperformed verbalized confidence and normalized p(true), suggesting that task complexity increased, logit-hased methods became more reliable with those models. However, for Distral-7B-Instruce and Phi-3-mini, normalized p(true) continued to provide better uncertainty estimation than logit-based methods, demonstrating that the best method can vary depending on the model and its underlying architecture in Controlled-Hane tasks. Uncertainty estimation scores in Controlled Hard were consistently lower than in other set- tings and no methods had reliably estimated uncertainty in this setting.
3.2.3 COMPARISON OF MODELS
Misttrl-7B-Instruee consistently demonstrates the strongest performance in verbalized confi- dence across all tasks (Controlled-Fasye Controlled- and Realistic), outperforming even the larger LLaMA2-133 13B model, highlighting its effective internal calibration for self-assessment. On the other hand, Phi-3-mini-1223 ROC across both easy and hard tasks in Controlled, as well as in Realistic versions, showcasing its strength in binary choice settings. In contrast, LLaMA-2-13B excels Perplexity, indicating its proficiency in logit-based uncertainty estimation. These findings are particularly interesting be- cause smaller models, such as Misttral-B-IIstruet and Phi-3-mini- 128k outperform the larger --MM----haa-13B in self-evaluation methods. This suggests that factors beyond model size, such as tuning or architecture, may contribute to better uncertainty quantification in certain tasks. However, LLaMA2-13B still shines in logit-based approaches like perplexity, indicating that differ- ent methods may favor different models.
leads in normalized p(true), consistently achieving the best AU-
3.2.4 COMPARISON OF INSTRUCTION TYPES
The relationship between success rate and AUROC varies significantly across instruction types As shown in Appendix Table 9 models can correctly follow instructions but still struggle accu- rately gauge their own uncertainty, as reflected in the original IFEval evaluations. For instruction types like "detectable-comtem' and 'keywords," there is clear positive correlation between higher success rates and higher AUROC scores for uncertainty estimation, indicating that when models excel at following instructions, they also tend to estimate uncertainty more reliably. However, this correlation is weaker for other instruction types, such as 'language', 'startend', and 'change-case'.
In particular, 'punctuation' presents unique challenge. In these tasks, models are instructed to not to use any punctuation in responses. Although this instruction type had the lowest success rates across all 0.2e (7B), 0.14 (13B), 0.17 (Mistral), and 0. 10 (Phi), 'punctuation' yields relatively high AUROC scores, especially in Controlled version with verbalized confidence. This discrepancy may arise because models recognize when they fail to follow the instructions but strug- gle to adhere to them due to strong priors in their training data. Since most training data includes proper punctuation, instructions to avoid it may conflict with learned patterns, making it difficult for models to follow the instructions while still being somewhat aware of their failure. These findings highlight that while success rates provide some insight into model performance, they do not fully capture the ability to estimate uncertainty in instruction-follovingg
RELATED WORK
Uncertainty estimation in LLMs. Existing uncertainty estimation methods can be broadly cate- gorized into four types based on the source of information: verbalized, based, multi-sample, and probing- based methods. Among them, verbalized methods (Lin et al. 2022, Xiong et al. 2023] Tian et al.) 2023) rely on model's self-evaluation by prompting LLMs to explicitly express their uncertainty in theiroutput. Logit-based methods, such as perplexity (Jelinek et al. 1977). sequence probability (Fomicheva et al. 2020), and mean token entropy (Fomicheva et al. 2020) mainly uti- lize information from the next token prediction distribution. Multi-sample methods (Aichberger et al. 2024 Kuhn et al. 2023 Farquhar et al., 2024) generate multiple responses for the same question, estimating uncertainty through the semantic diversity among the responses. However, these multi-sample methods are less applicable instructionn-ollooiing tasks, which only focus

---

Published as a conference paper at ICLR 2025
strict adherence to instructions rather than variations in semantic meaning. Lastly, probing-based methods (Liu et al., 2024, Ahdritz et al. 2024) train external supervised model on model repre- sentations to infer uncertanty. In addition, it is worth noting that most existing works focus on factuality-related tasks such as question answering (Xiong et al. 2023 Tian et al. 2023) and sum- marization tasks (Kuhn al. 2023), with little attention on Instruction-tuning tasks. Our work seek to bridge this gap by evaluating how well current uncertainty metries capture uncertainties specific to inslluction-folloiing scenarios.
Instruition-follovinng in LLMs Recent studies have introduced benchmark datasets to evaluate the cistruction-followinn capabilities of LLMs, which includes assessments of general instruction-
following (Zhou et al. [023]] Zeng et al. [023] Qin et al., (222). refutation tasks (Yan et al. 2024)) and format adherence (Xia et al. 2024). Among them, we chose the IFEval dataset (Zhou et al.)
2023) as the foundation for our work due to its objective evaluation framework, which uses verifiable
instructions to minimize ambiguity in assessment criteria. Additionally, several methods have been
explored to enhance pnatuuction-folloiing performance (Zhang et al. 2023al He et al. 2024 Sun
ct al. 2024). They highlight the growing interest in LLMs' ability in instruction-followingg
Benchmark datasets for evaluating LLM-as-evaluator Recent benchmarks (Zeng et al., 2023
Zheng et al., 2024 Zhang et al.) 2023b; Wang et al. 2023b) focus on evaluating LLMs' ability to act as evaluators, assessing how well they compare responses in instruction instruction-followingg
following. However, our comparing vari- ous baseline uncertainty estimation methods under controlled conditions. While previous research addresses evaluation biases when LLMs are used as avaluators-suce as sensitivity to presentation order (Wang et al. 2203bb Pezeshkpour & Hruschkal 2023) -our benchmark uniquely tackles the challenges posed by entangled factors in evaluating uncertainty estimation ability of LLMs, such as length bias, task execution quality, and varying difficulty levels.
work differs by concentrating on estimating uncertainty in cestruction-follwviiin
5 CONCLUSION
In this paper, we conduct the first comprehensive evaluation of uncertainty estimation in LLMs specifically in the context of instruction-folloiing tasks, addressing a gap in existing research that primarily focuses on fact-based tasks. We identify limitations associated with existing benchmark datasets and introduce new benchmark with two Pessions-Controlie and Realistic-designeel to provide a comprehensive framework for evaluating uncertainty estimation methods and models un- der both controlled and real-world conditions. Our analysis revealed that verbalized self-evaluation methods outperform logit-based approaches in Controlled-Easy tasks, while internal model states provide more reliable uncertainty signals in both Controlled-Eand all methods struggle with more complex tasks in Controlled Hard,, highlighting the limitations of LLMs and future direction for uncertainty estimation in instruction followwi...........
and Realistic settings. However,
Limitations and Future Work One limitation is the narrow scope of instruction types and domains included in the benchmark, which may not fully capture the diversity of real-world tasks. Further- more, similar other research evaluating L.Mss, there is potential risk of leakage, where the models may have been exposed to similar tasks during pre-training, potentially affecting the results. In future work, expanding the benchmark to include a broader range of domains and evaluating more LLMs would furthen deepen understanding of uncertainty estimation in instruction-followiii Additional analysis could also investigate why LLMs tend to fail to provide accurate uncertainty estimates in instruction-folloon following, which could lead to the development of trustworthy Al agents.
ACKNOWLEDGMENTS
This work was conducted during an internship at Apple AIML We sincerely thank Sinead Williamson and Udhay Nallasamy for their valuable feedback and insightful suggestions on this work. We are also grateful to Guillermo Sapiro for his unwavering support and guidance throughout the research.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re-
10

---

Published as a conference paper at ICLR 2025
port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
Gustaf Ahdritz, Tian Qin, Nikhil Vyas. Boaz Barak, and Benjamin L Edelman. Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.0356. 2024.
Lukas Aichberger, Kajetan Schweighofer, Mykyta lelanskyi, and Sepp Hochreiter. Semantically diverse language generation for uncertainty estimation in language models. arXiv preprint S4Kv::2406.43000 2024.
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. Lm- polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:231 2023.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, (00(801))625-660 2024.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguisties, 8: 539-555, 2020.
Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang and Yanghua Xiao. From complex simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint wwxiv:2444.1584 2024.
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity-4 measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06255, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds Nova DasSarma, Eli Johnson,, et Language mod- els (mostly) know what they know, arXiv preprint urXiv:2207.05221, 2022.
Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, and Chanjun Park. Evalverse: Uni- fied and accessible library for large language model evaluation. arXiv preprint arXiv:2404.00943. 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. arXiv preprint arXiv:2302.000 22200.00.44....... 2023.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Sun, et al. Personal 11m agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXhv:2401.05455, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv: 20....55..444... 2022.
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404. 159993 2024.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau, M. Brucher, M. Perrot. and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830. 2011.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of op- tions in multiple-choice questions, arXiv preprint arXiv:2308. 1483, 2023.

---

Published as a conference paper at ICLR 2025
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601. 2024.
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instraction-follooingg ability of large language models. arXiv preprint arXiv:240..02823, 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint 13XYv:2305..4975. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.09288 2023.
Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. Towards conversational diagnostic al. arXiv preprint arXiv:2401 .05654, 2024.
Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with Ilms. arXiv preprint arXiv:2305. 1792, 2023a.
Peiyi Wang, Lei Li. Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023b.
Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. Fofo: A benchmark to evaluate Ilms' format-following capability, arXiv preprint arXiv:2402.18667 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13065 2023.
Jianhao Yan, Yun Luo, and Yue Zhang. Refutebench: Evaluating refuting cntructiionfollowing for large language models. arXiv preprint arXiv:2402. 13463, 2024.
Fanghua Ye, Mingming Yang. Jianhui Pang. Longyue Wang, Derek F Wong, Emine Yilmaz, Shum- ing Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. arXiv preprint 2401 2024.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where attend: Post-hoc attention steering for llms. arXiv preprint arXiv:2311.02262 2023a.
Xinghua Zhang. Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper IIm networks are fairer Ilm evaluators. arXiv preprint 00t.::2300.0186222 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llueee-a-jude with mt- and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DETAILED RESULTS ON IFEVAL DATASET
In the main paper, Table presented the average AUROC scores across all instruction types, pro- viding an overview of how well the baseline uncertainty estimation methods perform instruction- following tasks. This section provides detailed AUROC results for each individual instruction type the EvEal dataset in Table
IFEval
AUC uncertainty method
SR
Model
Instructions
Verbal
Ppl
Seq
N-p(t)
p(t)
Ent
startend
0.58
0.33
0.66
0.47
0.50
0.36
0.58
detectable content
0.51
0.37
0.60
0.39
0.56
0.37
0.89
detectable.format
0.51
0.46
0.62
0.46
0.51
0.50
0.68
language
0.51
0.49
0.61
0.48
0.53
0.52
0.58
LLaMA2-
change_case
0.55
0.48
0.60
0.50
0.50
0.52
0.52
chat-13B
keywords
0.53
0.50
0.61
0.49
0.50
0.53
0.71
conttacconss
0.53
0.46
0.59
0.50
0,49
0.51
0.48
punctuation
0.53
0,45
0.58
0.50
0,48
0,51
0.14
Average
0.53
0.44
0.61
0.47
0.51
0.48
0.57
startend
0.55
0.30
0.56
0,57
0.52
0.32
0.67
detectable_content content
0.48
0.25
0.55
0.48
0.47
0.29
0.85
Aotrrta format
0.55
0.47
0.49
0.51
0.52
0.47
0.66
language
0.53
0.50
0.50
0.51
0.49
0.51
0.68
LLaMA2-
chat-7B
change_com _case
0.52
0.49
0.55
0.54
0.52
0.50
0.48
keywords
0.53
0.48
0.57
0.53
0.52
0.49
0.68
length_constee constraints
0.47
0.58
0.52
0.50
0.48
0.46
punctuation
0.47
0.57
0.53
0.50
0.49
0.24
Average
0.53
0.43
0.54
0.52
0.51
0.44
0.59
startend
0.48
0.70
0.51
0.57
0.51
0.63
detectable content
0.51
0.39
0.63
0.61
0.43
0.56
0.79
sioctaale_forman
0.46
0.49
0.54
0.61
0.43
0.54
0.78
language
0.46
0.50
0.50
0.57
0.47
0.49
0.87
Mistral-7B-
Instruct-v0.3
change_cuse
0.49
0.48
0.53
0.58
0.47
0.48
0.62
keywords
0.50
0.48
0.52
0.57
0.47
0.47
0.73
fnngh_constraint.
0.50
0.49
0.53
0.56
0.47
0.49
0.55
punctuation
0.51
0.50
0.52
0.56
0.48
0.49
0.17
Average
0.50
0.48
0.56
0.57
0,47
0.50
0.64
startend
0.62
0.47
0.60
0.53
0,49
0.38
0.22
detectable content
0.53
0.35
0.41
0.56
0,46
0.46
0.89
detectable_
0.53
0.39
0.45
0.59
0.42
0.42
0.67
language
0.49
0.43
0.47
0.53
0.41
0.48
0.97
Phi-3-mini- 128k-instruct
change_cuse .cuse
0,50
0.45
0.48
0.56
0.43
0.46
0.29
keywords
0.51
0.45
0.49
0.55
0.48
0.46
0.75
length_constrainn
0.51
0.45
0.49
0.56
0.47
0.47
0.41
punctuation
0.51
0.45
0.49
0.56
0.48
0.48
0.11
Average
0.43
0.48
0.55
0.45
0.45
0.54
Table 5: Detailed AUROC for baseline uncertainty estimation methods applied to individual instruction types in the EvEl dataset (Zhou al.) 2023). This table presents AUROC scores for six uncertainty estimation methods (verbalized confidence, perplexity, sequence probability, normal- ized p(true), p(true), and Entropy) across four LLMs all instruction types in the IFEval dataset. The AUROC evaluates how well each method's uncertainty estimates align with the ground truth on correct incorrect cestruction-folloiing ability, N-p(t) is normalized p(true), p(t) refers to p(true), and Ent indicates entropy. Success Rate (SR) represents the model's instruction-folloving accuracy, calculated using IFEval's evaluation function. These detailed results reveal variations in performance across instruction types, providing deeper insights into the challenges of estimating uncertainty in intruction-followinn tasks.
Ppl represents perplexity, Seq stands for sequence prob-
A.2 PROMPT OF UNCERTAINTY ESTIMATION METHODS
To evaluate uncertainty in cnatuuction-ollloiinn tasks, employed several baseline methods, in- cluding self-evaluation methods (verbalized confidence, normalized p(true), and p(true)) and logits- based methods (perplexity, sequence probability, and mean token entropy). For self-evaluation meth-
13

---

Published as a conference paper at ICLR 2025
ods, used specific prompts engage the models in assessing their uncertainty. Below, provide the exact prompts used our experiments for both verbalized confidence and p(true) meth- ods.
Prompt for Verbalized Confidence is used to assess the model's self-reported confidence, asking the model to score its perceived likelihood that its response follows the given instructions. The model instructed to provide confidence score ranging from 0 (no confidence) to (high confidence).
Prompt for p(true) and Normalized p(true) evaluates the model's probability estimate of whether its previous response correctly followed the instructions. The model is prompted to assign probabili- ties to binary choice, with p(true) reflecting the likelihood that the response is correct. Normalized p(true) adjusts the raw probabilities of both options to account for biases, normalizing the confidence scores.
Prompt for verhalized confidence
You are helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to score given output for the given instruction.
Score the output for the given instruction. The output is generated by an Al chatbot. You should give an overall score (an integer) on a scale of 0 to 9, where higher score indicates better overall performance.
Do NOT provide any explanation for your evaluation. Your response should be ONLY the score, integer between 0 and 9.
# Instruction: finput)
# Output: finstruction following response)
Score of the Output (Your response should be ONLY the score, integer between 0 and 9):
Prompt for p(true) and Normalized p(true)
You are helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to score given output for the given instruction.
Question: (instruction input)
Response: (instruction following response)
Does the response: (A) Follow instructions (B) Not follow instructions.
The answer is:
14

---

Published as a conference paper at ICLR 2025
A.3 PROMPTS FOR GENERATING OUR CONTROLLED BENCHMARK DATASET
To create the Controlled version of benchmark dataset, we employed carefully designed prompts that generate responses with controlled token lengths, varying difficullty levels, and high task quality. The aim was to ensure that the evaluation focuses solely on uncertainty from instruction-follwiiii without interference from factors such as task execution quality or response length. Below, we provide the full prompts used to generate both subtle off off-target and completely wrong responses, as well as the prompt for scoring task quality or instruction-folloving
Prompt for generating subtle off-target responses
I want generate synthetic data to test LLM's ability to evaluate instruction following. There are three types of responses in instructionffolooinng: correct, subtly off-target, totally wrong. Your task involves generating subtly off-target responses that are subtly disregard the instruction. Subtly off-target response should be classified as false according to pre-defined evaluation functions based on evaluation focus provided below.
For example, if the instruction is "Make sure to include at least 10 placeholders.." and kwargs is "number-placeholder 10" then subtly off-target response might include only placcholders. Another example, the instruction is "Make sure to include keywords: love, hate" and kwargs is "keywords:love, hate", then subtly off-target response may include only one of them.
Make sure the length of your generated subtly off-target response matches the length of the correct response provided below, aiming for similar level of detail and complexity.
# Instruction: [Instruction-anne and--ask input)
Evaluation focus: [Type}
# Details of evaluation criterion: (Kwargs}
Correctly followed response: (Correct Response}
# Subtly off-target response:
Prompt for generating totally wrong responses
I want generate synthetic data to test LLM's ability to evaluate instruction following. instruction-ooloviin correct, confusingly wrong,
There are three types of responses totally wrong. Your task involves generating totally wrong responses that are clearly disregards the instruction. Your totally wrong response should be classified as false according pre-defined evaluation functions based evaluation focus provided below.
For example, if the instruction is "Make sure include at least 10 placeholders..." and kwargs is "numbes-pacceooldes::100 then wrong response might include only placeholders. Another example, if the instruction is "Make sure to include keywords: love, hate" and kwargs is "keywords:love, hate", then wrong response may include none of them.
Make sure the length of your generated confusing response matches the length of the correct response provided below, aiming for a similar level of detail and complexity.
# Instruction: [Instruction-and-tase input)
Evaluation focus: {Type}
# Details of evaluation criterion: (Kwargs)
# Correctly followed response: (Correct Response)
# Confusingly wrong response:
15

---

Published as a conference paper at ICLR 2025
Prompt for generating subtle off-target responses
[Instruction-aaddtaass input)
Prompt for scoring task quality
You are helpful assistant evaluating the quality of the outputs for a given instruction. Your goal is to score given output for the given instruction. You should give an overall score (an integer) on a scale
9, where higher score indicates better overall performance. Do NOT provide any explanation for your evaluation.
Instruction: (Task-only-input)
# (Resposse
# Score the Output (Your response should be ONLY the score, an integer between O and 9):
Prompt for scoring instruction following
You helpful assistant in evaluating how well given response follows given instruc- tion. Your goal is to score a given output for the given instruction. You should give an overall score (an integer) performance. Do NOT provide any explanation for your evaluation.
scale of 0 to where higher score indicates better overall
Instruction: canpuucciiooool-iiipppei
# Output: (espoosse)
Score the Output (Your response should ONLY the score, an integer between 0 and 9):
A.4
WHY DO WE CHOOSE IFEVAL DATASET?
Here, we would like emphasize why we choose IFEval as our primary dataset instead of using real-world dataset with different contexts and domains.
First, we select IFEval to focus on our scope which is 'single, simple, and non-ambiguous instruc- tions'. Real-world datasets often involve complex, ambiguous, multi-instruction prompts, which can conflate multiple factors affecting uncertainty. For this first systematic evaluation, we chose to focus single, simple, and verifiable instructions ensure clarity and isolate the uncertainty esti- mation process. The IFEval dataset is well-suited for this purpose, as provides 25 distinct types of simple and clear instructions that align with our goal of establishing robust baseline.
Second, we want to avoid evaluator-induced uncertainties. Most real-world tasks and benchmark datasets rely on LLM-based evaluators determine whether a response follows an instruction. However, LLM-based evaluators may introduce their own uncertainties or make errors in assess- ing success or failure, which could obscure the true uncertainty estimation capabilities of the tested models. The IFEval dataset avoids this issue by including instructions with deterministic evaluation programs that objectively verify compliance. For instance, an instruction like "please do not include keywords: ..!" can be automatically validated using a simple program to check for the presence of those keywords. This feature eliminates ambiguity evaluation and allows us directly focus on LLMs' ability to estimate uncertainty.
Our main contribution the careful design of benchmark data specifically tailored to 'uncertainty estimation' in instruction-follovii contexts. We believe that those underlying considerations (e.g.. length bias, task quality entanglement, and varying levels of model difficulty) presented our study extend to future instruction-followt following datasets. While IFEval serves ideal starting point for this research, we hope our framework inspires future efforts to tackle uncertainty estimation in more complex, real-world tasks.
16

---

Published as a conference paper at ICLR 2025
A.5 IFEVAL DATA EXAMPLES
Table 6 presents examples from the IFEval dataset, such as tasks like writing a resume creat- ing joke about programmers. The instructions assigned to each task vary, requiring the model to follow specific guidelines such as including excluding certain keywords, ensuring word us- age meets a specific frequency, and adhering to formatting rules. The keywords that must be in- cluded excluded differ based the task. For instance, in the resume task, keywords might include "resume", "software", or "engineer", whereas in the joke task, the focus may shift to terms like "syntax" or "code". These varied instructions introduce diverse challenges for the model in instruction-folloinng
Type
Example
Task
Write resume for software engineer with 5+ years of experience in the Bay Area, CA.
keywordssexistence
Make sure to include the keywords: "skills", "technology" "career".
key deywords:orrbidenn
Do not include the following keywords: resume, software, engineer, experience.
kerwords.yryquecy
Instruction
Make sure to use the word "qualifications" at least times.
startend:end checker Your
end with the exact phrase "Looking forward contributing innovative projects."
detectable content:number placeholders Make sure to include at least 5 placeholders represented square brackets, such as [name].
Task
Write a joke abou programmers.
keywords:existence
Make sure to include the keywords: "humor", "code", "life".
keywords:forbidden
Do include the following kcywords: joke, programmers,
freeurrs:rrqueecc
Instruction
Make sure to use the word "syntax" at least times.
startend:end checker
Your programmer joke must end with the exact phrase "And that's the real bug in the code of life."
detectable content:number placeholders
Make sure to include at least 3 placcholders represented by square brackets, such as [name].
Table 6: Examples from the IFEval dataset. This table shows two tasks: writing resume and crafting joke about programmers. Each task paired with multiple instruction types, such as including/excluding keywords, ensuring word frequency, and adhering to specific content formatting rules.
A.6 STATS OF OUR BENCHMARK DATA
This section provides comprehensive statistics for our benchmark datasets, detailing the number of data points and token length distributions. These statistics illustrate the construction and character- isties of both the Controlled and Realistic versions of our benchmark dataset.
Figures 3a and Figures 3p show the token length distributions for the Controlled and Realistic datasets, respectively, highlighting the absence of length bias in the Controlled version and the presence of natural length variation in the Realistic version. Figures 4h and Figures 4p display the model contribution to total correct and incorrect responses the Realistic version, ensuring diverse data sources. Table presents the number of correct and incorrect responses in both the Controlled and Realistic versions. Table [ provides the mean token lengths across different instruction types, underscoring the careful balancing of token lengths in the Controlled version.
17

---

Published as a conference paper at ICLR 2025
Rea Listic version
led version
Correct Sublle-riataaee
Proportion
245
400
6K
1000
Token
Leng th
Length
(a) Length in Controlled
(b) Length in Realistic
Figure 3: Token length distributions for the Controlled and Realistic versions of our bench- mark dataset. The distributions are normalized by the total number of responses in each class. (a) Token length distribution in the Controlled version, where token lengths are carefully balanced be- tween correct and incorrect responses. (b) Token length distribution the Realistic version, where token length reflects the natural variability of model-generated responses.
Model contribution Correct responses Phu sma 228k-metuu
Model contribution Wrong responses
18.74
26.7%
25.8%
Misooi-PS PYSTO
233b-cale
27.9%
Jame chein!
(a) Realistic: Correct
(b) Realistic: Wrong
Figure 4: Model contributions to the Realistic version of the benchmark dataset. (a) Pie chart representing the model contribution to correct responses. (b) Pie chart representing the model contri- bution to incorrect responses. These contributions ensure that the Realistic dataset captures diverse responses from various LLMs.
Controlled
Realistic
wrong subtle off-target
wrong
startend
49
60
detectable content
43
45
21
15
detectable_Formaa
language
change case
61
78
74
60
61
keywords
127
134
109
102
108
length constraints
punctuation
20
12
Sum
378
609
367
474
485
Table 7: Summary of the number of data points for correct and incorrect cases in both the Controlled and Realistic versions. One example may contain multiple instructions, so the total number of examples is less than the sum of data points across all instruction types.
18

---

Published as a conference paper at ICLR 2025
Cantrolld
Realistic
wrong
confusing
wrong
startend
242.81
216.29
267 .4
410.08
243.03
detectable cuntent
282.72
270.15
272.27
366.86
296.20
detectable Format
244.69
244.16
233.14
349.14
320.44
language
91.37
115.75
144.57
238,00
140.29
change case
291.04
247.73
273.82
keywords
232.47
247.34
247.34
315,70
254.56
length constraints
336.82
punctuation
142.50
182.54
177.81
173,70
50.75
Average
Table 8: Mean token length for each instruction type in both the Controlled and Realistic versions of the dataset. The Controlled version neutralizes the impact of token length, while the Realistic version reflects natural length variation.
19

---

Published as a conference paper at ICLR 2025
A.7 MORE RESULTS ON OUR BENCHMARK DATA
In this section, provide additional results uncertainty estimation performance using crafted benchmark datasets. These results offer deeper insights into how different LLMs and uncertainty estimation methods perform across various evaluation scenarios, both controlled and realistic. The radar charts and tables present a detailed comparison models' performance, using AUROC av- eraged across instruction types for different uncertainty estimation methods. Figure [nd and Figure shows the radar charts on IFEvl data and our benchmark data. Table 9 provides detailed AUROC scores for each instruction type.
(a) IFEval data
(b) Controlled -asy
Figure 5: Model comparison of uncertainty estimation across different evaluation scenarios. Radar charts illustrate the performance of four LLMS on six uncertainty metrics, with AUROC averaged across instruction types. (a) Results based on IFEval data. (b) Results on Controlled- -asy version of our crafted data (distinguishing correct and incorrect responses).
Average
Average
(a) Controlled-Han
(b) Realistic
Figure 6: Model comparison of uncertainty estimation across different evaluation scenarios. Radar charts illustrate the performance of four LLMs six uncertainty metrics, with AUROC averaged across instruction types. (a) AUROC on the Controlled Hard (distinguishing correct and subtle off- target responses) and (b) AUROC on the Realistic version of our crafted data.
20

---

Published as a conference paper at ICLR 2025
Contrelled-Eay
Controlled-Hard
Realistic
Model
Indructions
Verbol
Ppl
N-pO
Ent
Verbal
Pyl
Sey
N-p0))
p(u
Ent
Vertul
Pyl
Seq
N-pt0
pto
Eat
carkend
0.65
0.5k
0.50
0.67
0:44
0.58
0.52
0.58
0.54
0.56
0.43
0.59
0.52
0:49
0.67
0.53
0.54
0.40
0.57
0.63
0.49
0.64
0.45
0.59
0.50
0.50
0.51
0.54
0.44
0.56
0.52
0.53
0.67
0.54
0.52
0.44
0.06
8.22
0.68
0.39
8.45
0.59
0.1
0.61
0:32
855
0.43
0.58
0.51
0.53
0.62
0.54
0.46
0.03
LLaMA2-
shanee cae
Ingrrage
0.87
0.66
8.4
0.66
0.39
0:44
0.00
8.5
081
0:31
8:1
0.51
0.42
0:57
0.57
0.50
0.5L
0:52
0.31
0662
0.33
0.1
0:46
0.67
keywoods
0.58
0.36
0.52
0.31
0.1
bnglt,comettinns
0.51
0.52
0.31
0.53
carterl
0.64
0.59
0.90
0.55
0.43
0.55
0.52
0.53
0.54
0.49
0.43
0.56
0.41
0.52
0.74
0.54
0.44
0.46
0.05
0.63
0.59
0.53
0.41
0.56
11.22
0.58
0.52
0.49
0.43
0.55
0.43
0.55
0.72
0.52
0.42
0.50
0.61
0.62
0.88
0.53
0:44
0.57
10.51
0.58
0.52
0.51
0.45
0.50
0.50
0150
0.60
0.54
0.48
0.51
LLMM
chat-78
languge
leywords
change cse
0.83
0.63
0.53
061
0.62
0.60
0.47
0.46
0.52
0:55
0.54
0.43
0.46
0.54
0.52
0.84
1..1
0.52
0.59
0.59
0.57
0.51
0:51
0.50
0.53
0.50
0.51
0.47
0.45
0.45
0.52
0.53
0:52
0.51
0.49
0.50
0:50
0.49
0.50
0.57
0.57
0.61
0-53
0:53
0.53
0:49
0.48
0.48
0.50
0.50
0.50
0.66
0.55
0.57
0:51
0.53
0.47
0.5%
0.51
0:50
0.56
0:51
0-48
pusttuatioss
0.65
0.56
0.55
0.51
0.51
0.47
0.55
0.51
0.57
0.51
0.48
0.51
Arerage
0.54
0.77
0.31
0151
0.35
0:31
033
0:41
nartedd
0.69
0.54
0.54
0.51
0.56
0.46
0.99
0:40
0.52
0.55
0.51
desectable content
0.70
0.49
0.54
0.48
0.53
0:43
0.49
0.53
0.53
0.71
0.6
0.44
0.66
0.4%
0.45
0.56
0.57
0.49
0.57
0.48
0.48
0.51
0.48
0.62
0.52
0.51
0.49
bastruc v0.3
Misual 7B
Inguug
change.care zore
0.10
0.71
0.57
0.61
0.63
0.41
0.66
0.67
0.49
0.49
0.42
0.41
0..56
11.6
0.55
0.54
0.48
0.48
0.54
0.56
0.49
0.50
0.45
0.44
0.50
0.53
0:46
0.47
0.58
0.62
0.51
0.51
0.51
0.51
0.46
0.45
krywords
0.72
0.58
0.65
0.66
0:49
0.60
13.88
0.54
0
8.55
0.50
0.45
0.50
0:48
0.5K
0.51
0.51
0.47
langth coustrainns
0.73
0.58
0.45
0.68
0.45
0.41
11.99
0.55
0,40
0.57
0.50
0.45
0.52
0.49
0.57
0.50
0.51
0.45
puncouation
0.73
0.58
0.46
0.67
0.49
0.41
0.58
0.55
0,49
0.57
0.49
0.45
0.52
0.49
0.57
0.51
0.50
0.48
Average
0.71
0.39
0.14
0,56
0.45
0.33
1.36
0.35
0.39
0.36
0.13
0.36
0.30
0746
0003
0.31
0.31
0.45
putted
0.48
0.53
0.45
3 mini 120k anstrect
Ingrage
0.53
0.62
0.50
0.47
0.47
0.59
0.1
0.55
0.52
0.63
0.85
0.44
0.42
0.54
0.56
0.55
0.51
0.5
keywords
0.63
0.59
0.39
0.72
0.59
0.54
0.36
0.52
0.48
0.1
0.52
0.52
0.50
0.42
0.60
0.53
0.53
0.50
Inngth constrainos
0.63
0.58
0.41
0.73
0.55
0.53
0.57
0.53
0.48
0.6
0.52
0.00
0.50
0.44
0.58
0.53
0.52
0.51
pancouation
0.65
0.57
0.43
0.73
0.58
0.54
0.57
0.53
0.48
0.63
0.52
0.52
0.50
0.43
0.58
0.54
0.52
0.51
Average
0.54
0.38
0223
0.72
0.56
0.5
1.35
051
0.47
062
0148
0.32
1142
0.56
0131
015
Table 9: AUC for each instruction type for different LLMs and uncertainty estimation methods across three settings. The uncertainty estimation methods include Verbalized confidence (Verb), Perplexity (Ppl), Sequence probability (Seq), Normalized p(true) (N-p(U)), p(true), Entropy (Ent),
and linear probing on internal states (Probe). Bold values indicate the best-performing method for each model and condition, while underlined values denote the second-bes performing method.
A.8 MORE RESULTS ON INTERNAL STATES OF LLMS
This section presents additional results on the effectiveness of linear probing (Probe) on the internal states of different LLMs across various instruction types. As outlined in the main paper, we applied linear models to early, middle, and late layers of each model to determine how well internal states capture uncertainty in follooin tasks.
Table 10 provides the AUROC performance of probes across different instruction types and layers for several models. Results are shown for both linear and projection based methods. Figure 77 these results through heatmaps, showing performance across early,
Figure X and middle, and late layers for each instruction type.
21

---

Published as a conference paper at ICLR 2025
Controlled-Eas
Controlled-Hard
Realistic
Model
Instructions
Early
Middle
Last
Early
Middle
Last
Early
Middle
Last
startend
0.74
0.88
0.81
0.53
0.48
0.39
0.74
0.83
0.80
detectable_ct content
0.86
0.81
0.57
0.52
0.44
detectable format
0.87
0.84
0,84
0.57
0.63
0.51
0.54
0.58
0.51
language
0.98
0.84
0.97
0.97
0.89
0.81
1.00
1.00
0.50
LLaMA-2-chat
change case
0.77
0.69
0.63
0.50
0.51
keywords
0.78
0.80
0.82
0.53
0.64
0.59
0.54
0.55
0.46
length construints
0.83
0.80
0.71
0.62
0.57
0.54
0.63
0.59
0.56
punctuation
0.58
0.62
0.54
0.51
0.34
0.36
0.78
0.75
1.00
Average
0.79
0.76
0.58
0.52
startend
0.73
0.56
0.56
detectuble_comet content
0.88
0.92
0.88
0,47
0.45
0.45
0.93
0.73
0.44
detectable_ format
0.74
0.73
0.71
0.53
0.50
0.48
0.57
0.58
0.59
language
0.76
0.75
0.71
0.86
0.72
0.58
0.67
0.50
LLaMA-2-cha-7B
change case
0.63
0.51
0.54
0.49
keywords
0.58
0.67
0,70
0.50
0.44
0.48
0.56
0.61
0.46
Longth.constraing
0.77
0.75
0.75
0,54
0.51
0.55
0.53
0.42
0.51
punctuation
0.67
0.55
0.44
0.49
0.44
0.44
Average
0.72
0.72
0.67
0.52
0.31
0.50
0.61
0.60
startend
0.80
0.79
0.76
0.33
0.43
0.54
0.54
0.53
0.76
detectable_comteg
0.67
0.80
0.88
0.47
0.21
0.38
0.81
0.63
0.60
detectuble formal
0.75
0.75
0.79
0.58
0.68
0.51
0.52
0.59
0.59
language
0.83
0.94
0.81
keywords
0.57
0.59
0.66
0.61
0.50
0.58
0.44
0.48
0.43
change_case
0.73
0.87
0.85
0.49
0.58
0.58
0.59
0.66
0.58
length constraints
0.82
0.86
0.84
0.61
0.73
0.65
0.49
0.60
0.53
punctuation
0.47
0.48
0.49
0.41
0.78
Average
0.73
0.75
0.77
0.56
0.56
0.58
0.65
0.66
0.62
startend
0.68
0.82
0.78
0.48
0.47
0.45
0.80
0.89
0.93
detectuble detettublc.comtett content
0.78
0.84
0.95
0.47
0.45
0.45
0.93
0.81
0.92
detectable_forme
0.61
language
0.72
0.97
0.90
0.54
0.56
0.86
1.00
1.00
0.67
Phi-3-mini- Moriini-12kk-inme 128k-instruct
change_case
0.55
0.60
0.61
0.54
0.50
0.63
0.46
0.51
0.46
keywords
0.61
0.88
0.85
0.43
0.46
0.65
0.72
0.49
0.43
length constraints
0.58
0.84
0.85
0.60
0.73
punctuation
0.65
0.57
0.47
0.49
0.64
0.47
0.44
0.89
0.67
Average
0.67
0.79
0.78
0.51
0.33
0.60
0.68
0.72
0.66
Table 10: AUROC performance of linear probing (Probe) applied to internal states of early, middle, and late layers for various models and instruction types. Results are reported for both linear and projection methods, showing that middle layers generally offer more informative representations for uncertainty estimation.
AUEOC Layer
ALROC Midds
ALAOC Lats
AM
0.0
C.
6M
673
618
GM4
173
6.3
675
G7
6/
61
5:5
618
821
Figure 7: Controlled-Easy: Heatmaps showing the AUROC performance of probes on early, middle, and late layers for different instruction types across various L.MMs.
22

---

Published as a conference paper at ICLR 2025
44
128
629
GM
62
831
1B
GH1
624
512
241
1M4
640
6.1
AMI
Figure 8: Hontrolled-Hard: Heatmaps showing the AUROC performance of probes on early, middle, and late layers for different instruction types across various LLMss.
037
602
5:18
5.3
G28
472
6.5
Figure 9: Realistic version: Heatmaps showing the AUROC performance of probes on early, middle, and late layers for different instruction types across various L.LMs.
A.9 LENGTH EFFECT MODELS
This section analyzes the impact of token length on uncertainty estimation across different instruc- tion types and models. We examine the relationship between token length and correctness in both correct and incorrect responses, revealing prevalent length signal in existing instruction-folloiin datasets like IFEval. This effect can introduce biases in uncertainty estimation methods that rely on token length, complicating accurate assessments of model performance.
Table 11 presents the mean and standard deviation of token lengths for both correct and incorrect responses across four LLMs. The results show that, on average, incorrect responses are consistently longer than correct ones, further underscoring the influence of token length on uncertainty estima- tion. Figure 10 provides a visual representation of the token length distributions for three models: ALaMA-2-chate 13B, Mistral-7B Fiatral--B-Ins--O.... Inst-vO.3, and Phi-3- 128k-intt... The figure illustrates how incor- rect responses tend to be longer than correct responses, reinforcing the presence of a length signal in naturally generated datasets like IFEval al. 2023).
23

---

Published as a conference paper at ICLR 2025
respomes tonken)
Hattral-U--Indi-tte
Phi-S-minn 1nik-nvrrac
cotrect
cscrect
correct
startend
2:0
105.36
314.00 2: 115..4
154.75
JAN
213
301
124,00 07
1031 0: 220-87
detectuble
118 06.880
100//0
127.34
83.71
901-71
153.
81.38
119. 119.33
74.30
150110 170.64
changr.oe zo1e
010077
12k
135.33
179-42
965. 11
171.00
970.75
133
102 08
159106
190
179.00
56t 57
157.87
0911
706.32
keywerdss
272
179.54
276.84
164.55
252
185.00
250.62
172.52
253.00
248.16
164.24
000000 288.57
680000 285.86
Ieggh, pusstraints
171/00
09
1038
puactuation
204.70
2000.0
Average
241.84 181.00
236.67 4: 197.9
276.22 100.57
267.50 177.74
250.08 180.78
267.00
178.66
680 240.22
015.80 220.55
Table 11: Mean and standard deviation of token lengths for correct and incorrect responses across four LLMs, broken down by instruction type. The consistent length difference between correct and incorrect responses indicates that length bias is prevalent naturally generated responses, poten- tially affecting uncertainty estimation.
Llane-2-chat-330
Mistral-78-Tnst-oo
Phi-3-128k-inst
Carrect
Gerrect
Dicarrect
Proprrtos
tangth
largts
Langth
(a) Gaam--2-33-chhe
(b) Mistral-7B-Inst-ve
(c) Phi-3-128k-inss
Figure 10: Token length distributions for ILaMA-2-chat- 13B, Mistral-7B-Inst-v0.33 and Phi-3- 128k-inst. The distributions are normalized by the total number of responses in each class. The distributions show that incorrect responses tend to longer than correct ones, a pattern observed in existing datasets like IFEval, where naturally generated responses used. This length signal highlights the potential bias in uncertainty estimation methods that are sensitive to token length.
24


Reviews:
[{"review_id": "piOfxfMxWe", "reviewer": "Reviewer_zm1y", "summary": "The paper evaluates how well LLMs can estimate uncertainty when tasked with instruction following, given that existing research largely focuses on uncertainty in factuality. The authors systematically evaluate the uncertainty estimation methods using the IFEval dataset and further introduce a new benchmark to disentangle the multiple factors. Experimental results highlight the potential of self-evaluation and probing methods, though they still fall short under difficult conditions, underscoring the need for further research on uncertainty estimation.", "questions": "1. Could you please provide more details in benchmark data generation process? For example, what's the prompt for generating correct response in controlled version, and natural response in realistic version? What are the domains for the arguments (e.g., Type, Kwargs) in the prompts? Are these parameters inherited from IFEval?\n2. It seems that all the token length distributions are not normalized by the total number of total number of responses in each class (Figure 2, 3, and 10). Or to say, the total numbers of correct and incorrect responses are not the same, which means you may not draw a conclusion that incorrect responses tend to be longer than correct ones when only seeing that the orange curve is significantly higher than the blue curve on larger tokens. Normalizing the distribution would be better, though it may not affect the conclusions that token lengths introduce biases.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors provide a systematic evaluation, identifying multiple factors that affect uncertainty estimation and are entangled within naturally generated responses, which helps isolate the influence of various factors and allows for a more accurate assessment of estimation methods' capabilities.\n2. The study offers thorough comparisons between different uncertainty estimation methods, various LLMs, and distinct datasets.", "weaknesses": "1. The Probe method is mentioned in Table 4 and highlighted as the best-performing method. However, it is not introduced or discussed in Section 3, leading to some confusion when comparing results in Section 3.2.1. A clearer explanation of this method earlier in the paper would improve the comprehensibility of the findings. Or at least keep the scope of methods consistent when discussing \"best-performing\".\n2. As noted by the authors in their limitations section, the types of instructions included in the benchmark may not comprehensively reflect real-world tasks. Additionally, the dataset size is relatively small, potentially impacting the reliability of the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rifUKORbRY", "reviewer": "Reviewer_9E6i", "summary": "This paper presents a first of its kind systematic evaluation of uncertainty estimation methods in instruction-following tasks, identifies key challenges in existing datasets and introduce two new benchmark datasets created to address those challenges. The results indicate that there is still a gap in the uncertainty estimation performance of common LLMs, especially when the errors made by the model are subtle. On top of this, internal model states, particularly the middle hidden layers, show an improvement but still remain unsatisfactory in complex scenarios. The main contributions and the motivation are presented clearly, with experiments performed on multiple LLMs and datasets.", "questions": "1. Can the authors elaborate on how training procedures like RLHF can affect the models' performance in uncertainty estimation, especially in Controlled-Hard tasks?\n2. Are there potential methods beyond linear probing that could better utilize internal model states for uncertainty estimation in nuanced tasks?\n3. Would expanding the evaluation to include LLM agents provide more comprehensive insights?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper introduces new benchmark datasets (Controlled and Realistic versions) that isolate factors influencing uncertainty estimation in instruction-following, filling an existing research gap.\n- the methodologies employed are rigorous, with comprehensive experimental setups involving multiple LLMs and uncertainty estimation techniques.\n- the writing is clear, and the results are presented in a way that is easy to follow, supported by well-designed figures and tables.\n- the findings offer valuable insights for advancing the field, particularly the observation that models’ internal states can be leveraged for improved uncertainty estimation.", "weaknesses": "- the analysis might benefit from extending the scope of instruction types and domains included in the benchmark to cover more diverse real-world tasks\n- while the paper identifies the use of internal states for uncertainty estimation as promising, it falls short in exploring more sophisticated methods that could leverage this information in nuanced tasks\n- the use of GPT-4 for task quality assessment introduces a potential risk of pre-training overlap affecting the evaluation, though this is acknowledged as a limitation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pho44WrHsv", "reviewer": "Reviewer_N6r8", "summary": "This paper evaluates the uncertainty estimation abilities of LLMs in instruction-following tasks. It introduces a controlled evaluation setup with two benchmark versions to isolate and compare methods under various conditions. Findings reveal that existing uncertainty methods struggle, especially with subtle errors, and that internal model states provide some improvement but remain inadequate in complex scenarios.", "questions": "The study provides valuable insights into the limitations of LLMs in estimating uncertainty during instruction-following tasks. Given these findings, could you elaborate on specific strategies or methodologies that could be implemented to enhance the instruction-following accuracy of LLMs?", "limitations": "", "rating": 1, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The study provides crucial insights into LLMs's uncertainty estimation in instruction-following problems.", "weaknesses": "1. The paper relies on a single IFEval for its initial evaluation. It would be beneficial to include additional datasets to validate the findings across different contexts and domains.\n2. The paper raises many novel concepts and findings, but it does not seem to provide much direct help in enhancing instruction-following capabilities.\n3. The article did not use LLMs larger than 13B in its tests, so the conclusions may not be sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZF1thMW5b", "reviewer": "Reviewer_8MWv", "summary": "This paper studies uncertainty in instruction following, an important topic for interpretability and trustworthiness. It reveals the discrepancy between accuracy and uncertainty estimation, and among different uncertainty estimation methods themselves. It also accentuates the entanglement problem between instruction following and task completion.\n\nThen the paper proposed a new benchmark separating control and realistic inferences, and shared more findings. IMO the most crucial one is that probe-based method provides the best estimation.", "questions": "Besides trying more inference method, please also consider impact of regularization such as dropout rate if you plan to do any pre- or post-training works. This reveals another problem which is training details are not sufficiently exposed even for open LLMs.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper reveals the uncertainty challenges faced by today's LLMs and presents new methods to better gauge LLM ability to estimate its own uncertainty. I think this is a timely work raising awareness to the community.\n\nI'm not surprised by the top performance of probe-based method because internal states do not lie. Unclaimed by the authors, but the paper hints at open models' advantages to critical tasks, because its white box properties welcome scrutiny and checking by anyone. IMO internal states should be accessible by read APIs.", "weaknesses": "The poor uncertainty estimation by modern LLM is also not surprising because of its strong alliance with MLE methodology instead of MAP. Therefore, besides future directions outlined by authors, I strongly suggest trying more inference methods other than greedy, e.g. setting temperature, beam search, etc. These are also tunable parameters to closed-source models. This way you can measure closed models although the insights will be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)