Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling
Authors: Hang Zheng
Hongshen Xu
Yuncong Liu
Shuai Fan
Lu Chen
Pascale Fung
Kai Yu
Venue: COLM 2025 (2025)
Keywords: reliability
knowledge boundary
self-awareness
TL;DR: We present the Explicit Knowledge Boundary Modeling (EKBM) framework, which improves large language models' reliability by integrating fast and slow reasoning systems to minimize hallucinations while ensuring efficiency in error-sensitive tasks.

Abstract: Large language models (LLMs) are prone to hallucination stemming from misaligned self-awareness, particularly when processing queries exceeding their knowledge boundaries. While existing mitigation strategies employ uncertainty estimation or query rejection mechanisms, they suffer from computational efficiency and sacrificed helpfulness. To address these issues, we propose the \textit{Explicit Knowledge Boundary Modeling} (EKBM) framework, integrating fast and slow reasoning systems to harmonize reliability and usability. The framework first employs a fast-thinking model to generate confidence-labeled responses, enabling immediate utilization of high-confidence outputs, whereas uncertain predictions trigger a slow refinement model for accuracy improvement. To align model behavior with our proposed object, we propose a hybrid training pipeline, enhancing self-awareness without degrading task performance. Evaluations on dialogue state tracking tasks demonstrate that EKBM achieves superior model reliability over uncertainty-based baselines. Further analysis reveals that refinement substantially boosts accuracy while maintaining low computational overhead. The framework establishes a scalable paradigm for deploying reliable LLMs in error-sensitive applications, effectively balancing accuracy and practical utility.

Published as a conference paper at COLM 2025
Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling
Hang Zhengl.2 12+ Hongshen Xu Yuncong Liu
Shuai Fan
Pascale Fung* Lu Chen Kai Yu
IX-LANCE Lab, School of Computer Science
MoE Key Lab of Artificial Intelligence, SJTU AT Institute
Shanghai Jiao Tong University, Shanghai, China
"Jiangsu Key Lab of Language Computing, Suzhou, China AlSpeecch Co., Ltd., Suzhou, China
*Center for Artificial Intelligence Research (CAiRE)
Hong Kong University of Science and Technology, Hong Kong, China SSuhoou Laboratory, Suzhou, China fazure123, xuhongshen, chenlusz, kai kui.yu] tuu.uucun
Abstract
Large language models (LLMs) are prone to hallucination stemming from misaligned self-awareness, particularly when processing queries exceeding their knowledge boundaries. While existing mitigation strategies employ uncertainty estimation query rejection mechanisms, they suffer from computational efficiency and sacrificed helpfulness. To address these is- sues, we propose the Explicit Knowledge Boundary Modeling (EKBM) framework, integrating fast and slow reasoning systems to harmonize reli- ability and usability. The framework first employs fast-thinking model to generate confidence-labeled responses, enabling immediate utilization high-confidence outputs, whereas uncertain predictions trigger slow refinement model for accuracy improvement. To align model behavior with our proposed object, we propose a hybrid training pipeline, enhanc- ing self-awareness without degrading task performance. Evaluations on dialogue state tracking tasks demonstrate that EKBM achieves superior model reliability over uncertainty-based baselines. Further analysis reveals that refinement substantially boosts accuracy while maintaining low com- putational overhead. The framework establishes scalable paradigm for deploying reliable LLMs in error-sensitive applications, effectively balanc- ing accuracy and practical utility.
1 Introduction
Recently, large language models (LLMs) have demonstrated impressive text generation capabilities (Abdullah al., 2022). However, LLMs are susceptible to hallucinations, where generated content misaligns with context or factual information (Zhang et al., 2023). Halluci- nations can be particularly detrimental in applications with low tolerance for error, eroding user trust in LLM reliability.
Extensive works have been devoted to mitigating hallucinations in LLMs (Tonmoy al., 2024). Hallucinations often arise from misalignments between an LLM's outputs and its intrinsic knowledge boundaries, particularly when when addressing queries beyond its expertise (Li et al., 2024b). Improving the model's self-awareness its capacity to accurately assess its own knowledge boundaries and outputs correctness, can effectively mitigate this misalignment. Uncertainty-baat bbsed methods estimate model confidence and indirectly
"Han Zheng and Hongshen Xu contribute equally to this work.
"uu Chen and Kai Yu the corresponding authors.
1

---

Published as a conference paper at COLM 2025
reflect the model's self-awareness (Huang et al., 2024), facing challenges of inherence to the model's endogenous capabilities, frequently sufferring from high computational costs or instability across thresholds. Alternative strategies emphasize aligning knowledge boundaries with actions by rejection of out-of-scope queries (Xu et al., 2024b), avoiding mistakes but sacrificing helpfulness- trade-off that undermines model utility particularly in complex, multi-step tasks where partial correctness remains valuable.
To address these limitations, we propose novel alignment objective that enhances model awareness of its knowledge boundaries, enabling models to respond more effectively by explicitly distinguishing between high-confidence ("sure") and low-confidence ("unsure", outputs. A reliable system should deliver near-perfect accuracy for "sure" predictions and provide helpful information in the "unsure" category, serving as a form of soft rejection, appropriately managing user expectations regarding accuracy.
Building on this concept, we propose the Explicit Knowledge Boundary Modeling (EKBM) framework. EKBM integrates both Fast and Slow systems: fast-thinking model generates responses annotated with confidence labels, allowing for immediate use of high-confidence outputs, while a slow-thinking refinement model engages in deliberate reasoning to improve low-confidence predictions. Crucially, EKBM does not merely filter uncertain outputs but leverages them opportunities for im- provement, ensuring both precision and re- call. The collaboration between these two
Dialog history
User: Book me train, like leare Cambridge sometime after 7100.
systems strikes a balance between reliabil- ity and efficiency, with the fast-thinking model's self-awareness being crucial for the system's 's effective functioning.
SYSTEM: lave that leavos at 7:50.
Stondard Diologue State Tracking
destination Cambridgo, leareat
Emarl Leavest 7:59 not user infention.
To operationalize this paradigm, we de- sign a training pipeline that aligns the
Rejection Aligned Model
model's capability boundaries and im- proves self-awareness. Evaluations on di- alogue state tracking (DST) tasks demon- strate that our method effectively and reli- ably distinguishes the confidence levels of model output compared to baseline meth- ods. Additionally, we develop an auto- matic refinement model for "unsure" out- puts. Experiments show that the EKBM sys- tem achieves superior overall performance compared to traditional approaches that di- rectly tune models for the task.
confirm depurture or 7:99 Further user confirmation is needed.
Mistokes aroiced, help previced well
Reliabiityy Aligned Model (ours)
Sure: dlesiintion Cambiraige
Unsure: leaveat 7:59
To Refinement Model
lesvent :n 7159 is not dinect user intention,
conrected
Mistokes avoided and help provided.
Figure 1: case study on Dialog State Tracking: comparison of different alignment objectives.
Our contributions are summarized as follows:
We present the EKBM framework, integrating fast-thinking assessment with slow- thinking refinement, along with metrics for evaluating self-awareness and reliability.
We develop comprehensive training pipeline that enhances LLM self-awareness,, yielding more reliable models.
We perform extensive experiments to demonstrate the effectiveness and scalability of the EKBM framework, providing valuable insights for future research.
2 Problem Formulation
2.1 Alignment for Reliability
To enhance the reliability of LLMs, a prevailing method involves aligning their performance with high-quality training data. This alignment aims to achieve the following objective:
s(x,yc) > s s(x,
(1)
where a scoring function s(x, y) ensures that the score of a correct response ye to an input prompt x is higher than that of an incorrect response yu-
2

---

Published as a conference paper at COLM 2025
Recent studies have introduced rejection mechanism to help the model differentiate between answerable (in-boundary) and unanswerable (out-of-boundary) queries, where model is encouraged to directly reject the unanswerable ones. The corresponding alignment objective can be articulated as:
s(x,ye) > s(x, > s(,
(2)
where y/ denotes truthful rejection, and V/ incorrect response. This suggests that refusal is preferable to providing an incorrect answer.
While this rejection strategy ensures reliability, it may unduly limit the model's helpfulness. Therefore, we propose a more nuanced objective: the model should provide an answer when- ever possible, categorizing its output either high-confidence ("sure") or low-confidence ("unsure"). The model must ensure high accuracy for "sure" responses while tolerating some errors in the "unsure" category, which can serve as references for users or inputs for further refinement. Our proposed alignment objective is formalized as:
Cc, > s(x, ye, Cu) s(x, ywo, Cu) > s(x, Yar, Cs)
(3)
Here, Ve and yeu denote correct and incorrect responses, while c, and cu represent "sure" and "unsure" confidence levels. This objective guarantees that correct "sure" predictions are prioritized over correct "unsure" predictions, which, while less confident, still provide utility. Incorrect "unsure" predictions are discouraged but considered more acceptable than incorrect "sure" predictions, as the latter significantly undermine overall reliability.
2.2
Reliability Evaluation
In this work, we focus on complex multi-slot problems and have modified several metrics for better evaluation of model reliability.
2.2.1 Weighted-F1
The Fl-score, calculated using True Positives (TP), False Positives (FP), and False Negatives (FN), is typically the evaluation metric for multi-slot problems:
TP
TP
Precision
Recall
(4)
TP + FP
TN+
F1 = 2 Precision Recall
(5)
Precision Recall
We modify the F1 score to align with the objectives of our EKBM framework. Since "unsure" predictions represent state of uncertainty, we assign weighted contribution (denoted by
d1 and i2 both ranging from 0 to 1) to reflect their partial utility as candidates for refinement. The modified precision and recall are defined
STP &1
UTP
STP + a2 UTP
Precision(as) STP + SFP + a1 (UTP UFP)
Recall(a2)
(6)
STP + UTP + FN
Here, STP and SFP denote True Positives and False Positives for "sure" predictions, while UTP and UFP are the corresponding values for the "unsure" category. For recall, we consider a prediction
be a UTP if it is partially correct (i.e., refinable). Unlike precision, we do not apply the weighting parameter d2 to the denominator of the recall calculation, measures the proportion of gold labels retrieved, which is independent of the introduction of a. Notably, incorporating does not affect the range of precision and recall, both of which remain within [0,1]. The modified Weighted-F1 score is defined as:
recall
Weighted-Fl((1.n)) = 2 Precision(a:) (a;)))oon(()))...)))))))))))))) Recall (22) Precision(a1) + Recall(A2)

---

Published as a conference paper at COLM 2025
(1) Framework of Explicit Knowledge Boundary Modeling
Sure Parts
Execution
Fish rnlicandd
Thr's
Fost & Reliable
Model
Input
Parts
Refinement
Parts
(2)
Training Data Construction Pipeline
(3) Refinement Data
SFT
GPT-40
L.M
Cerrect
Mlatin
DPO Preference bata Construction
OOO
Gold
Srame
Resoring Steps
X
Sanglig
R2
Praference
BS
Combine
Drpert
LLM
Wighted
#2: Score U.
Rn
Refirenent Madel
Figure 2: The EKBM framework and the data construction methods.
2.2.2 Metric-Objective Consistency: Weighted-Fi as a Parametric Alignment Measure
The Weighted-F1 metric aligns with our four-tiered objective (Equation (3)) by using parametric formulation that encodes hierarchical priorities, which ensures: (1) Strict en- forcement for sure predictions: Unweighted terms prioritize high-confidence correctness (STP) and apply maximal penalties for errors (SFP). (2) Controlled utility for unsure predic- tions: The a-scaled terms modulate low-confidence outputs, with attenuated UFP penalties tolerating unsure errors, and partial UTP rewards incentivizing refinable correct predictions.
This dual-parameter mechanism enables scenario-specific adaptation: Weighted-Preeisionn regulates error tolerance via a1 (higher 41 tightens unsure-error constraints) and Weighted- Recall controls reference-value incentives via #2 (higher 42 promotes helpfulness through unsure-correct outputs). The composite metric thus operationalizes the reliability- helpfulness trade-off, permitting calibration from conservative (a1 1, a2 0) to ex- ploratory (a1 + 0, 82 + 1) strategies.
We discuss few corner cases: (1) 01 = 0, 42 0: Weighted-F1 disregards all "unsure" predictions. (2)
0, &2 = 1: Errors in the "unsure" category are ignored, rewarding effective recall, representing the theoretical upper limit after perfect refinement. (3) a1 =,, 42 0: Lacks significant interpretation. (4) a1 = 1, a2 1: Eliminates confidence labeling, reverting to traditional methods.
2.2.3 Reliability Metrics
We use two specific Weighted-F1 scores: Quality-F1 as the primary metric for assessing model reliability, and Optimal-F1 as a supplementary reference.
Defined as (oiighted-F1(0.5 0.5), this metric assigns half the weight to "unsure" predictions compared to "sure" ones. This encourages the model to provide high-confidence "sure" outputs while mitigating penalties for "unsure" ones. Quality-F1 directly measures the model's self-awareness and reliability.
Quality-F1:
Optimal-F1 Defined as Weighted-F11 (0, 1), it represents the theoretical upper limit achievable after perfect refinement of "unsure" predictions, reflecting the overall potential the system.
3 Method
3.1 Explicit Knowledge Boundary Modeling
We formalize our proposed framework, Explicit Knowledge Boundary Modeling (EKBM), which aims to improve model reliability and self-awareness by explicitly modeling its
4

---

Published as a conference paper at COLM 2025
knowledge boundaries and improve overall performance by incorporating a refinement model to improve the "unsure" predictions. As shown in Figure 2 (1), the framework divides the model's makini process into two distinct stages, balancing immediate usability with comprehensive coverage, making the system both reliable and helpful. We provide complete example of the two-stage execution flow and model outputs in Appendix B.
Fast Prediction with Confidence Labeling In the first stage, the model performs the primary task and simultaneously assigns confidence label ("sure" "unsure") each prediction. "Sure" predictions are directly accepted as final outputs, ensuring high precision.
Slow Refinement for Unsure Prediction the second stage, "unsure" predictions un- dergo further refinement through strategies like user confirmation, automated multi-step reasoning, or other post-processing techniques, incuring extra cost but increases overall accuracy and reliability of the system.
3.2 Reliability Alignment
To align the model's behavior with our alignment objective, we explore various training strategies to enhance its intrinsic self-awareness (Xu et al., 2025), including Supervised Fine-Tunnig (SFT) and Direct Preference Optimization (DPO). The data construction process is shown in Figure 2(2), with two detailed examples provided in Appendix &C.
SFT Data Construction We assume that model's task-specific knowledge boundary can be approximated by repeated sampling on training data. Accordingly, our SFT data construction is straightforward: for each training sample, we perform multiple samplings. Predictions consistently deemed accurate are labeled "sure," while those with errors or omissions are marked "unsure." " Crucially, we intentionally refrain from correcting erro- neous outputs to avoid introducing supervisory signals that might inadvertently improve the model's task performance. That is, when the model produces an incorrect prediction, we simply assign a confidence label of "unsure" without rectifying the error. The erroneous output remains in the final training data, reinforcing the model's ability to recognize and categorize uncertain information rather than improving its accuracy directly.
The number of sampling rounds a hyperparameter. A larger results in a more conser- vative model, as the proportion of "unsure" labels in the training corpus increases. In our study, we set 1 by default to accurately reflect the model's realistic capability boundary.
DPO Preference Data Construction DPO enables the model to learn directly from pref- erence pairs, making the design of an effective preference strategy crucial. Given our alignment objective, is intuitive to use Weighted-F1 as the preference score function. The a values in Weighted-F1 significantly influence the model's behavior and performance. Through experimentation (see $4.5), we determined that setting 41 0.25 and a2 0.75 best balances the model's 's immediate helpfulness and its potential performance after refinement. Therefore, this setting will be used by default unless specified. To enhance the stability of DPO training, we ensure that all data is generated through multi-sampling from the model itself, aligning with the model's output distribution.
3.3 Refinement Model
We train automated refinement models for each dataset further optimize unsure pre- dictions by multi-step reasoning, leveraging supervised fine-tuning (SFT) based on the LLAMA3 8B model. Our refinement operates fine-grained level, addressing each unsure prediction individually. We employ reasoning paradigm similar to the Chain of Thought (Wei et al., 2022) methodology and use GPT-4o (Hurst et al., 2024) to generate detailed reasoning process for each training prediction. This approach is illustrated in Figure (3), and detailed information can be found Appendix SD.
4 Experiments
We conduct experiments focusing on three core research questions:
RO1
Does training pipeline outperform current methods in enhancing self-awareness?
RO2
Does EKBM framework improve the overall performance of LLMs on complex tasks?
RQ3
Does our approach scale robustly across foundation models with varying capabilities?
5

---

Published as a conference paper at COLM 2025
Method Type Method
BITOO
Opt.
ReGuer!
Ope.
Oph
12A4/455411 Turbey
(2074)
Diress
Prompt
65.85
76.50
27.43
14.16
$6.45
81.31
$1.36
61.43
92.29
76.32
78,84
1560
65.32
33.61
33.37
21.30
Pros
60.85
8066
77.6
09:13
83.50
72.44
23.01
60.30
36305
52.28
27.34
22.86
Unoetsinty
Sc
80.1
89.31
86.15
64-88
83.16
75, 66
72:33
6243
44.6
41.00
42.75
16.55
Pitrue)
76000
6306
Rediavility (Cung))
SPT
93.73
01.65
80.44
65.65
88.50
55.08
56.74
85.01
$2.65
61.64
DP0
$5.59
Qaen22
7B Ying st al. (2104)
Primpt
Vertzose
83.02
71.34
69.5
59.88
79.62
77:22
75.23
64.76
65.75
31.30
32.83
21.74
5R
73.54
72.06
69.69
67307
23.23
06 46
67.90
66.34
35.67
36.82
35.43
34.06
Prub
77.12
71,47
72.02
63.19
$4.57
74.43
70.85
63.31
46.66
38.91
34.85
24.5
Uncestainly
$3.55
90.30
$4.67
07000
87.45
74.85
77.42
51.43
71.47
61.00
55.36
PCTrue)
Reisbility
SFT
83.96
070 Post
94.02
92.23
92.12
82.89
89.29
84.32
96.70
84.61
$1.00
07.53
Table 1: Reliability performance three DST datasets. Method denotation: DPO Joint: SFT and DPO Joint Tranining. DPO Post: DPO Post Training. Metric denotation: Precsure' precision "sure" predictions. ReCutall: recall of the "sure" and "unsure" predictions. Opti. Fl: Optimal-F1, Qual. F1: Notably all outputs of the Direct baseline are treated as "sure".
4.1 Experiments Setup
4.1.1 Model and Baselines
We utilize the LLaMA3 8B model and the Qwen2.5 7B model as the backbone for our experiments. For reliability alignment, we conducted Reliability-SFT and integrated DPO to create two additional variants: Reliability-SFT-DDO Joint Training, which undergoes joint training with SFT and DPO, and Reliability-FFT+DDP Post Training, where DPO training follows initial SFT training, optimizing the model beyond the Reliability- foundation. The key distinction lies in the different source distributions of preference data sampled for DPO. fraining details could be found in Appendix $A.
To evaluate the reliability of our proposed models, we compare them against six baselines: three prompt-based and three uncertainty-basee approaches (see Appendix ŞE). Notably, our focus is on the model's s self-awareness rather than task performance, so we do not include SOTA DST models. By default, we use only 1,000 samples for Reliability-SFT training and 2,000 for DPO, which is significantly less than traditional models.
Prompt-based methods encompass three approaches: Direct, Verbose, and Self-Reflection (denoted SR, Ji al. (2023a)). The Direct approach generates predictions without confi- dence estimation, while Verbose produces both predictions and confidence labels simultane- ously. SR assesses each prediction through self-evaluative reflection.
Uncertainty-based methods include Token Probability (Prob, Manakul et al. (2023)), Self- Consistency (SC, Chen & Mueller (2024)), and P(True) (Kadavath et al., 2022). These methods use different techniques to classify outputs "sure" or "unsure". Prob calculates average token probabilities; SC measures the frequency of predictions across multiple samples; and P(True) determines the probability of the "True" token during model self-evaluation.
4.1.2 Dataset
We evaluate our methods on the Dialogue State Tracking (DST) task in task-oriented dialogue systems, which involves extracting slot-value pairs from multi-turn dialogues based on predefined ontology (Xu et al., 2024a). We focus on the intricate multi-slot problem, where the model is likely to exhibit varying confidence levels for different parts of responses to the same query. Consequently, we do not adopt traditional QA or mathematical da datasets. To ensure a robust evaluation, we utilize three datasets: MultiWOZ-2.4 (Ye et al., 2021), BiTOD (Lin et al., 2021) and SGD (Rastogi et al., 2020). For BiTOD, we adopt the English version, while for SGD, we randomly select 10,000 instances from the testset to mitigate excessive computational overhead. Common evaluation metrics for DST include Joint Goal Accuracy (JGA) and Slot-F1. For JGA, sample is deemed correct only if all predictions are accurate without omissions, while Slot-F1 offers slot-level assessment.

---

Published as a conference paper at COLM 2025
4.2 Reliability Evaluation
The experimental results on model reliability are summarized in Table 1. We evaluate our reliability-training pipeline against baseline algorithms, with particular focus on the model's self-awareness capabilities. For RQ1, our method significantly outperforms tradi- tional prompt-based and uncertainty-based methods across various datasets, demonstrating a substantial improvement in self-awareness.
Our approach consistently achieves superior Quality-Fl scores, demonstrating improved ability distinguish predictions as "sure" or "unsure" based on confidence. A high Precisionsure indicates reliable "sure" predictions, fostering user trust with minimal need for verification. Simultaneously, a high Recallootl ensures the "unsure" category contains a broad range of potentially correct outputs, which which forms a solid basis for refinement. Together, these metrics lead to a higher Optimal-F1 score, representing the theoretical maximum post- refinement performance. In essence, our system optimally balances precision and recall, providing both immediate reliability and a strong potential for post-refinement accuracy.
In our approach, DPO-based methods generally outperform SFT-only methods. Specifically, models trained with DPO Joint Training achieve higher Optimal-F1 scores, suggesting more conservative stance that defers subsequent refinement. In contrast, those undergoing Post Training yield superior Quality-F'l scores, indicating greater confidence and higher- quality predictions. This discrepancy arises because Joint Training samples from a pre- SFT distribution, which has poorer performance. This results in dataset with a higher proportion of "unsure" predictions, leading more cautious model behavior.
Notably, some baselines, such as Self-Consistency, show high Precision Recall and Optimal-F1 scores, suggesting initial superiority. However, this often indicates an overly conservative model that frequently labels predictions as "unsure." This leads to a suboptimal balance between prediction types, as evidenced by significantly lower Quality-F1 score. In an extreme case, model could label all predictions as "unsure", maximizing Optimal-F1 but shifting the entire burden to the refinement stage. This approach sacrifices immediate usability and fails to balance reliability and helpfulness.
4.3 Refinement for Unsure Prediction
As described in Section $3.3, we trained an automatic refinement model for each dataset. We refined the "unsure" predictions from the initial stage and merged them with the "sure" predictions to generate the final results. Table 2 shows the overall evaluation. For RQ2, our method achieves significantly better performance after refinement, with substantial improvements in Slot-Fl and JGA.
Referring to $4.2, despite their higher theoretical performance limit (Optimal- F1), DPO Joint Training models exhibit suboptimal Slot-F1 and JGA after re- finement compared to Post Training models. This discrepancy stems from the limitations of the refinement model. Since the refinement model is not per- fect, an excessive number of "unsure" predictions can lead to residual errors after refinement, incurring significant costs without proportional gains. This highlights the importance of a balanced "sure" and "unsure" classification. An ideal model should minimize unneces- sary "unsure" predictions and refine only when truly warranted, maximiz- ing both efficiency and overall perfor- mance.
Method type
Method
MullWOZ 24
BITOD
Slee-F1 1
JGA
Slo-F1
JGA " SloHFT 1
JGA
LLAMA3 4B
96.01
10.41
Prompt
Verbose
36.69
23.11
78.68
20.72
76.50
30.74
29.90
11:25
23:05
Unsertainty
20.20
P(True)
19.18
36.29
Reliability
75.47
CPO
96.35
73.56
Qhve225 78
Direct Verbose
96-35
81.25
47102
16:95
Prompt
26:45
27.99
Prob
74.18
23.56
73.87
25.64
3691
10.51
Uncertainty
st
23:45
68
$4.4
P(True)
17.05
66.23
3366
Reliabllity (Cung)
SFT
CRO Zodnt
$4.50
17.06
51.00
52.41
85.44
65.29
58.30
62.31
7486
76885
25.02
27.06
und
$3.19
65.86
63.61
74.95
Table 2: Comparison of different methods on overall performance after refinement. Note: for Direct baseline we conduct fully refinement since there's 's no confi- dence label.
7

---

Published as a conference paper at COLM 2025
4.4 Scalability Analysis
To evaluate scalability, we experimented with foundation models of varying capabilities. We consider the original LLAMA3 8B as low-performance baseline (denoted as Low), then fine-tuned two models with 1,000 (Medium) and 10,000 (High) samples to enhance their DST capabilities. We then continually conducted reliability training and evaluated the performance of the obtained reliable LLMs against baseline methods. As shown in Figure 3, our methods consistently outperform baselines after refinement, demonstrating strong scalability. This particularly relevant for real-world scenarios where models are often domain-sperrffcally tuned and possess high task capabilities. Detailed results in Appendix SF further support these findings.
As illustrated in Table 7 in Appendix ŞF, the advantage of DPO Post Training over Joint Training becomes more pronounced as foundation model performance improves. This is because our offline DPO algorithm's limitations foundation models, negatively affecting the Joint Training data distribution and overall performance. Notably, in the SGD dataset, the benefits of DPO Joint Training diminish with increasing foundation model performance, even falling behind the SFT-only approach.
exacerbated by higher-performing
Medlin
Tigh
160
16) MuIWOZ-2.4
BITOD
(6) 800
Prondd-beeed
Uncar Uncoslarft-bbaee
Ruiiabllty
Figure 3: Comparison of different methods under foundation models of various task abibility. Figure (a), (b) and (c) illustrates the JGA after refinement (%) on three datasets. Chosen representative methods: Prompt-based: Direct; Uncertainty- Prob; Reliability: SFT+DPO Post Training.
4.5 DPO Preference Strategies
We constructed our DPO preference data using Weighted- Fl as the ranking metric, as detailed in $2.2.1. Weighted-F1 incorporates two adjustable a parameters: 01 for Weighted Precision and 02 for Weighted Recall. Using the Reliability trained LLAMA3 8B model, we generated DPO preference data with varying & values and performed subsequent Post Training. Results the MultiWOZ-2.4 dataset (Figure show that the model achieves optimal performance at a1 = 0.25 and 42 0.75, reaching near-best results at relatively low cost.
89.7
36.1
32.7
29.1
27.9
40.3
23.3
202
2922
19.5
S
1522
2.7
29
3.0
05
075
(a) Unsure Proportion
Analysis indicates that increasing 41 heightens sensitivity to errors "unsure" predictions, reducing tolerance and the number of "unsure" outputs. Conversely, increasing a' am- plifies rewards for successfully recalling "unsure" predictions, encouraging more conservative stance. The optimal alpha settings depend on the specific scenario and refinement model performance; if the refinement model underperforms, the cost of excessive "unsure" predictions may not be justified. The flexible adjustment of both a parameters allows for adaptation to diverse real-world scenarios.
578
511
512
538
42.4
101
43.
357
44.
144
314
0.5
0.25
0.5
0.75
(b) JGA
Figure 5:
Comparison of DPO preference strategies on model behavior and perfor- mance.
4.6 Detailed Analysis
4.6.1
Prediction Types Analysis
To evaluate our model's ability to make reliable confidence judgments, we further analyzed the distribution of its output categories and compared them to baselines. We examined five output categories: correct and incorrect predictions classified as "sure" or "unsure", along with missing predictions. The results are presented in Figure 6.
8

---

Published as a conference paper at COLM 2025
100
80
Parcentage (N3
60
40
MUIIAOZ 244
BRTDD
E6D
NUZINOZ 24
BrTD
500
Qwen2.5-78
Uncerallity
Figure Percentage of predictions types among different methods. Chosen representative methods: Prompt-based: Direct; Uncertainty-base based:: Prob; Reliability: SFT+DPO Post Training.
Our findings highlight two key improvements: (1) Enhanced reliability: We observed a notable reduction in incorrect "sure" predictions, with most errors concentrated the "unsure" category. This indicates stronger self-assessment capability, allowing the model to better identify potential errors than baselines. Additionally, the "unsure" category consistently contains fewer correct predictions than incorrect ones, distinction baseline models struggle to achieve. (2) Improved recall:: Our approach significantly reduces missing predictions by capturing previously omitted relevant information within the "unsure" category. While some of these may still contain errors, this shift enhances overall helpfulness and provides stronger foundation for refinement.
4.6.2 Cost Analysis
In this work, Cost refers to the additional overhead from the refinement process. We define the cost of refining "unsure" prediction value of 1, equating Cost to the number of "unsure" " predictions. For better comparison, we express Cost as the proportion of "unsure" predictions.
constant
Coit
Sell Rallaiin
PYTuu!
RanelllysTTcC SFTICPO
Figure 7: Comparison of different meth- ods of Cost-JGA on MultWOO-224.
We compare the performance and Cost of various baselines in Figure 7. Results are shown for the MultWOZ-2.4 dataset, with complete findings in Appendix SG. Our methods show sig- nificant performance improvements at relatively low Cost. Additionally, as shown in Appendíx Table 8, Cost decreases significantly as the foundation model's performance improves, which is expected as stronger model produces fewer uncertain predictions.
4.6.3 Refinement Model Comparison
Relismmnn Acmunay (%)
We used various refinement models to en- hance predictions from the LLAMA3 Reliability- SFT+DPO Post model. As shown in Table 8,
8%
20
54,77
JOA
48.21
we compared our trained models against GPT- 40 and the DeepSeek series (DeepSeek-Al et al.,
35.55
2025) (8B and 70B), all known for strong rea- soning. Results show significant performance differences, with DeepSeek 8B underperform- ing while the 70B model nearly matches GPT--4o. Our task-specific fine-tuning with CoT substan- tially improves refinement accuracy.
R1-16
R1-708
GPT-4o
Figure Comparison different refinement models. The line chart represents "Refine- ment Accuracy", while the histogram repre- sents "JGA"
4.7 Extension to Mathematical Problems
To validate the generalization of our method to different tasks, we conducted supplementary experiments on the widely used GSM8K mathematics dataset (Cobbe et al., 2021) using the Swen2.5-7B-Instruue model with 1,000 training samples. We evaluated the model under two test settings: (a) with Chain-of- -fhougght (CoT) reasoning (Wei et al., 2023) and (b) by directly predicting the final answer. We tested both settings as LLMSs generally perform

---

Published as a conference paper at COLM 2025
differently each. For the refinement model, used the DeepSeek-RI--Dsstll-Laaams model, which has strong mathematical reasoning capabilities.
We adapted the Fl-based evaluation metric from Section $2.2 to an accuracy-based definition:
Σ x,E5ure
((x;)
5(xg)
X, xyElssure
Weighted-Accurac((X..))
Nsure + & x Nunsure
Quality-Accuracy(2) = Weighted-AcccrracyX..
As shown in Table 3, our method consistently outperformed both prompt-based and uncertainty-based baselines in improving the model's self-assessment ability, achieving higher Quality-Accurant Furthermore, after applying refinement with the same refinement model, our approach yielded a higher final accuracy at a reasonable computational cost.
Method Type
Quality Accuracy
Final Accuracy
Method
Quality Accuracy
Final
Method
Method
Type
Accuracy
Prompt
Direct Verbose
86.81
86.81
Prompt
Direct Verbose
19.94
19.94
90.98
21.44
24.34
Uncertainty
Prob
87.69
88.55
Uncertainty
Prob
24.49
76.50
SC
89.41
91.48
SC
24.37
70.05
Reliability (ours)
SFT
89
91.05
Reliability (ours)
SFT
27.31
77.94
DPO Joint
90.14
93.10
IPO Joint
26.16
78.24
DPO Post
92.73
95.30
DPO Post
27.83
79.83
(a) Inference setup: Chain-of-Thoon Thouuh
(b) Inference setup: Answer Directly
Table 3: Comparison of different methods on GSM8K dataset. Quality Accuracy assigns a weight of 0.5 to unsure samples. Final Accuracy represents the overall accuracy after refinement.
Related Work
The knowledge boundary of LLMs has emerged as a critical topic in recent research, high- lighting their limitations in generating reliable outputs (Yin et al., 2024; Li et al., 2024b). Misalignment between model behavior and knowledge boundaries can result in factual hallucinations (Huang al., 2023) and ambiguous responses (Liu et al., 2023).
Various methods have been proposed to assess these boundaries. Uncertainty- ap- proaches quantify prediction confidence through token probabilities (Manakul et al., 2023) and consistency (Chen & Mueller, 2024). Calibration strategies, including prompting (Tian et al., 2023) and fine-tuning (Tao et al., 2024) for improved confidence expression, align model confidence with prediction accuracy. Internal State Probing evaluates prediction fac- tuality by analyzing model states, such as activations (Li et al., 2024a). While effective, these methods often rely on external post-hoc techniques, which are computationally expensive and lack integration with the model's endogenous reasoning processes (Huang et al., 2024; Zhou et al., 2024; Pan et al., 2025; Shan et al., 2025).
To further align model behavior rather than merely estimate uncertainty, some studies guide models in self-reflection to minimize anffcontradictions (Wei et al., 2022; Ji et al., 2023b), albeit with increased computational overhead. More recently, RL-based approaches leverage uncertainty estimation to train reward models, enhancing LLM truthfulness by rejecting queries beyond their capabilities (Xu et al., 2024b; Chen et al., 2024; Xue et al., 2024). While effective, these methods rely on the accuracy of uncertainty estimation algorithms and meanwhile may unintentionally reduce overall helpfulness.
Conclusion
In this work, we propose a two-stage framework that enhances the reliability of large lan- guage models by integrating fast and slow thinking paradigms. Through precise confidence assessment and high-accuracy refinement, we achieve a balance between reliability and usability. Extensive evaluations demonstrate that method significantly improves model self-awareness and performance across complex tasks, establishing a new paradigm for enhancing the reliability of language models in error-eensitive applications.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This work is funded by the China NSFC Projects (62120106006, 92370206, and U23B2057) and Shanghai Municipal Science and Technology Projects (2021SHZIZZX1002 and 25X010202846).
References
Malak Abdullah, Alia Madain, and Yaser Jararweh. Chatgpt: Fundamentals, applications and social impacts. In 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), pp. 1-8. leee, 2022.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 5186-5200, 2024.
Lida Chen, Zujie Liang, Xintao Wang, Jiaqing Liang, Yanghua Xiao, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han, and Wei Wang. Teaching large language models to express knowledge boundary from their own signals. arXiv preprint ar orXiv:2406.1088
2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.ct wttp://.oog//bb//////// 14168.
Daya Guo DeepSeek-AL, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXit:2501.129 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arviv preprint ar Xiv::444 21783, 2024.
Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. A survey of uncertainty estimation in llms: Theory meets practice. ar Xiv preprint ar Xrx::2410.11326,
2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. arv preprint ar xrXi::2311.05232, 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et Gpt-4o system card. arxiv preprint arXie:2410.21270 2024.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigat- ing LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/ URL https: http:::://aaaaaliiee hatlaaaaaannnoyy.... org/2023. findings-emnlp. 123.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards miti- gating 11m hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, 2023b.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Slatfeld-Doddee Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arxiv preprint arXiv:2207 220:002200.0,21,,,, 2022.
11

---

Published as a conference paper at COLM 2025
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024a.
Moxin Li, Yong Zhao, Yang Deng, Wenxuan Zhang, Shuaiyi Li, Wenya Xie, See-Kiong Ng, and Tat-Seng Chua. Knowledge boundary of large language models: A survey. arXiv preprint ar arXio:2412.12472, 2024b.
Z Lin, A Madotto, GI Winata, P Xu, F Jiang, Y Hu, Shi, and P Fung. Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling. arxiv 2021. arv preprint #XXv:2106.02788, 2021.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're re afraid language models aren't modeling ambiguity. arviv preprint ar arXiv:2304.14399 2023.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. ar Xiv preprint ar arXit:2303.08886, 2023.
Renjie Pan, Jihao Dong, and Hua Yang. Discovering clone negatives via adaptive contrastive learning for image-text matching. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. htt.://.eee/oorrmmmmaa//ya/aaaa
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, PP. 1689-8896, 2020.
Rong Shan, Jianghao Lin, Chenxu Zhu, Bo Chen, Menghui Zhu, Kangning Zhang, Jieming Zhu, Ruiming Tang, Yong Yu, and Weinan Zhang. An automatic graph construction framework based on large language models for recommendation, 2025. URL https: //arxiv.org/abs//412 18241.
Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. When to trust llms: Aligning confidence with response quality. arviv preprint arXio:2404.17287 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning, Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arxiv preprint ar XrXi::2305.1495, 2023.
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint ar Xriv:2400..03313, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35.24824-24837, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-offthought prompting elicits reasoning in large lan- guage models, 2023. URL https::/arxiv..tt org.
Hongshen Xu, Ruisheng Cao, Su Zhu, Sheng Jiang, Hanchong Zhang, Lu Chen, and Kai Yu. A birgat model for multi-intent spoken language understanding with hierarchical semantic frames, 2024a. URL https: //app::///rxiii wow.lo.rgggaae absp//22.2222... 18258.
Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. Rejection improves reliability: Training Ilms to refuse unknown questions using rl from knowledge feedback. ar Xiv preprint ar 240.24433.844 2024b.
12

---

Published as a conference paper at COLM 2025
Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. Reducing tool hallucination via reliability alignment, 2025. URL https: 04141.
Boyang Xue, Fei Mi, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Erxin Yu, Xuming Hu, and Kam-Fai Wong. Ualign: Leveraging uncertainty estimations for factuality alignment on large language models. arXiv preprint XrXi:22412.11803, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. ar preprint ar dxXto:2412.15115, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. Xiv preprint arXto:2308.0132 2023.
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. Multiwoz 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. Xrviv preprint ar srii:22004.0073,, 2021.
Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. Benchmarking knowledge boundary for large language model: A different perspective model evaluation. arxiv preprint 1114:22002.1143,, 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ocean: a survey hallucination in large language models. ar Xiv preprint ar (rXi:23099002219 2023.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreli- able: The impact of language models' reluctance to express uncertainty. arxiv preprint xxXiv:2401.06730 2024.
13

---

Published as a conference paper at COLM 2025
A Training Details
We modiff the DeepSpeed-Chiee (Yao al., 2023) framework accommodate our experimen- tal setup. We adopt most of the default training parameters provided by DeepSpeed- -hhat except that the learning rate for DPO training is set to le-7 and the maximum sequence length is set to 8096 the SGD dataset and 2048 the others. All models are trained for 1 epoch using NVIDIA A800 GPUs.
B Execution Example for Two Stages
We provide an example sample that includes the complete execution flow and model outputs for the two stages as shown in Table 4.
Input
Dialog history:
USER: Hi, I'm looking for hotel to stay in.
USER: Okay, please book that for 3 people and 2 nights starting from Friday.
SYSTEM: Booking was successful. Reference number is 9HMD04UW. anything else?
USER: I would love to find a restaurant well.
Reliable dialog state:
Stage One: Reliable LLM Predictions
["hotel": ("bookday": ("value": "friday", "confidence": "sure"), "bookpeople": ("value": "3", "confidence": "sure"), "bookstay"; ("value": "2", "confidence": "sure")}, "restaurant": ("bookpeople": ("value": "3", "confidence"; "unsure"]] }}}
Stage Two: Refinement for "Unsure" Predictions:
Target: ("domain": "restaurant", "slot": "bookpeople", "value": "3"}
Refinement model:
Thinking steps:
stepl: structure aspect: "bookpeople" is one of the slots in the restaurant domain and the value is specified as "3"
step2: semantic aspect: In this case, the user never specifies the number of people for the restaurant booking. The number "3" is related to the hotel booking, not the restaurant. Therefore, the slot-value pair is irrelevant to the user's intent for the restaurant and should be removed.
step3: the predicted slot-value pair is incorrect in semantic, and the slot should be removed. Refine action: removing
Refined slot-value pair: none
Final Refined Predictions:
("hotel": [ ("bookday": ("value": "friday", "confidence": "sure"), "bookpeople". ("value": "3", "confidence": "sure"), "bookstay": ("value": "2", "confidence": "sure")
Table 4: An example of the complete execution flow and model outputs.
C Training Data Construction
We provide detailed examples to illustrate training data construction process, including model inputs and the expected outputs the model is intended to learn. The data construction for SFT is outlined in Table 5, while that for DPO is detailed in Table 6.
14

---

Published as a conference paper at COLM 2025
Input
Dialog history:
USER: Is there modern European restaurant in the north?
SYSTEM: found city stop restaurant. They are located at Cambridge City Football Club
Milton Road Chesterton.
USER: That sounds lovely!
SYSTEM: Would you like me help you with reserving table?
USER: Maybe. Is that restaurant in the moderate price range?
SYSTEM: No, it is in the expensive price range. Is this okay, or would you like to try another area?
USER: How about a restaurant that serves Indian food?
SYSTEM: There are several restaurants what part of town would you like it?
USER:
restaurant in the north, please.
Reliable dialog state:
Gold Labels
("restaurant": ("area": "north", "food": "indian", "pricerange": "moderate")
LLM Original Predictions
("restaurant": ("area": "north", "food": "european", "nape"""" "Cambridge City Football
Club Milton Road Chesterton"))
Constructed SFT Sample Output
("restaurant": ("area": ("value": "north", "confidence": "sure"), "food"; ("value": "Eu- ropean", "confidence": "unsure"], "pricerange": {"value": "moderate", " "confidence": "unsure"), } "name": {"value": "Cambridge City Football Club Milton Road Chesterton", "confidence": "unsure"]]]
Explanation
In the LLM's original predictions, the value for area is correct, food is incorrect, name is extraneous, and pricerange is missing. Consequently, in the constructed SFT sample output, area is labeled as "sure," " while the other three are marked "unsure," " with erroneous information left uncorrected.
Table An example of the SFT data construction.
D Refinement Model
We trained an automated refinement model for each dataset. The refinement model con- ducts in-depth step-by-step reasoning unsure predictions based on task-specific criteria, operating within a defined action space. Our refinement models are built upon LLAMA3 8B utilizing SFT training with data constructed using OpenAl's GPT-4o.
Action Space The action space of the refinement model includes three actions: remove, reserve, and correct. A prediction should be reserved if entirely correct, should be corrected partially correct and should be removed if entirely unnecessary. For any unsure predictions, proper action from these three options is sufficient to achieve the required refinement.
Data Construction We adopt a multi-step reasoning Chain-of-Thought (CoT) paradigm. The specific reasoning instructions and processes are detailed in Table 10. For data construc- tion, we used OpenAl's GPT-4o model. During the data generation process, we provided the model with the prediction to be refined and the corresponding golden label, then instructed it to generate reasoning process based predefined principles, shown in Table 11. We combined the instruction, input, the prediction to be refined, the thinking steps, and the refinement result into final training sample. For each dataset, sampled 5,000 examples
15

---

Published as a conference paper at COLM 2025
Input
Dialog history:
USER: Is there modern European restaurant in the north?
SYSTEM: found city stop restaurant. They are located at Cambridge City Football Club
Milton Road Chesterton.
USER: That sounds lovely!
SYSTEM: Would you like me help you with reserving table?
USER: Maybe. Is that restaurant in the moderate price range?
SYSTEM: No, it is in the expensive price range. Is this okay, or would you like to try another area?
USER: How about a restaurant that serves Indian food?
SYSTEM: There are several restaurants what part of town would you like it?
USER:
restaurant in the north, please.
Reliable dialog state:
Gold Labels
("restaurant": ("area": "north", "food": "indian", "pricerange": "moderate")
Multi-Sampling With the Foundation LLM
Sample;
Sample,
Constructed DPO Sample Output Pairs
Chosen
("restaurant": {"area": {"value": "north", "confidence": "sure"), "food": ("value": "in-
dian", "confidence": "sure"), "pricerange": {"value": "moderate", "confidence": "un- sure"]]}
Rejected
{" ("restaurant": ("area": ("value": "north", "confidence": "sure"), "food": ("value": "in- dian", "confidence": "sure"}]}
Explanation
Compared to the rejected sample, the chosen sample exhibits higher recall for correct pricerange within the "unsure" category, aligning more closely with our alignment objectives. This is reflected in higher Weighted-Fi score, justifying its selection over the rejected sample.
Table An example of the DPO data construction.
from the model's s unsure predictions as well as from randomly selected data in the training set, ensuring a balanced distribution data across three actions.
E Baselines Details
Here we detail the specifics of the baselines.
Prompt-based
Direct:
Utilizes prompts similar to those in Table 13 to guide the model in generating predictions directly, without incorporating confidence labels.
Employs prompts akin to those in Figure 13 to direct the model, which outputs confidence (sure or unsure) alongside predictions based on its self- assessment.
Verbose:
16

---

Published as a conference paper at COLM 2025
Self-Reflection (SR): Builds predictions from the Direct baseline, allowing the model to reflect on each prediction to assess its confidence level.
Uncertainty-based
Token Probability (Prob): Computes average token-level probabilities, classifying outputs as "sure" or "unsure" based on threshold t, with probabilities normalized and t set to 0.8.
Self-Consistency (SC): Samples N times, classifying predictions that appear consis- tently at proportion p as "sure," " while others marked as "unsure." We conduct ten samples (N = 10) and set p to 50%.
P(True): Based on predictions from the Direct baseline, this method evaluates the correctness of each prediction individually, marking those with higher probability responding "True" than "False" as "sure," otherwise as "unsure."
Scalability Analysis
The detailed results are presented in Table 7, demonstrating that our method maintains superiority across various capability baselines, achieving more reliable outcomes (higher suec Recall and Quality-F1) and enhanced overall performance post-refinement (higher JGA).
Methed Hype
Method
MullWWZ-244
BTTD
SGD
Precare
Mequsl
Qual F11
JGA |
Ptelure 1
ReCyel
Qual, FL
KiA
Preeeer 1
Requua
Qual F1
JGA
10005umple-Taatee (LAM5
Procape
Direct
9L077
86.25
$7.84
42.75
93.01
92.78
92.52
70.91
62.25
85.10
62.77
38.13
Uncertabrr
92.68
91884
92.37
86.37
43.15
95.70
94.95
92.05
70.37
66.45
91.30
7871
26.62
39.46
Relishilny
(Ours)
cm0 Foot
96.33
95.11
97.15
00.91
99.13
96.22
05 54
95.45
77.00
73.64
66.07
03.44
63.20
41.48
100006
Taiknnd LLAAAA3 Bawe!
Procupt
Diroct
94.51
50.76
Uncerbatttt
Prob
95.77
91.06
92.95
92.13
67.61
67.39
93.95
97.83
95.53
95.23
76.0
$1.26
$4.37
84.61
$3.08
43.52
Reliablliy (Dung)
SET
05.56
94 40
70.85
96.44
96.20
95.05
50.54
85.30
83.02
83.57
45.11
Jolntt
95.63
90.71
70.84
97.15
96.60
9336
82
92.31
82.36
50.75
00:45
Paat
97.71
94.80
71.51
96.59
$2.89
Table 7: Reliability performance of methods on models with higher task capability. Note: the JGA column indicates results after refinement.
G Refinement Cost
The detailed overall results of cost are presented Table and The relationships between cost and performance for the BiTOD and SGD datasets are illustrated in Figure 9a and Figure 9b, respectively, where cost refers to the proportion of unsure predictions.
Meliol Type
Mathed
24llWWZZ22444
ELA
Qween
11AMA
Cheen
AMA
Owven
Mithod Sype
Mathed
BJTOD
SCD
Direct
100.00
100.00
10300
100.00
100.0
101.00
Mediuum
High
Mediuum
High
Prompt
Verbase
$7.20
Promut
Direst
10000
Uscertainty
50
66.36
55.01
58.05
62.59
95.90
92.71
Uncertatsty
Prob
12.17
625
Relability
SF1
PCTrue)
32.07
14.3
30:88
10:23
20:33
13.35
11.7%
$1.24
1.52
55.66
1.71
Relabiity Khhng)
2P0 Joint
Poos
1070
13.55
3,44
6.99
216
1.56
1.6
11:6
2,7
Roxt
27.09
28.38
11.54
18.14
40668
33.00
Table Cost comparison of reliability-trained models using different foundation models: Medium denotes model that has acquired cer- tain task capabilities through SFT with 1,000 task- specific samples, while High refers the 10,000- samples-traine one.
Table 8: Cost comparison of different methods using the original LLAMA3-8B and Qwen2.5-7B models.
17

---

Published as a conference paper at COLM 2025
83
40
30
20
100
Direct
Token Py Corsiseeyyy P(Trus)
Relatiny-SFT
Tokan Protwolity
Pylatsiy-8FT
Verbose
Ratabilty-e SFT-FFTT-- T-OPO Pos:
Verboss 3:1 Ratection
Carsissnyyyy P(Tru))
Rolabilty it 8FT SFT-OPO Post Rollbblty 3FT-OPO Jent
3c Reflection
Potabilty SFT-3PO Jain
BITOD
(b) SGD
Figure 9: Comparison of different methods in terms of performance and cost.
Refinement Examples
Give you a dialog history between USER and SYSTEM, and a slot-value pair extracted from the dialog, I want you to analyze the dialog history and review /refine the given dialog state slot-value pair to make them more accurate and reliable. ## Domain Slot Spaces
"hotel": "name": {
"data tattattpe": str, "example": "hamilton lodge"
}
## Question
Dialog history:
USER: Hello. I need train to London liverpool Street. SYSTEM: Where are you departing from and do you have a time preference? USER: Yes, I'd like to leave Cambridge sometime after 7:00. SYSTEM: I have one that leaves 7:59 and more that depart every two hours after. USER: The 7:59 will be fine.
Predicted slot-value pair:
train: leaveat = 7:59
Refinement:
Thinking steps:
stepl: structure aspect: "leaveat" is one of the slots in the train domain and the value is incorrectly specified "7:59" (formation error). Precise time should strictly follow the "HHMM" format.
step2: semantic aspect: "leaveat" is not a thame"-related slot, so the value must be extracted from the user's 's direct intent. Although the system provides a train that departs at 7:59, the user only mentions that he'd d like to leave Cambridge after 7:00. Considering the formation, the correct value should be 07:00.
step3: domain train correct; slot leaveat correct; value 7:59 incorrect.
step4: the predicted slot-value pair is incorrect in both structure and semantic, and the value
should be corrected.
Refine action: correct
Refined slot-value pair: train: leaveat = 07:00
Table 10: An example of the Refinement process.
18

---

Published as a conference paper at COLM 2025
GPT4o Prompt For Refinement Thinking Steps Generation
Give you dialog history between USER and SYSTEM, a predicted slot-value pair and gold refinement, I want you to analyze on the dialog history, refer to the examples I've provided and the golden refinement, and then continue to provide the thinking steps following the predefined principles.
Firstly, we use json dict to describe the slots and their corresponding value space. Then, we will specify the requirements you need to comply and provide some examples. Last, we will present you with the dialog history, the predicted dialog state slot-value pair and the gold refinement for you perform continuation. ## Domain Slot Spaces
"hotel": {
"name": {
"data -hattttype:: str, "example": "hamilton lodge"
}
## Meta Requirements
Analyze the dialog history and the predicted slot-value pair carefully, correct and refine the pair according to the following Detailed Refinement Principles.
For the given slot-value pair, you should only take one of the actions of "reserve", "remove" or "correct" according to the following principles. a. reserve: if the slot is relevant to the dialog history and the value is correctly specified, you should keep the pair.
b. remove: the slot is irrelevant to the given dialog history or there's not corresponding value could be extracted from the history, you should print "none" indicating that the pair has been removed.
correct: if the slot is relevant to the dialog history and the value incorrectly specified, you should extract the correct value and replace it.
You need to think and reason in the same way as the Examples. Please refer the Examples for formatting requirement. Make sure the output is all lowercase. Only focus the given slot-value pair and do not consider any other slot. only provide the reasonable thinking steps, do not provide any extra prefixes suffixes. ## Detailed Refinement Principles
## Examples
## Question
Dialog history:
USER: SYSTEM:
Predicted slot-value pair:
train: leaveat = 7:59
Refinement:
Refine action: correct
Refined slot-value pair: train: leaveat = 07:00
Thinking steps:
Table 11: The prompt used instruct GPT-4o generate thinking steps for refinement.
H LLM Instructions
The instructions used in this work are presented in this chapter, taking the MultiWOZ-24 dataset as an example. The instructions for the BiTOD dataset are similar.
19

---

Published as a conference paper at COLM 2025
H.1 Inference Instruction
The prompts used with the original untrained LLAMA3 model are shown in Tables 12 and 13. Among them, the Direct Inference Instruction requires the model directly output the prediction results, while the Reliability Inference Instruction additionally requires the model to output confidence label alongside the prediction.
Direct Inference Instruction
Give you dialog history between USER and SYSTEM, I want you to analyze on it and generate the dialog state.
Firstly, we use json dict to describe the slots and their corresponding value space in each domain. Then, we will specify the requirements you need to comply. Last, we demonstrate some cases.
## Domain and Slot Space
The availabel ontology of the dialog state is follows:
"hotel":
"name": {
(data-type) str, "example": "hamilton lodge"
}
## Requirements
Analyze the dialog history carefully and fill the relevant domains and slots.
2. For slots with specified value range, responses must fall within the provided range. For slots without specified value range, the answer must be extracted from the history. Set value as "dontcare" if user doesn't 't have a preference.
3. You only need to consider the domains and slots that are relevant to the conversation history. Do not include those irrelevant in your response and avoid presenting empty domains or slots.
Your answer should also be in one-line jsonl format and make sure the output all lowercase. Do not provide any extra prefixes or suffixes or any explanations. ## Output Dialog State Example
("hotel": :"area": "centre", "name": "alexander bed and breakfast", "parking": "yes", "type": "guesthouse"), "attraction": ("name": "kambar"]]
## Examples shots...
Table 12: Instruction used in Direct, Token probability and Self-consistency baselines.
H.2 Training Instruction
The prompts used with the trained LLAMA3 model are shown in Tables 14 and 15. The Direct Training Instruction designed to train foundation models capable of directly outputting prediction results. In contrast, the Reliability Training Instruction is designed to train models capable of explicitly outputting confidence labels alongside predictions.
20

---

Published as a conference paper at COLM 2025
Reliability Inference Instruction
Give you a dialog history between USER and SYSTEM, I want you to analyze on it and generate the dialog state.
Firstly, we use json dict to describe the slots and their corresponding value space in each domain. Then, we will specify the requirements you need to comply. Last, we demonstrate some use cases.
## Domain and Slot Space
The availabel ontology of the dialog state is as follows:
"hotel": {
"name": {
"data_tppe": str, "example": "hamilton lodge"
}
## Requirements
Analyze the dialog history carefully and fill the relevant domains and slots.
2. For slots with specified value range, responses must fall within the provided range. For slots without specified value range, the answer must be extracted from the history. Set value as "dontcare" if user doesn't 't have a preference.
You should try to cover as many domians and slot-value pairs relevant to the conversation history possible. Mark each slot with either "sure" "unsure" according to your confidence in it. For slot-value pairs that you are pretty sure that they are truly relevant and should be included, and meanwhile you have great confidence in the correctness of the value, you should tag it with "sure". Otherwise, you should tag the slot-value pair with "unsure".
4. Principles:
Goal one: Achieving as close to 100% accuracy possible in those "sure" slot-value pairs.
b. Goal two: The "sure" part along with the "unsure" part should cover all possible slots
involved in the dialog history as much possible.
C. Heavy penalization: Providing incorrect slot-value pairs in the "sure" part.
d. Heavy penalization: Missing slot-value pairs that should be extracted. Providing incorrect or redundant slot-value pairs in the "unsure"
e. Light penalization:
part.
5. Your answer should also be in one-line jsonl format and make sure the output is all lowercase. Do not provide any extra prefixes suffixes any explanations. Please refer to the Output Dialog State Example for formatting requirement. ## Output Dialog State Example
["hotel": ("area": ("value": "centre", , "confidence"; "sure"), "name": ("value": "alexander bed and breakfast", "confidence": "sure"), "parking", ("value": "yes", "confidence": "unsure"), "type": ("value": "gguestouse", "confidence"" "sure")], "attraction": {"name": ("value": "kambar", "confidence": "unsuree))] ## Examples shots...
Table 13: Instruction used with original LLAMA model in Verbose baselines.
Direct Training Instruction
Generate the dialogue state based on the given dialogue context.
Table Instruction used Direct SFT Training.
21

---

Published as a conference paper at COLM 2025
Reliability Training Instruction
Generate the dialogue state based on the given dialogue context. Ensure the results are as reliable possible, the 'sure' parts are accurate as possible, and the overall coverage includes all relevant slots.
Table 15: Instruction used in Reliable Training, guiding the model to explicitly extinguish between "sure" and "unsure" responses.
22


Reviews:
[{"review_id": "XQVmHAhC3V", "reviewer": "Reviewer_GHeu", "summary": "The work tries to make LLM generations more reliable and less hallucinated by generating \"sure\" and \"unsure\" labels. The unsure labels then undergo a COT refinement process. The method first labels correct responses as \"sure\" and incorrect as \"unsure\". Then performs DPO on after collecting the training data. It then uses a refinement model to correct the output for \"unsure\" labelled generations.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel technique for improving alignment in language models. \n- Performs extensive evaluation and shows that the method has its merit.", "reasons_to_reject": "- The motivation for this problem is unclear. The motivation and the examples presented in the paper can be solved by providing specs for the model. In Figure 1, why is 7:59 not a valid answer?\n- Although the work talks about using sampling for collecting sure and unsure labels, the work samples only one generation and labels all the correct output as \"sure\" and all the incorrect ones as \"unsure\". This goes against their motivation of having \"unsure\" for correct and incorrect generations.", "questions_to_authors": "- Are all the performance gains statistically significant?\n- Any intuition why the improvement with SGD is low?\nCan you look at the logits for sure and unsure to better understand? There can be cases where the model spits sure, but the probability is just marginally higher than unsure. \n- \\`\\` '' for quotes in latex.\n- Can you apply this method for a more meaningful task?", "extra_scores": {}}, {"review_id": "eEQqYhAzUe", "reviewer": "Reviewer_MJZ5", "summary": "Authors introduce a method for better dialogue state tracking (DST) in the context of task-oriented dialogue systems. The broader principle is to have models scrutinize their own outputs, rating their own certainty, and doing additional refinement on outputs that the model deems it's unsure of. Although tested only on DST in the context of this paper, the principle may have wider applicability. The method the authors propose is composed of two components:\n\na) additional training to increase accuracy of sure/unsure self-eval (via both SFT and DPO)\nb) training a refinement model to refine the unsure responses\n\nAuthors hope to add additional nuance to \"classical\" selective prediction (where the model either predicts or abstains) by transforming unsure outputs into something that is still partially useful for the user.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Method is validated quite comprehensively on DST datasets.\n\nEmpirical performance gains are significant and clearly demonstrated.\n\nBroader principle is interesting.", "reasons_to_reject": "Authors make implicit claims about broad applicability yet only test on DST. I think some of the claims/framing needs to be toned down. \n\nRefinement model training is essentially a distillation of GPT-4o reasoning traces, of a closed model which limits reproducibility.", "questions_to_authors": "Fig. 2 \"Resoning steps\" -- typo", "extra_scores": {}}, {"review_id": "MJLFkChH3z", "reviewer": "Reviewer_xXcS", "summary": "This paper proposes to use a reliability model to improve task oriented dialogue systems. Using such a judge model is not a new idea (e.g.,  Dey et al., Accountability Model) but clearly it is impactful and necessary for any agentic model.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper seems to be revised multiple times with extensive experimental results and ablations. As is, I believe it is ready for COLM. Maybe the authors can consider a newer benchmark dataset like Tau-Bench to add.", "reasons_to_reject": "Most agentic systems have severe latency requirements. Having a judge model may not be feasible. It would have been good if the authors also show latency numbers and solutions for that such as caching.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g3kqJ9CZYD", "reviewer": "Reviewer_gFAY", "summary": "The authors propose a method to increase the generation reliability of LLMs by first training the model via SFT and DPO to produce confidence-labeled responses (using a binary schema of \"sure\" and \"unsure\") and then refining unsure responses in a second step with a separate refinement model. This approach explicitly models the inherent trade-off between reliability (producing correct responses) and helpfulness (producing responses as often as possible instead of deflecting) and the authors present various experiments illustrating this trade-off. The method outperforms existing methods for uncertainty-based rejection on three different dialog state tracking datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and clearly outlines the motivation and methodology. The experiments are informative and clearly demonstrate the utility of the proposed method. I don't see any flaws in the experimental setup. The method is a valuable contribution to increase the reliability of LLM generations.", "reasons_to_reject": "The paper could explain more clearly how exactly the methodology is applied to the datasets used in the experiments, the details are in the paper but almost all are buried in the appendix. The paper would also be stronger if the experiments would cover a broader range of tasks (especially given that most parts of the paper present the methodology very generically without references to DST). It further remains unclear if previous work also evaluated in the same setup or if this paper is the first one to apply the selected baselines in this setting.", "questions_to_authors": "Please comment on the points mentioned under reasons to reject", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the Explicit Knowledge Boundary Modeling (EKBM) framework, a two-stage process designed to improve the reliability of LLMs and reduce hallucinations. The system first uses a "fast-thinking" model to generate answers and label each part of the output with a confidence level, either "sure" or "unsure." High-confidence "sure" outputs can be used immediately, while "unsure" outputs are passed to a second, "slow-thinking" refinement model for more deliberate analysis and correction. The framework was tested on dialogue state tracking (DST) tasks, showing that it improves the model's ability to assess its own accuracy and boosts overall performance.

Overall, the agreement among reviewers is that the paper has a sound experimental setup, makes substantial contributions to the scientific community, and presents solid empirical results.Upon the reviewers' request, the authors further show that the performance improvements are statistically significant. Weaknesses identified by the reviewers included a narrow experimental setup focused on DST tasks, but the authors successfully addressed this by expanding the experiments to mathematical reasoning. The authors also extended their findings to open-source models (from the initial GPT-centric experiments). Other issues, such as clarity and paper presentation, can be easily addressed for the camera-ready version.

Overall, I agree with the reviewers' recommendation to accept this paper.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept