Conformal Linguistic Calibration: Trading-off between Factuality and Specificity
Authors: Zhengping Jiang
Anqi Liu
Benjamin Van Durme
Venue: NeurIPS 2025 poster ()
Keywords: Conformal Prediction
Calibration
Semantics
NLP
LLM
Factuality
Primary area: applications
TL;DR: We propose a new method for LLMs to express uncertainty by adjusting the specificity of their responses.

Abstract: Language model outputs are not always reliable, thus prompting research into how to adapt model responses based on uncertainty. Common approaches include: abstention, where models refrain from generating responses when uncertain; and linguistic calibration, where models hedge their statements using uncertainty quantifiers. However, abstention can withhold valuable information, while linguistically calibrated responses are often challenging to leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinterprets linguistic calibration as answer set prediction. First we present a  framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics. We then describe an implementation of CLC that allows for controlling the level of imprecision in model responses.  Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-aware adaptive claim rewriting, offering a controllable balance between factuality and specificity.

Conformal Linguistic Calibration: Trading-off between Factuality and Specificity
Zhengping Jiang Johns Hopkins University zjiang31@jhu. edu
Anqi Liu Johns Hopkins University aliu. cs@jhu. edu
Benjamin Van Durme Johns Hopkins University vandurme@jhu edu
Abstract
Language model outputs are not always reliable, thus prompting research into how adapt model responses based uncertainty. Common approaches include: abstention, where models refrain from generating responses when uncertain; and linguistic calibration, where models hedge their statements using uncertainty quan- tifiers. However, abstention can withhold valuable information, while linguistically calibrated responses often challenging leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinter- prets linguistic calibration as answer set prediction. First we present framework connecting abstention and linguistic calibration through the lens of linguistic prag- matics. We then describe an implementation of CL.C that allows for controlling the level of imprecision in model responses. Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-awant adaptive claim rewriting, offering a controllable balance between factuality and specificity
Introduction
Large language models (LLMs) can provide knowledge responses based on their comprehensive
training sets [4% 59 78 but these responses may not be trustworthy [41] 26], and can be stated in an over-confident tone [2). One solution to communicate calibrated uncertainty measure to the user, as to enable more informed decision-making 80). This uncertainty information can be expressed in different ways, which we hereby classify into two groups. The first set of approaches 65). When such uncertainty expressions are conveyed through text generation and are adjusted to faithfully represent accuracy of the response, this is called linguistic calibration [42] 6 69]. For example, model uncertain in its response may state: "Possibly the largest city in the US is Los Angeles" (Fig 1 However, how interpret linguistic uncertainty can be unclear and subjective. Also, the outer structure introduced by linguistic calibration hampers the downstream task / evaluation [34,, as it is unclear what special treatment should be given to these uncertain assertions. Another set of approaches have the model generate prediction sets instead of a single response, such that with high probability some acceptable [50
marks response explicitly with the model's level of confidence [66] 12
32
55 A special case of this is when model abstains from responding when it
is not confident 58 28
39
E.g., don't know" is trivially equivalent to predicting the 45 18
'Code available at: ht tps: htpp://githig /gtthtttthhttttgggg www/cooo/oa///e com/
Answwr by population is New York City, by area is Sinka.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

set of all possible answers. However, this breaks the normal human- monal interaction paradigm, and the set prediction
directly actionable [T].
To address these limitations, we propose a novel unifying view, Conformal Linguistic Calibration: having the model express uncertainty levels by gener-
ating less specific claims until they are confident. We
achieve this by reinterpreting uncertainty quantifiers as set prediction using recent advances [99 in possi-
Q:
What is the largest US city?
ble world semantics 30 29 At high level, being more confident in a claim means that subject would only regard it as false counterfactual worlds that are drastically different from the actual one. These grad- ually less similar counterfactual worlds form a series of nested sets corresponding to the subject's belief From this perspective, linguistic calibration can be seen form of soft abstention, whereas tradi- tional abstention [28] 44 corresponds to committing to the universal set of all possible worlds. As shown in inFFiguee I] this allows us to improve the flexibility and controllability over an abstention -based approach. Also, our formulation naturally supports conformal prediction techniques [.. offering probabilistic guarantees on factuality while producing confident, clear claims that easy for downstream In summary, we:
Lingnistic Cal.
+
Possibly Angeles
Calibration
Angeles
36 63
Abstention
Angelees
CLC (This Work)
coastal city
Figure 1: Distinct from previous approaches CLC allows generating a less precise but more confident statement.
1. Propose a new paradigm for model to communicate their uncertainty.
2. Provide an algorithmic instantiation of the process by probing semantic uncertainty [3]] building nested subsets, and summarizing beliefs, which can be post-hoc calibrated through learn-then-test (LTT)
3. Show that our approach enables smooth trade-off between factuality and specificity [74] for example, allowing 7B model to outperform GPT-4o in factuality on challenging QA dataset by answering less specifically [7].
4. Train an amortized, uncertainty-aware claim rewriter that explicitly trades off specificity for factuality following user instructions. We release both our rewriting model and dataset.
2 Related Work
LLM Factuality Evaluation Large language models enable rich long-form generation S
yet hallucinations remain persistent [41 26 24 To monitor factual precision, evaluation pipelines
typically decompose model outputs into atomic claims and verify each against trusted sources or
serrival-auugmente evidence 43 71 72 Subsequent work sharpens these pipelines by improving
claim extraction and coverage 773 21 27), but the focus remains on enumerating verifiably true
statements. As result, they offer limited guidance on how to modulate hedging other forms of controlled imprecision that influence downstream usability
Conformal Prediction and Risk Control
frameworks complement precision-centric
evaluation by directly bounding the probability erroneous outputs. Conformal prediction has been adapted to over- generation, abstention, delegation, and clarification settings in language applications
[50 28 20 49 [7 [56]. providing explicit coverage guarantees when the model defers, expands,
or filters its answers. The closest approach to ours, Mohri and Hashimoto 44]. drops uncertain claims to improve faithfulness, and Cherian al. 110 refine the procedure via adaptive conformal prediction. We maintain the original answer space but rewrite claims so that conformal guarantees translate into interpretable hedging, allowing users to trade specificity for reliability while retaining coherent narratives.

---

3
Preliminaries
We focus on the setting where given a prompt z, a language model L generates response y = (a) where y E y. The overall goal is to find a post processor T : y + v that ensures a probabilistic guarantee for a user specified probability a E (0, 1):
(F( is factuality correct) 2 - a.
One particular challenge is to find a T that will almost always work without too many constraints on V. For example Mohri and Hashimoto [44] shows that for longer generations one can come up with simple solution for 'I where one can drop subset of claims, process they called back-off. but this will not work for more atomic generations like in question answering, as back-off highly restrained by explicitly stated claims. Yet atomic-claim-leve operation is very desirable, previous works have demonstrated the benefit of decomposition for various fact verification problems [43 To allow being less specific beyond the surface form, we need a more sophisticated process for identifying plausible alternatives to the input claim to guide post- post-processing. We now describe how the post-prot pest-prcessing step T can be formalized in terms of belief. This helps connect back linguistic calibration, and we provide a way to achieve our desired guarantee despite the additional complexity proposal introduces.
27
53
73 64].
3.1 Possible World Semantics
The notion of a possible world has long tradition in philosophy, described the "limit of series of increasingly more inclusive situations" [16]. Following Kripke [29]. modal expressions are evaluated by asking which alternative situations remain accessible from the world we currently inhabit. We therefore treat a claim the set of worlds in which it remains true: relaxing the claim (i.e., hedging) enlarges that set, lowering the risk of factual error while preserving informativeness.
Formally, Kripke model M = (W; IH) consists of a set of possible worlds W, an accessibility relation R where R(w, w. u) indicates that world accessible from w, and a satisfaction relation It that evaluates formulas at a world. These ingredients let us define the familiar necessity and possibility operators for a claim +
Definition 3.1 (Necessity). Under M at w E W, a necessitation De is true iff Vu E W such that R(w, u), we have u IF
The "possibly" operator O then follows in the usual way: Definition 3.2 (Possibility).
Oc - -
Accessibility relations organize the worlds around into concentric spheres that reflect increasingly remote departures from what the agent regards as normal or likely. A claim is necessary within sphere it holds throughout that sphere, and merely possible if it holds least one world contained in it. Different modals can therefore adopt different relations: Hintikka 2 extends Kripke semantics belief with a plausibility- model graded confidence, Goodman [19] combines such relations with similarity orderings, where u ZA.w means that agent A judges u to be at least as similar to w as is. The resulting nested family
relation R3 calibrated to empirical constraints on human belief. To
SA, = {fu W s.t. u A tn s.t. U W W
collects precisely these spheres 35 Given subjective probability Pra,w: one can associate each confidence level d with the set of worlds
Cont (w) -NpE SA,0 s.t. PrA (p) d).
which is itself nested and can be repackaged as an accessibility relation that yields graded confidence modals. Viewing claims nested sets of worlds thus reveals a direct bridge between claim generation and set prediction, allowing us to adapt conformal prediction techniques for risk control in Section 4 We defer richer illustrations of this perspective the expanded revision, where additional space available.
Throughouu this work view claim nullary predicate that does not take arguments.
3

---

3.2 Conformal Prediction
Split Conformal Prediction Vovk al. 667, Shafer and Vovk [61]. Papadopoulos provide standardized tools to construct prediction sets that provide coverage guarantees. Concretely, using calibration dataset (X,, i=) split conformal prediction gurantees that for i.i.d. sample (Xess, Yees) with a prediction set ((Xust) E 2" then for any designated target threshold a E (0,
PP E ((Xee) 2 11-a. a.
(1)
Following the view of Gupta et al. [22]. the split conformal prediction procedure starts from sequence of nested candidate output sets, and use calibration data to select the correct level of the
nested set until the coverage guarantee is achieved.
However, this approach requires that the prediction sets to select from are nested, or similarly. the non-conformity score, or equivalently the set construction is by thresholding on sequence of monotonous non-conformity scores [2. Instead of using the quantiles of scoring function, a more general extension of conformal prediction called Searn-Thee--ieee (LTT) [1] relies on hypothesis testing to identify a viable region to control any nyper-parameter sensitive risk.
(LTT) extends conformal prediction to find hyperparameter configuration control the expectation of any risk function R such that
P(sup RT(A) - <)
xEA
Unlike conformal risk control 22, LTT does rely on the risk function being monotonous A. To achieve this, LTT associates the null hypothesis: HA R(A) > to each configuration X, and calculates a conservative p-value [ for each of the hypotheses, from which the LTT guarantee directly follows.
Methods
In this section, we outline the procedure for deriving the risk-controlled process T. as illustrated in Figure 2] Building on our previous discussion in Section 3.1 the objective is leverage conformal prediction techniques to manage the risk associated with adherence each level of the nested sphere. The underlying intuition is that the confidence level V can equivalently be represented by claim č that describes the sphere V We formalize this property follows:
Definition 4.1 (Description). A claim è is said to describe sphere V iff
(yy)) A (-)w)v
(2)
Thus given set of source claims {c1, C2...., CN }. we aim to rewrite them to set of less specific (or imprecise) claims (b1, ba.................. ... by that each properly describes its corresponding possible world set (sphere) (W1, W2. WN3 such that for any designated target threshold (0, 1)
IP WwclW w.] 2 11-a.
which matches the guarantee in Equation (T. However, there is no practical way directly evaluate ae as there is no way to constructively derive the set Wo. In this section, we describe a data processing pipeline that for each claim e derives a sequence of candidate target claims b2. 12 Vie..... bk
such that the corresponding possible world sets in theory satisfy the nested assumption W/1 C W C W.K
4.1 Less Specific Rewriting
In this section, we outline the pipeline to derive less specific claim from a source answer string, with the goal of having the factuality risk of the generated claim properly controlled. We call this process claim rewriting. Our claim data is sourced from QA datasets, motivated by recent advancements in decomposition 772 21), which demonstrate that complex text can be broken down into relatively simple, targeted questions g 76.. Furthermore, prior work has established that conversion between questions and claims is feasible and natural
"With a slight abuse notation, we denote the necessity operator associated with modal logic M (W, Ry, IH) by Dp. where the accessibility relation is defined as Ry: (w,, u) = Il[u U... A similar convention applies the possibility operator O.

---

Mentify
Nested
Beliff Proking
Callration
Aocuacy
Q: Whatis the lagest
New Yolk City
Los Angslox
Change
Angoka
NVC
Chrcago
Chiogg
Chicago
New York
Iarget the United Stuscs conecal cit
Maliplicyy
Figure 2: The overview of our conformal linguistic calibration approach. Instead of relying direct operations like subclaim drop-off, we probe the model's 's internal belief by semantically clustering all sampled answers into nested sets, and writing less specific claims that are associated each answer set level through Equation
Answer Sampling Given a question q. we repeatedly sample K responses from language model T. denoted as o% This approach is standard technique for estimating predictive uncertainty in natural language generation 70 31
Identifying Clusters From the sampled answer set fa}, we identify semantically unique clusters
4==1 31]. Previous approaches typically employ an entailment model or its extensions to establish
equivalence relationship-suu as bidirectional entailment-to induce clusters [31 [8]. However, while the number of clusters identified using this method serves as a useful indicator of response uncertainty, the clustering itself tends to be quite noisy. This is partially due to the inherent difficulty of fully defining an equivalence relationship in real-world scenarios. When operationalized through Natural Language Inference (NLI) model [13 40,, such relationships often suffer from lack of deep semantic understanding and excessive sensitivity to surface variations.
To address these issues, instead use single LLM call to directly generate list of identifiable unique answer cluster names from {a}. The details of this prompt, along with other relevant prompts, provided in Appendix A
Estimating Answer Multiplicity Given the answer set {a} and a set of representative unique answers Z C {a), we estimate the multiplicity of each cluster representative z; Z by counting the number of answers that are semantically equivalent to the corresponding cluster name. The cluster assignment 5, (ax) is determined using similarity metric s(;, :) E [0, 1]. which assigns ag to the i-th cluster such that
- {
if i arg max s(ax; 2;)
62((x)
otherwise
with arbitrary tiebreak. And the multiplicity is simply given through m(z:) 1 8a()).
We acknowledge that while accurately estimating m(c) is important, minor noise in answer cluster assignment does not compromise the validity of the pipeline. This is because the entire rewriting process is calibrated using the Learn- Then-Tes (LTT) framework, as described in Section
Constructing Nested Cluster Sets In line with the approach of Wang et al. 70 we employ majority voting to identify the most confident answer, which we designate as the original answer æ where
2 = arg max m(x))
2, = }
We then incrementally add other clusters into the set to form a set of nested cluster sets
2, c 2 c ... C ZN.
To better align with the interpretation described in Section 3.1 we prioritize clusters that are se- mantically closer to the representative of the most frequent cluster. For example, given the question "When did Brexit happen?", if the most confident answer is E "2020" and other unique responses include "2016" and "2019", we would prefer to include "2019" first due to its temporal proximity, even if "2016" appears more frequently. Notice that while conformal prediction for probabilistic
For simplicity, we omit the superscript when the dependence on g clear.
5

---

classification typically constructs the predictive set incrementally by adding classes in the reverse order of their predicted membership probabilities, we find that this approach results in a predictive set that is difficult to distinguish from the remaining answers, making the task of rewriting less specific claim unnecessarily challenging.
We observe that embedding baasdd similarity metrics often fail to accurately capture spatial, temporal, or numerical distances. To address this, we propose LLM-based incremental selection scheme, in which an LLM is repeatedly prompted to select the cluster names most similar to those already included. While ideally, we would extend the predictive set by one cluster at time, doing for large N would be computationally prohibitive. Instead, we begin with a predefined set of target thresholds A1 , X and, given a subset Zn, already selected, we prompt the LLM to select an additional d items, where
Z Zz
d d= min (01-MM(((2) 22n) XEA
s.t. Mult(Z,) > 0,
Mult(Z Z 2n)
where Mult(?) =x Eeez m(z). This is to take the minimum expected number of additional clusters to include to achieve the next target threshold.
Belief Probing The final step in the claim rewriting process involves associating more general claim with each nested cluster set 7,, using rewriting function f 2 + y. To accomplish this, we verbalize both Zn and its complement Q \ Z, as the beliefs of a hypothesized question question-answerte answering agent. We then prompt the LLM to summarize this belief a less specific claim b, that arims to satisfy Equation (2).
We find that explicitly framing these clusters the belief of an error-prone agent- than as objective facts is crucial (See Table 3 in Appendix A). Without this outer structure, the LLM generating the claim often disregards clusters that contain non con-facuaal responses, making faithful belief generation challenging. This belief summarization process helps the model adhere to the coherence theory of truth 72 The theoretical validity of this approach is established in the following theorem.
Theorem 4.2. For claims b1 that describe V1 and b2 that describes V2, b + b2 then
V12 C V/2
The proof follows directly from the argument presented in Appendix B While our approach performs well in practice, the unconstrained nature of the prompting mechanism means that it does not inherently guarantee f(2,) + f(Z,) for all j > i. This limitation further justifies our choice to calibrate using the Learn-Then- Test (LTT) framework [I] rather than simpler method like Conformal Risk Control (2). Still the validity of the summaries has been confirmed by small-scale manual verification conducted by one of the authors. We sampled 11 instances from each of five question types of SimpleQA 774 ("Date," "Number," "Place." " "Person," "Other") and asked a human annotator to identify all answers entailed by the LLM-generated belief. Two "Person" questions were excluded due to vague criteria involving subjective judgments (c.g.. whether name belongs to a famous chemist or is "traditional" surname), On the remaining examples, annotation precision and recall were relatively high: 0.81 and 0.91, respectively.
4.2 Conformalizing Rewriting with LTT
After generating a sequence of progressively less precise claims starting from the most frequent answer cluster for each question, we apply Leam-Then-Test (LTT) [ to linguistically calibrate the response, ensuring it aligns with the optimal specificity level while maintaining the desired factuality guarantee.
Risk Score In theory, the expectation of any loss function 1, where the risk is defined as R(F x[F EI(F o L(x), Y),, satisfies the requirements for L However, for a controlled comparison, we specifically focus on factuality metries that do penalize generality. Many automatic evaluation metrics are overly rigid, as they fail to account for semantic equivalence (c.g., exact match) or reject
We use to denote function composition.

---

claims that differ in specificity from the gold target 74] In LLLM-basee evaluations, this issue can often be mitigated by slight modifications the evaluation prompt, as detailed in Appendix A] to yield to following loss function:
[(. y) = { 1, if ÿ is admissable
o. otherwise
In this setting, naturally use the multiplicity threshold as the hyperparameter X. Specifically
(x1) = f(Z,)).
where n' arg min Multi(
Multi 2..) 2 A.
Notice that due the discrete natural of M, on the same question g, different might lead the model generalize the original claim to the same vaguer claim. Then given a the goal is to find valid À such that
P (R o L) S 2
with regard to risk-tolerance 5 and error level a. This called by Angelopoulos al. as an (a. l--risk-controlliig pereeiiiing--edicctoonn prediction (RCP). Following Bates et al. [4, we calculate Hoeffding-Benttee inequality p-values.
To summarize, for a given claim e and designated risk control level a, our method finds Aa and corresponding (c) to form a four element tuple (c, Aa, (c))..
5 Experiments and Results
We conduct three sets of experiments to validate key claims in the paper. Section 5.1 shows that by carefully controlling the LLMs to respond less specifically, we can greatly improve their factuality and maintain valid guarantee over factual errors. Section 5.2 further demonstrates that this specificity- decreasing rewriting indeed reduces new information provided by responses from LLM responses.
Since the CLC process involves oversampling and multiple rounds of LLM processing- can be computationally expensive-we also explore, in a different domain, whether this procedure can be distilled into the [.LM itself to enable efficient, high-quality rewrites, shown in Section 5.3
5.1 Linguistically Calibrated Question Answering
Dataset We apply the Conformal Linguistic Calibration pipeline described in Section 4 to two Question Answering (QA) tasks, and examine how gradually less specific rewritings affect the factuality of the LLM response: SimpleQA [74] is a recent benchmark designed to evaluate language models' ability to answer short, fact-seeking questions. Despite its simple format, the dataset presents significant challenge, as the answers are adversarially curated against GPT-4. The authors demon- strate that even state-of-thh-art models like GPT-4o perform poorly on this benchmark, achieving only 38.2 2% accuracy while exhibiting overconfident predictions. For our experiments, we create calibration set and test set making sure that each answer type is split evenly. Natural Question (33 a widely studied QA dataset with short or long answers paired with real search queries issued to the Google search engine. To better align with the SimpleQA setting, we focus on evaluating subset of NQ dev that admits short answers that are not yes or no. As NQ can have more than one human annotation for the correct short answer, we consider an LLM response factual if it is evaluated as consistent against any one of the human- extraneet correct answers.
Evaluation Metric While traditional QAs are typically evaluated with exact matches (EM) or semantic equivalence [79], as discussed in Section 4.2 we need a coverage metric that does not unfairly penalize responses that are less specific than the gold answers, as detailed in Appendix A We thus modify the evaluation prompt from SimpleQA 77 to evaluate whether the less specilic response from the model is relevant and consistent with corresponding gold answer, similar to factuality evaluation [43] 644. We use Llama3-8B Hamma--B-IInssee 8B-Instruct [15] base model C, generating K 100 responses per question. We then produce progressively less precise claims as described in Section 4.1 using GPT-4o while targeting multiplicities 7 E {20,30, 30, 40. 50, 60, 70, 80, 90, 100). Notably, this process can still result in finer-grained set of cluster levels than the predefined target multiplicities. Consequently, during the LTT stage, the controlling multiplicity threshold parameter any integer within (0, 100).
7

---

Baseline We compare our approach against a standard abstention baseline, where the model may choose not to answer its confidence below a certain threshold. Under this setting, we define the percentage characterizing how much the model back off from the original generation as the percentage of questions the model abstain from answering, and we compare performance with our approaches at the same marginal answer set coverage percentage. That is, an abstantion model that answers percent of questions are compared against a CLC model that responds with more generic answers at the level that on average covers p percent of the sampled answer set. Notice that As P - 1, both approaches provide little information, and as p + 0 both approaches converge to not modifying original answers at all. To make sure the comparison is fair, for each question use the same set of 100 sampled responses to quantify answer uncertainty for both approaches. and use the multiplicity of the top-1 I answer the confidence score. Since the answer formats in both datasets are short and skomic-admitte single-claim generation setting described by Mohri and Hashimoto 44 The corresponding results are shown in the left and middle plots of Figure 3]
no further decomposition [72] setup, in the limit, aligns with the
Our results show that for both SimpleQA (Figure Left) and NQ (Figure 3 Middle), the factuality of responses from our approach tightly aligns with the lower bound of the conformal guarantee region, indicating it avoids generating trivially true answers. Compared to traditional abstention, our method offers three advantages. First, while abstention improves factuality by answering only a small subset of questions, CLC answers all questions while still enhancing factuality. Second, challenging datasets like SimpleQA- where models struggle to confidently answer even a few questions our method significantly outperforms abstention. Third, CLC yields more consistent factuality gains as the relaxation percentage increases, whereas abstention becomes unstable at higher abstention ratios.
5.2 Informativeness Scoring
We conduct further evaluation to show that the improvements in factuality comes in trade-off with less specificity. To achieve this, we evaluate the informativeness of our generation at different answer multiplicity. We adopt the Conditional Pointwise Mutual Information (CPMI) 27 as the informativeness measure, which quantifies informativeness as the negative log likelihood of generation given what was already known in the prompt. We operationalize CPMI with publically- available general conditional probability estimator [68] To make sure that the conditional probability is not confounded by the parametric knowledge, we call Qwen2.5-72B-insste ------ittttt to mask personal entity with "PersonX", similar to how ATOMIC commonsense reasoning data is created [60] We also compare our approach to naive rewriting baseline where the base model (lamma---b-IInstuuct) is prompted to directly rewrite the response to boost confidence at least equal to the multiplicity.
The result is shown in Figure 4 For both SimpleQA and NQ, while the factuality of the responses in- crease with growing multiplicity threshold, their CPMI steadily decrease. Our results demonstrate that CLC enables explicit factuality-infe informaiiiiommiieeeesss trade-off. What's more, the naive rewriting baseline failed to facilitate fine- grained trade-off, and easily degenerates to rewrites with low informativeness.
5.3 Claim Rewriting on FActScore
Since the original CLC process is costly as it requires answer over sampling and a pipeline for claim rewriting, we further show that this process can be amortized. To this end, we fine-tune Llama-3-8B-Insteeee
[13] on the synthetic data generated in Section 5.1 which consists of 2,042 instances from SimpleQA and 1,728 instances from NQ with various level of back-off generation. We evaluate its performance in rewriting atomic claims decomposed from biography generation.
Specifically, we assess changes in FActScore 43 before and after applying the claim rewriter T to all decomposed atomic sub-claims. FActScore is a method that evaluate the precision of supported claims decomposed from generation. For evaluation, of the FActScore dataset, which consists of an extended list of biography topics beyond those in the original FActScore dataset and includes frequency values associated with each topic.
use the dev split of the Core (27 expansion
As shown in Table 3

---

0.5
0.7
Factuality (Simple0m)
CLC
Abstention
Factuality (NOT
GPT-40
FActScore
0.3
Guarantee
0.5
0.7
base
0.2
0.4
CLC
Abstention Guarantee
DPO
ORPO
80
"guess"
"s.w,"
"conf."
"cert."
Percentage (%)
Percentage (%)
Controlled Confidence Level
Figure 3: Left: On SimpleQA, the prediction risk on the test set falls within the conformal guarantee region and remains closely aligned with the respective lower accuracy thresholds, while outperforms abstention by a large margin We sample answers to the same question, "Pepi Litman was born in ...", different multiplicity thresholds:
"Kiev / Kyiv." X), "a major cultural and historical center in central or eastern Ukraine." x), "a major cultural and historical center of Ukraine." " V). This demonstrates how rewriting process effectively trades factual specificity for controlled imprecision. Middle: On datasets where the model already achieves non-trivial performance, such as Natural Questions, our guarantee remains valid, and the model still benefits significantly from adjusting precision levels. Abstention does not steadily improves factuality. Right: Fine-tuning an LLM the imprecise claim rewriting data enables adaptive rewriting. While instruction models exhibit better alignment with designated confidence levels, all models trained on our claim rewriting data adhere confidence specifications more effectively than out-of-the-box model. Dash line indicates the out-of out-of-the-boo performance without CLC.
Given the series weaknesses identified in the standard FActScore configuratior restrict verification to Core-filtered (27] subclaims. These subclaims are assessed against BM25 [57]- retrieved text chunks from full Wikipedia dump.
7
NQ (CLC)
Naive
CPMI
5
SimpleQA (CLC)
Naive
1
20
40
60
80
100
1
20
40
60
80
100
Multiplicity Threshold
Figure 4: Average CPMI scoring of claims for SimpleQA (left) and NQ (right). CLC rewriting results in smooth and steady decrease in CPMI scores while naive rewriting does not, Shades corresponds to 95% confidence interval.
We compare the following four claim rewriters trained on 2 A100 80G:
Thmee: This rewriter directly prompts the off-the-shelf Llama3 8B model with the instruction: "Rewrite the following claim be less specific until you verbal 1 matios it is true: claim."
TSFT: This model is fine-tuned using the same prompt on triplets of the form (verbalia iiaaiiiioos claim, newrite-at-o derived from the SimpleQA and Natural Questions rewriting data. The threshold À for each verbalized uncertainty level determined based on an analysis the conformal guarantee presented in Figure 3
TDPo: Following best practices for Direct Preference Optimization (DPO) 52 51 54), further fine-
tune TSeT on preference data. Specifically, we always prefer rewrite-at-A over alternative rewritings that exhibit noticeable deviations.
Including susceptibility to trivial claims, inability to verify claims involving secondary entities
75
62
81

---

TORPO: Odds Ratio Preference Optimization (ORPO) 225 offers an alternative alignment approach by applying minor penalty disfavored generation styles the SFT stage. We train this model using the same dataset as TDPO-
The FActScore of the rewritten generations is presented in the right panel of Figure 3 The transition from "guess" through "somewhat confident" and "confident" to "certain" indicates the gradual raising of the required confidence level 12 By generalizing individual claims, all of our claim rewriters effectively enhance the FActScore of the generated outputs. In contrast, the out-of-the-box Llama model (TBase) does not consistently follow the instruction to rewrite claims to the designated uncertainty level. It is also interesting that the tuned model seems to learn to adapt its level of rewriting specificity based on the difficulty of the claim, as shown in Appendix
6 Conclusion
We propose a novel linguistic calibration paradigm that enables models to express and communicate uncertainty through imprecise language. Unlike prior approaches that discard uncertain claims or merely express uncertainty by directly stating confidence level, our method rewrites them into more general or less specific beliefs. Our results show that, unlike prior uncertainty-aware risk control methods, CLC enables models to respond unanbiguously to all questions while steadily and effectively improving factuality and maintaining valid risk control guarantees.
We further demonstrate that conformal linguistic calibration can be fine-tuned into base model, enabling it to adaptively adjust claim specificity based on confidence levels. We believe this work will also encourage a more nuanced evaluation of automatic factuality pipelines, emphasizing the need to account for factors beyond factual aceuracy. Future work will expand this analysis to examine how Al confidence communication shapes human peeiisionmmaking while acknowledging the challenges of conducting human-base based factuality trade-off evaluations due to known biases
7 Acknowledgements
AL and BVD are partially supported by ONR grant N00014-241-2088
References
[1] Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms achieve risk control. arXiv preprint 1rX00:2110.01052, 2021.
[2] Anastasios Nikolas Angelopoulos. Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In The Twelfth International Conference Learning Representations, 2023.
[3] Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. Linguistic calibration of long- form generations. In Forty-first International Conference on Machine Learning, 2024. URL https:://peerrvview httpi/ewwweee.oooomm wtt.://ooorumm1dddr1300000
[4] Stephen Bates, Anastasios Angelopoulos. Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-fiee rist-controlling prediction sets. Journal of the ACM (JACM), 68(6))1-34, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Arslan Chaudhry, Sridhar Thiagarajan. and Dilan Gorur, Finetuning language models emit linguistic expressions of uncertainty, arXiv preprint 1xXXi:2409.112880 2024.
Please refer to Table 5
Appendix A to see the exeact verbalization of each confidence level.
10

---

[7] Jifan Chen, Eunsol Choi, and Greg Durrett. Can NLI models verify QA systems' predictions? In Sarrie-rancnne Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors. Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3841-3854, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 18533v1/20022 htt.//2...nnngeeeeeim.. findings-emmlp.322. URL htttps:://alaaattoooogyy http////orr.///////
findings-emnip 324/
[8] Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. Generating literal and implied subquestions to fact-check complex claims. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 3495-3516, 2022.
[9] Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't t know? In Forty-first International Conference on Machine Learning, 2024.
[10] John J Cherian, Isaac Gibbs, and Emmanuel Candès. Large language model validity via enhanced conformal prediction methods. arXiv preprint arXiv:2406.09714. 2024.
[11] Jesse C Cresswell, Bhargava Kumar, Yi Sui, and Mouloud Belbahri. Conformal prediction sets can cause disparate impact. arXiv preprint arXiv:2410.01880 2024.
[12] André F Cruz, Moritz Hardt, and Celestine Mendler-Dünner. Evaluating language models as risk scores. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[13] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177-190. Springer, 2005.
[14] Yifan Ding, Matthew Facciani, Ellen Joyce, Amrit Poudel, Sanmitra Bhattacharya, Balaji Veeramani, Sal Aguinaga, and Tim Weninger. Citations and trust in Ilm generated responses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23787-23795, 2025.
[15] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
[16] Paul Egré and Hans Rott. The Logic of Conditionals, In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition, 2021
[17] Yizirui Fang and Eric Nalisnick. Learning to defer with an uncertain rejector via conformal prediction. In NewrIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024.
[18] Shangbin Feng. Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.0080 003667 2024.
[19] Jeremy Goodman. Degrees of confidence are not subjective probabilities. In Proceedings of Sinn und Bedeutung, volume 28, pages 329-344, 2024.
[20] Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees. arXiv preprint arXiv:2405. 10301, 2024.
[21] Anisha Gunjal and Greg Durrett. Molecular facts:
Desiderata for decontextualization in LLM fact verification. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3751-3768. Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ URL URL/Intps://aacaanthoooo org/ 2024. windinnns-eenll 215/ 215/
2024. 1924.finding-eemmlp...
[22] Chirag Gupta, Arun K Kuchibhotla, and Aaditya Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022.
[23] Jaakko Hintikka. Modality and quantification. Theoria, 27(3):119-12) 1961.

---

[24] Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie. Yu Zhao, Laura ceee-aeelr,, Max Ryabinin, Xuanli He, Clémentine Fourrier, al. The hallucinations leaderboard-au open effort to measure hallucinations in large language models. arXiv preprint 0.Xiv:2404.05594, 2024.
[25] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11 11170-1118, 2024.
[26] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang. Qiang- long Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
[27] Zhengping Jiang. Jingyu Zhang, Nathaniel Weir, Seth Ebner, Miriam Wanner, Kate Sanders, Daniel Khashabi, Anqi Liu, and Benjamin Van Durme. Core: Robust factual precision with informative sub-claim identification. arXiv preprint arXiv:2407.03577 2024.
[28] Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684-5696, 2020.
[29] Saul A Kripke. A completeness theorem in modal logic1. The journal of symbolic logic, 24(1): 1-14, 1959.
[30] Saul A Kripke. Semantical analysis of modal logic normal modal propositional calculi. Mathematical Logic Quarterly, 9(--6)667-96, 1963.
[31] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
[32] Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David R Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. CoRR, 2023.
[33] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7,452-466, 2019. doi: 10.1162/tacl a 00276. URL https:://aclanthol.ge htwp:/oooogooog/99111909
[34] Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers Ilm-based evaluation. arXiv preprint wxXiv:2410.20774, 2024.
[35] David Lewis. Counterfactuals. John Wiley & Sons, 1973.
[36] David Lewis. Attitudes de dicto and de se. The philosophical review, 88(4):513-543 1979.
[37] Jingshu Li, Yitian Yang, Q Vera Liao, Junti Zhang, and Yi-Chieh Lee. As confidence aligns: Understanding the effect of confidence on human self-confidence human-ai decision making. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1-16. 2025.
[38] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quan- tification for black-box large language models. Transactions on Machine Learning Research, 2023.
[39] David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving faireess and accuracy by learning to defer. Advances in neural information processing systems, 31, 2018.
12

---

[40] Christopher D. Manning, Local textual inference: It's hard to circumscribe but you know when you see it and nlp needs it. 2006. URL https://api. semanticscholar.. org/ CorpuSID:60340094
[41] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, 2020.
[42] Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing Conversational Agents' Overconfidence Through Linguistic Calibration. Transactions of the Association, for Computational Linguistics, 10:857-872, 08 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00494.
URL http::/p:://o1. wwtp://..org//.....///eeee//// wttp1/11///taaaaae110
[43] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision long form text generation. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v1/200 1/2.33//1/2222../ http://e.222mm..2-..m... www.l-mminnn..... main.741. URL https::/aclanthol..gg org/ 2023. em23p-mmnn-miin 741
[44] Christopher Mohri and Tatsunori Hashimoto.
Language models with conformal factuality guarantees. In Forty-first International Conference on Machine Learning, 2024.
[45] Christopher Mohri, Daniel Andor, Eunsol Choi, Michael Collins, Anqi Mao, and Yutao Zhong. Learning to reject with a fixed predictor: Application to decontextualization. In The Twelfth International Conference on Learning Representations, 2024.
[46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[47] Harris Papadopoulos. Inductive conformal prediction: Theory and application neural net- works. In Tools in artificial intelligence. Citeseer, 2008
[48] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473. Hong Kong, China, November https
2019. Association for Computational Linguistics. doi: 10.18653/v 100/3////DD1111//0 "/aclanthology.orrg/D333
[49] Alexandre Piché, Aristides Milios, Dzmitry Bahdanau, and Chris Pal. Llms can learn self- restraint through iterative self-reflection. arXiv preprint arXiv:2405.13023 2024.
[50] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representationss 2024.
[51] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From 9": Your language model is secretly a q- qffunction. arXiv preprint xxXv:2444.12238, 2024.
[52] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[53] Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Tulia Ture, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777-840, 2023.
13

---

[54] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unaligmment: Likelihood displacement in direct preference optimization, arXiv preprint arXiv:2410.08847 2024.
[55] Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng. and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference on Robot Learning, 2023. httpthttt///ppereeeeee net/forrun2dd442X00NNYE
[56] Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference on Robot Learning. 2023,
[57] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995.
[58] Pedro Rodriguez, Shi Feng, Mohit lyyer, He He, and Jordan Boyd-Grabed Quizbowl: The case for incremental question answering. arXiv preprint 000i::/99400044 2019.
[59] Tara Safavi and Danai Koutra. Relational world knowledge representation in contextual language models: A review. arXiv preprint arXiv:2/04.088: 2021.
[60] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027-3035, 2019.
[61] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
[62] Yixiao Song, Yekyung Kim, and Mohit lyyer. VeriScore: Evaluating the factuality of verifiable claims in long-form text generation. In Yaser AI-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9447-9474, Miami, Florida, USA. November 2024. Association for Computational Linguistics, doi: 10. 2024. wiwdinnnnnnnnn.....m 552/
URL httpsts///cllaathoooog org/
[63] Robert Stalnaker. Inquiry. MIT Press, 1984.
[64] Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of iims on grounding documents. arXiv preprint urXiv:2404.10774 2024.
[65] Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. Quantifying uncertainty in natural language explanations of large language models. International Conference on Artificial Intelligence and Stutistics, pages 1072-1080. PMLR, 2024.
[66] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433-5442. 2023.
[67] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
[68] Liaoyagi Wang, Zhengping Jiang, Angi Liu, and Benjamin Van Durme. Always tell me the odds: Fine-grained conditional probability estimation. arXiv preprint arXiv:2505.01 2025.
[69] Peiqi Wang, Barbara D Lam, Yingcheng Liu, Ameneh Asgari-Targhi, Rameswar Panda, William M Wells, Tina Kapur, and Polina Golland. Calibrating expressions of certainty. arXiv preprint arXiv:2410.043/5 2024.
14

---

[70] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
[71] Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. Factcheck-bench: Fine-grained evaluation benchmark for automatic fact-checkers. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Lin- guisties: EMNLP 2024, pages 14199-14230, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics. doi: 10. 112.////////2../////// httt://.n......sseee.... wiw.iiings--eepp.88000 URL https://aclaatthlogg..orr.
findings emaidinnns-eenpppppmm 830/
[72] Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, and Benjamin Van Durme. A closer look at claim decomposition. In Danushka Bollegala and Vered Shwartz, editors, Proceedings of the 13th Joint Conference on Lexical and Computational Semantics ( (*SEM 2024), pages 153-175, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 1.13. URL https://aclanttol.ggr. org/ 2024. starsem-1.13
11118653/1122244taaaae
[73] Miriam Wanner, Benjamin Van Durme, and Mark Dredze. Dndscore: Decontextualization and decomposition for factuality verification long-form text generation. arXiv preprint arXiv:2412.13175. 2024.
[74] Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv e-prints, pages arXiv-2411, 2024.
[75] Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du and Quoc V. Le. Long-form factuality in large language models, 2024. URL
18802
[76] Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. QUDeval: The evaluation of questions under discussion discourse parsing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3344-5363, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/v1/2023 http://////22...mmmmmmmmma. mmn.p-main.n22... 2023. emnlp-man. 325/
URL https://aclanttollggy org/
[77] Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, luliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, et al. Mitigat- ing IIm hallucinations via conformal abstention. arXiv preprint arXiv:2405.0/565 2024.
[78] Jiaqing Yuan, Lin Pan, Chung Wei Hang, Jiang Guo, Jiarong Jiang, Bonan Min, Patrick Ng, and Zhiguo Wang. Towards a holistic evaluation of lims on factual knowledge recall. arXiv preprint 4rXiv:2404.16164, 2024.
[79] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
[80] Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: novel approach to multi-class calibration. Advances in Neural Information Processing Systems, 34:22313-22322 2021.
[81] Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, et al. Wildhalluci- nations: Evaluating long-form factuality in Ilms with real-world entity queries. arXiv preprint arXiv:2407.17468, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that propose novel paradigm for LLMs communicate their confidence that enables explicit factuality-peecffiity trade-off. This paradigm is outline in Section 4 and our experiments in Section provides solid evidence for the effective of our approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our pipeline relies heavily on LLM for data processing, that are always guaranteed to satisfy the requirements. We discuss potential impact and our mitigations near the end Section 4 Also, per reviewer request we further discuss future work and
challenges in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explain algorithm design in details in Section 4 and for experimental settings we discuss Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We submit codebase for reproducing all results, and experiments all based on open-access datasets as described in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed and made sure that our paper complies with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data are fully built public available dataset, endpoints and open-source LLMs. And this work advocates transparency and intend to improve LLM reliability.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com com/daaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the rewriting dataset and the fine-tuned model through hug- gingface upon camera-ready.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

for We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy what should or should not be described.
22

---

A
Prompt Templates
Table I: Prompt for identifying unique answer to given question.
Field
Content
Examples
[User]
Given list of answers to question, identify semantically distinct answers.
**List of Answers**;
$ (Answers)
[Agent] **Semantically Distinct $ (SD Answers)
Input
[User] Given list of answers to a question, identify semantically distinct answers.
** List of *nswers**:
(Answrss)
Table 2: Prompt using iterative clustering fornested set construction.
Field
Content
Instruction
Given a list of already *"selected items your task is to find **K additional items** from the list of * candidates that **most similar** the items already selected. The similarity can based on **specific attributes, characteristics, metrics** relevant the nature of the items, such style, influence, or shared features.
You should return a list of **K items** from the **candidates** list and store them in variable named *#icrements***
Inputs:
1. K: The number of additional items to add to the selected list.
2. selected: A list of already selected items (could be names, objects, etc.).
3. candidates: A list of candidate items from which we need to select the K most similar items.
For Example:
python
K ==
selected = ["Red"]
candidates ["Yellow", "Black", "White"]
Return a list of **K items** from the **candidates** list that most * the items the **selected** list.
Expected Output:
'python
increments ["Yellow"]
23

---

Examples
pyython
selected ['William Butler Yeats"]
candidates ["Agatha Christie", "Benjamin Franklin", "Napoléon Bonaparte"]
[Agent] Based on your inputs, we are tasked with selecting item from the list of candidates that is most similar the selected item, **"William Butler Yeats" "*0, The similarity could based on characteristics like occupation, influence, style.
**illiiam Butler Yeats** figures of 20th-century literature.
Irish poet and one of the foremost
*CCndidattes*
BBnjaamin Franklin American polymath, writer, scientist, diplomat.
Napoleon French military leader and emperor. ** Agatha Christie** British writer known for her detective novels.
In this case, based on occupation (both are writers), ** **Agatha Christie** is most similar to **William Butler *
Thus, the result would be:
python
increments = [ ("Agatha Christie")
Input
python
K $K
selected = $SELECTED
candidates $CANDIDATES
Table 3: Prompt for generating belief that describes belief.
Field
Content
Instruction
Suppose a human subject is going to respond to a question but they don't know the exact answer. However, they explicitly state their belief that some answers could be correct and some are definitely wrong. Given their underlying belief reflected please provide vaguer claim that best summarizes the respondent's belief by focusing on the characteristics that the could-be-comee answers share while the definitely-wrom answers don't. The answer will be used to evaluate the human subject's 's response, so please be faithful to the discussions provided and don't correct them even if they are wrong. Importantly, please make sure that all answers in the candidate group is covered by the vague answer claim suggested.
[User]
Questions and beliefs are provided below. Please respond after reason- ing about the similarities and differences between the positive and negative answers. Wrap your answer a code block ( answer"") The answer should start with "The respondent believes that".

---

Examples
[User] **Question Who's the best football player of all time?
**elieff*** The respondent believes that the answer either Pele, Diego Maradona, Lionel Messi, but not Cristiano Ronaldo, Zinedine Zidane, nor Johan Cruyff.
[Agent] **Positive Group: Pelé, Diego Maradona, Lionel Messi** *Time Period#: Spanning from the 1950s to the present.
*Geographic Origin#: South America (Brazil and Argentina).
*Influence*: Known for their extraordinary dribbling skills, creativity, and playmaking abilities.
"Philosophy*: * These players are celebrated for their flair, technical prowess, and ability to change the course of game hingle-haddedl. Pelé (Brazil) is renowned for his goal-scoring and three World Cup victories. Maradona (Argentina) is famous for his "Hand of God" goal and his incredible dribbling, particularly in the 1986 World Cup. Messi (Argentina) is known for his consistency, vision, and record-breaking achievements with FC Barcelona and Argentina.
*Ditinntt Features Emphasis on individual brilliance, creativity, and a deep connection with their home countries' footballing culture.
*Negative Group: Cristiano Ronaldo, Zinedine Zidane, Johan Cruyff** Time Period*; Spanning from the 1970s to the present. GGograaphic Origin#: Europe (Portugal. France, Netherlands).
IIffluene**: Known for their versatility, tactical intelligence, and leadership the field.
"Philosophy*: These players are celebrated for their physical prowess, strategic thinking, and ability to perform in erucial moments. Ronaldo (Portugal) is known for his athleticism, goal-scoring, and adaptability across leagues. Zidane (France) is celebrated for his elegance, control, and pivotal role France's 's 1998 World Cup win. Cruyff (Netherlands) is pioneer of "Total Football," influencing modern football tactics and philosophy. *Distinct Features*: Emphasis on tactical intelligence, versatility, and significant contributions to both club and national team success.
**e * The positive group is characterized by South American flair, individual brilliance, and a deep cultural impact on football. The negative group is defined by European tactical intelligence, versatility, and strategic approach to the game.
The respondent believes that the best football player of all time comes from South America. ..s
Input [User]
** **Qustion**;; $ (question)
**Belief**; (belief)
Table 4: Prompt for evaluating factuality when imprecision occurs on SimpleQA 775
Field Content
Instruction
[User] Your job is to look at question, gold target, and predicted swer, and then assign a grade of either ["CONSISTENT", "INCONSISTENT"; ANOT_ATTEMPTED", First, will give examples of each grade, and then you will grade a new example.
25

---

Examples
The following are examples of CONSISTENT predicted answers.
What are the names of Barack Ohama's children? Gold target: Malia Ohama and Sasha Obama
Question:
Predicted answer 1: sasha and malia obama
Predicted answer 2: Each of them has a girl's name.
Predicted answer 3: They have the same last name as their father, Obama.
These predicted answers are all CONSISTENT because:
They do not contain any information that contradicts the gold target.
Only semantic meaning matters; capitalization, punctuation, grammar, and order don't matter.
Hedging and guessing are permissible, provided that the response is consistent generalization of the gold answer.
The following are examples of INCONSISTENT predicted answers.
Question:
What are the names of Barack Obama's children?
Gold target: Malia and Sasha
Predicted answer 1: Malia.
Predicted answer 2: Barack Obama does not have any children.
Predicted answer 3: It's possible you may mean Betsy and Olivia. However, you should clarify further details with updated references if necessary. Is that the correct answer?
Predicted answer may be the case that Obama's child is named James. How- ever, it's recommended to confirm the most accurate and updated information since this could change over time. This model may not always reflect the most current information.
These predicted answers are all INCONSISTENT because:
An inconsistent statement in the answer that contradicts the gold target. Inconsistent statements that have some hedging (e.g.. "it is possible that", "although i'm not sure, think") are also considered inconsistent.
Notice that hedging answer inconsistent only if it contradicts the gold target.
The following are examples of NOT_ATTEMPTED predicted answers.
Question: What are the names of Barack Obama's children? Gold target: Malia and Sasha
Predicted answer 1: I don't know.
Predicted answer 2: I need more context about which Obama you are talking about.
These predicted answers are NOT ATTEATTED because: They explicitly state that the model does not know the answer or needs more information to answer the question.
Also note the following things:
Do not punish predicted answers if they omit information that would be clearly inferred from the question.
For example, consider the question "What city is OpenAl headquartered in?" and the gold target "San Francisco, California". The predicted answer "San Francisco" would be considered consistent, even though it does not include "Califomia".
Consider the question "What award did pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity win at NAACL 24?", the gold target "Outstanding Paper Award". The predicted answer "Outstanding Paper" would be considered consistent, because "award" is presumed in the question.
26

---

Examples
For the question "What is the height of Jason Wei in meters?", the gold target is "1.73 m". The predicted answer "1.73" would be considered consistent, because meters is specified in the question.
For the question "What is the name of Barack Ohama's 's wife?", the gold target is "Michelle Obama". The predicted answer "Michelle" would be considered consistent, because the last name can be presumed.
Do not punish for typos in people's name if it's clearly the same name. For example, the gold target is "Hyung Won Chung", you can consider the following predicted answers consistent: "Hyoong Won Choong", "Hyungwon Chung", or "Hyun Won Chung".
A response being too general is not reason to mark it INCONSISTENT.
For example, if the gold target is "Lion", you can consider the following predicted answers as consistent: "Living thing". "Something". For the question "Who wrote the book "The Great Gatsby"?", the gold target is "F. Scott Fitzgerald". The predicted answer "Some 20th century authors", "Somebody wrote it" would be considered consistent, because it is generalization of the gold target.
When the response could be correct under certain interpretation, you should grade it as consistent, favor the model.
If the answer asks for multiple pieces of information, you should only grade whether the answered information correct.
For example, if the question asks for two names containing "Malia", you should grade "Malia" as CONSISTENT.
If the question asks for month and year and the gold target year is 2004, you should grade "2004" as CONSISTENT.
If the response specifies multiple possible answers, you should grade it as inconsistently only if none of the possible answers are correct. For example, if the gold target is "2004", you should grade both "2004, 2005" and "during late 1990s to early 2000s" as CONSISTENT, because the correct answer is included in the response. [User]
Input
Question: {question)
Gold target: $ [target)
Predicted answer: $ s(predicted_answer
Table 5: Claim rewriting template. The verbalization associated the designated level. For the levels qualitatively examined in the paper, Guess corresponds to the verbalization "would guess", S. Conf. corresponds "are somewhat confident", and Conf. corresponds "are confident".
Field
Content
Input
[User] the following claims to be less specific until you $ (verbaliza tion] is is true: $ source_ccaat
Your response should only contain the claim itself, without any addi- tional context.
B Nested Property
Proof of Theorem
4.2 Suppose there's a possible world v E V1, but & V2. Given that
(bz + by) + (L, + bz))
Thus by Definition 3.1
and Definition we have v | b2. However, since v & V/2 we have W V22 again by Definition we have v || b2, which is a contradiction.
27

---

C More Examples
In this section of the appendix, we examine additional examples illustrate how our trained claim rewriter adaptively modifies given claims as shown in T shifting them toward varying levels of uncertainty. We observe that the extent rewriting differs across claims, with some undergoing more substantial transformations than others. Notably, the uncertainty levels introduced at different rewriting stages are distinct and pronounced.
D System Evaluation
We conduct additional evaluation critical components in system make that they robust enough for the main conclusion to hold.
D.1 Number of Answer Samples Needed
We study how many sampled answers are required for our clustering pipeline to recover the diversity of plausible completions. For each question we run the answer sampling and iterative clustering routine described in Section while capping the sampling budget at K E {20, 40, 60, 100]. We then record the number unique clusters that survive the pruning step and average across 500 held-out prompts. The resulting curve, summarized in Table 7 shows diminishing returns: quadrupling the budget from 20 to 80 samples (interpolated between reported points) yields only moderate gains in cluster coverage. While Learn- Learn-Then- Then-Test Test (LTT) remains valid even when K 20, observe that higher K provides more nuanced uncertainty control, but even K=20 maintains sufficient cluster diversity for effective calibration.
D.2 Belief Probing Robustness
To gauge how the belief probing stage behaves in practice, one author manually inspected 50 sampled prompts across different SimpleQA question types. Each inspection checked whether the model-authored belief summary entailed answers in the set it needs to summarize. The review achieved precision 0.81 and recall 0.91, suggesting that the summarization process reliably captures the intended answer sets and that occasional mis-clusterings do not matcrially weaken our risk guarantees.
E Impact Statement
Our work introduces a promising approach to making the trade-off between imprecision and factuality more explicit, which could have meaningful impact across various LLM applications. By proposing a new way to express models' uncertainty, we open potential opportunities for more effective human-LLM interaction. This may contribute increased transparency and accountability critical domains such as law, journalism, and science, where understanding the reasoning behind uncertainty valuable. Additionally, our approach has the potential inspire further advancements in automatic fact-checking systems, encouraging a more comprehensive evaluation of factuality.
F License and Agreement
The SimpleQA dataset 774 is released under the MIT license. We use the version released with the simple-eval repository! [3 The Natural Question dataset (33) is released under the Apache-2.0 license. use the open subset The open-weight model we use includes models from the Llama3 family, which released under the Llama3 license, and models from or fine-tuned from the Qwen2.5 family, which are released under Apache-2.0 license.
28

---

Table 6: More examples occurring when backing off claims from the CORE dataset.
Claim
"Lost in White" is horror-thriller
Guess "Lost in White" is a horror-thriller.
S. Conf. "Lost in White" is type of film that blends elements of horror and thriller genres.
Conf. "Lost White" " film that blends elements of thriller and horror genres.
Carl Rackemann played 10 One-Day Internationals (ODIs).
Guess Carl Rackemann played 10 One-Day Internationals (ODIs).
S. Conf the number of One-Day Internationals played by Carl Rackemann could be any positive integer from 0 to 143.
Conf.~ the number of One-Day Internationals played by Carl Rackemann a positive
integer.
Kadegaon is located in the Kolhapur district.
Guess Kadegaon is located in the Kolhapur district. S. Conf.~ Kadegaon located in the Kolhapur district.
Conf. the village Kadegaon is located in district of Maharashtra, India.
Tim Whitnall was born May 12, 1954.
Guess Tim Whitnall was born on the 12th of a month in 1954.
S. Conf.~ Tim Whitnall was born on day that is any date within wide range of possible dates, or they are unsure of his birthdate.
Conf. there is no available information about Tim Whitnall's birthdate, and they are uncertain or unaware of it.
Yoon Seok-ho's films explore themes of trauma. Guess Yoon Seok-ho's 's films explore themes trauma. S. Conf.~> Yoon Scok-ho's 's films explore themes related to human emotional experience and social issues.
Conf.~> Yoon Scok-ho's films explore themes of human emotional experience and social commentary.
Tadashi Imai died in 1999.
Guess Tadashi Imai died in 1999.
S. Conf.~ Tadashi Imai died sometime between the 1960s and 2000.
Conf.~ the date of Tadashi Imai's ' death is uncertain or not specifically known.
Ernie Dingo gained national recognition for his role as Jacko in the television series "The Sullivans."
Guess Ernie Dingo gained national recognition for his role as Jacko in the television series "The Sullivans."
S. Conf.~ there is a possibility that Ernie Dingo gained national recognition for his role as Jacko in "The Sullivans." " but they are uncertain about the specific year character name. Conf. the actor who gained national recognition for his role as Jacko in "The Sullivans" " is Australian entertainer from the mid to late 20th century.
Zeki Demirkubuz's 's films feature non-linear narratives.
Guess Zeki Demirkubuz's films feature non-linear narratives.
S. Conf.~ Zeki Demirkubuz's films feature complex narrative structures that explore themes of identity and human experience.
Conf. Zeki Demirkubuz's s films feature complex and unconventional narrative structures.
Zia Fatehabadi used the "radif" in his ghazals. Guess Zia Fatchabadi used the "radif" in his ghazals.
S. Conf the use of "radif" in Zia Fatehabadi's is ghazals is traditional and structural element of classical Urdu poetry.
Conf. the use of "radif" in ghazals is a traditional feature of classical Urdu poetry.
29

---

Table 7: Average number of clusters while sampling budget K increases.
K
Avg. # Cluster
20
8.676
40
13.735
60
17.806
100
24.461
30


Reviews:
[{"review_id": "LvKI5WvkNO", "reviewer": "Reviewer_YT5P", "summary": "This paper proposes a novel paradigm for Large Language Models (LLMs) to communicate uncertainty called Conformal Linguistic Calibration (CLC). Instead of abstaining or using verbal hedges like \"possibly,\" CLC enables a model to controllably reduce the specificity of its answer until it can guarantee its factuality with a certain probability. The method is grounded in possible world semantics and uses Conformal Prediction's Learn-Then-Test (LTT) framework to provide formal guarantees. The CLC pipeline involves sampling multiple answers, clustering them semantically, building nested sets of these clusters, and generating progressively vaguer claims that describe these sets. The authors demonstrate that CLC effectively trades informativeness for factuality, outperforming standard abstention baselines on challenging QA datasets. Finally, they show that this complex capability can be successfully distilled into a single, fine-tuned model that performs adaptive, uncertainty-aware claim rewriting.", "questions": "The \"Belief Probing\" step, where an LLM summarizes answer clusters into a vaguer claim, is very creative. How sensitive is the final performance to the specific prompt used for this step? Could a slightly different prompt lead to summaries that inadvertently break the nested property required for calibration, and does the LTT framework fully absorb this risk?\n\nThe computational cost of the pipeline is a notable limitation. The paper shows that the skill can be distilled into an 8B model. How well does this distilled rewriter generalize? For example, having been trained on QA data, can it effectively rewrite claims from out-of-distribution domains like scientific or legal text while maintaining the desired factuality-specificity trade-off?\n\nYour method for constructing nested cluster sets prioritizes semantic similarity over the standard conformal approach of using model confidence scores (e.g., class probabilities). You justify this as helping generate more coherent summaries. Could you elaborate on the theoretical implications of this choice? Does this deviation from standard practice affect the tightness or validity of the LTT conformal guarantees in any way?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "1. Originality and Significance: The core idea is highly innovative and significant. It reframes the problem of LLM uncertainty in a practical and powerful way, moving beyond existing paradigms. The ability to explicitly control the trade-off between factuality and specificity, backed by formal guarantees, is a major step toward building more reliable and trustworthy AI systems.\n\n2. Technical Quality: The work is technically strong, impressively unifying concepts from linguistic pragmatics, possible world semantics, and conformal prediction into a coherent framework. The experimental validation is comprehensive: it clearly demonstrates the factuality-specificity trade-off, shows superior performance over a strong baseline, and validates the approach by successfully distilling the complex pipeline into an efficient, instruction-following model.\n\n3. Clarity and Impact: The paper is well-written, and the authors do an excellent job of explaining a complex, multi-stage process. The figures (especially the pipeline overview in Fig. 2) and results are clear and compelling. The release of the fine-tuned rewriting model and dataset is a valuable contribution to the community.\n\nWeaknesses:\n\n1. Pipeline Complexity and Cost: The main limitation is the computational expense and complexity of the CLC data generation pipeline. Requiring 100 sampled answers per question and multiple processing steps with a large \"processor\" LLM (GPT-40) creates a significant barrier for replication and adoption. While the distillation of the final model mitigates this for inference, the initial cost is very high.\n\n2. Fragility and LLM-Dependency: The pipeline's quality is highly dependent on the capabilities of the LLMs used for intermediate steps like answer clustering and belief summarization. The authors astutely note that the summarization step is imperfect and use the LTT framework to account for this, but this reliance on prompted LLMs remains a point of potential fragility in the overall process.", "was_revised": true, "final_justification": "I keep my overall score at 5 (Accept). The rebuttal addressed my main questions about (i) robustness of the belief-probing step and the impact of occasional nesting violations—LTT does not require perfect nesting and the authors’ small manual check (P≈0.81/R≈0.91) suggests reasonable fidelity; (ii) practicality—showing that the expensive pipeline can be amortized via a distilled 8B rewriter; and (iii) the choice of semantic clustering over confidence scores—consistent with LTT since any nested family suffices for validity. I still see compute cost and multi-stage LLM dependence as limitations, but they are acknowledged and do not outweigh the contribution. No further issues remain.", "rebuttal": "Thank you for your encouraging review and for speaking high of our work's originality and significance. We address your three specific questions (as well as related weaknesses) below.\n\n## Q1: Belief Probing and Nested Property\n \n**Effectiveness of the current probing setup:** We thank the reviewer for raising the concern about potential errors in the pipeline and their implications. To address this, one of the authors conducted manual verification of whether a unique answer is entailed by the summary. The manual verification study shows **precision: 0.81, recall: 0.91** across different question types, indicating robustness of our current belief probing component. \n \n**Nested Property Violation:** As the reviewer correctly noted, relying on the LTT framework fully absorbs this risk—it doesn't require perfect nesting properties. Even if belief probing occasionally breaks nesting (producing irregular trade-offs), LTT's theoretical guarantees remain valid. Also empirically we observe more smooth and effective trade-off in Figure 3, and monotonic relationship between CPMI and multiplicity threshold levels, validating the effectiveness of our implementation.\n\n## Q2: Distilled Model Generalization\n\nWhile providing explicit guarantees for OOD rewriting is challenging, our experiments suggest that the learned trade-off rewriting behavior has some generalizability. While our model is trained on QA data, the evaluation is conducted on Core (FActScore-like biographical claims) that is different from the QA data we trained, yet maintains effective factuality-specificity trade-offs. The distillation learns general hedging principles rather than domain-specific patterns. We’ll make this clearer in the revision. In theory, there are more recent conformal prediction methods that can deal with distribution shift explicitly, by modeling the shift directly (Prinster et al., 2024). So we can potentially leverage those work in the future. We do hope that future work will find better and more efficient rewriting that achieves a better tradeoff with higher correspondence.\n\n## Q3: Semantic Clustering vs. Confidence Scores\n\nUsing semantic similarity instead of confidence scores doesn't affect LTT guarantee validity. In fact any method that produces nested prediction sets is valid for conformal prediction, regardless of how those sets are constructed. As discussed by the reviewer, the major benefit of our approach is that it leads to more coherent nested sets, making belief probing easier and producing more interpretable claims.\n\n## References\n\n- *Prinster, Drew, et al. \"Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them).\" International Conference on Machine Learning. PMLR, 2024.*"}, {"review_id": "eXkcm4rgxc", "reviewer": "Reviewer_1Ph2", "summary": "This paper proposes Conformal Linguistic Calibration ( CLC ) framework to express uncertainty in language model outputs by controlling the level of linguistic generality in the outputs. This is achieved via generating increasingly less specific claims until a certain level of confidence is achieved. The authors reinterpret linguistic uncertainty through the lens of possible-world semantics and connect this idea with conformal prediction to provide probabilistic guarantees on factuality. Their method is instantiated via a Learn-Then-Test (LTT) framework and evaluated on two question answering datasets. The presented empirical results suggest that CLC surpasses traditional abstention-based methods in terms of informativeness and factual accuracy.", "questions": "Suggestions/Questions:\n\n- Definition 2.1 and 2.2: These are difficult to parse due to the dense notation and minimal explanations. Could the authors clarify the key symbols and the relationships involved?\n\n- Definition 3.1 and Theorem 3.2 : Hard to follow. Could you clarify these and provide intuitions and explanations ? What s the semantic meaning of the \"sphere\" in definition 3.1 and how should we interpret this in the context of language models? In theorem 3.2 what is the capital V represent? Is it equivalent to the sphere from Definition 3.2 ? if so this should be stated explicitly along with a explanation of what it means for a claim to describe such a sphere. \n\n- Regarding utilizing an LLM for different components of the pipeline such as belief probing and nested cluster constructions, could you comment on how errors and hallucinations here would affect the pipeline and are there controls or mitigations strategies in the pipeline to correct these ? \n\nOn the experiments: \n- K =100 samples per question introduces a significant computational overhead. In practice, many of these responses are duplicates and likelihood of observing novel responses diminishes rather quickly. How does the framework work under more constrained sampling budgets ? ( K =20 ? ) as commonly done in Uncertainty quantification literature? \n\nThe writing and presentation: The writing particularly sections 2.1 and 3 can be substantially improved. Many of the core theoretical contributions are stated without sufficient explanations. improving the clarify of these sections specially the definitions and formal statements would go a long way in improving clarity. I am open to revisiting my score if these are addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- this paper addresses a very important issue in current LLM research which is how to convey the uncertainty in a manner that is both accurate and useful for end users. the problem of overconfident but incorrect answers is an important topic of research, and the paper takes a meaningful step towards addressing it. \n\n- Connecting and combining possible-world-semantics from philosophy with conformal prediction techniques is indeed novel and interesting and can lead to meaningful avenue for future research as well. \n\n- The presented experimental results seem promising. \n\n\n\nWeaknesses\n\n- The paper is difficult to follow ( specially section 2.1 and 3 ) due to unclear notation, undefined terms, and most importantly a lack of intuition behind many formal definitions ( e.g Definition 3.1, Theorem 3.2 ). Many claim/defs are introduced without sufficient explanations which makes it hard to follow. \n\n- The empirical baselines are insufficiently justified. Many recent works also combine conformal prediction with abstention mechanisms for uncertainty calibration, and the paper should either compare to these methods or provide a clear argument for why a direct comparison is not meaningful. I believe a small related work section is also needed with such clarifications whic would improve clarity and highlight the novelty for the reader. \n\n\n- the CLC pipeline has many moving parts ( belief probing, clustering, rewriting, nesting, calibration ) and a pseudocode or summary diagram mapping each component would be extremely helpful.\n\n- It is unclear how robust the pipeline is to errors that is introduced by the LLM itself, particularly during belief probing and rewriting and how errors due to hallucinations in different parts of the pipeline could accumulate.", "was_revised": true, "final_justification": "After considering the rebuttal, I have increased my score from 3 to 4. The authors provided clarifications that helped resolve some of my earlier concerns, particularly regarding the related work. \n\nHowever, some key concerns remain insufficiently addressed. In particular, I found the discussion around robustness to hallucinations across the multiple LLM components in the pipeline to be inadequate. My concern is that hallucinations originating in one module can propagate or compound in downstream components—potentially leading to information loss, the addition of irrelevant content during belief probing, or even the introduction of new information that was not part of the original LLM’s output distribution (and may affect evaluation as well). The authors did not adequately explore or acknowledge this issue in their response, which limits my confidence in the overall robustness and reliability of the proposed approach.", "rebuttal": "Thank you for recognizing the importance of uncertainty communication in LLMs and our novel integration of possible-world semantics with conformal prediction. We address your concerns about clarity and technical presentation below.\n\n## Q1: Clarified Explanation of Preliminaries\n\n**Definition of Sphere:** In possible world semantics, spheres represent different levels of certainty or likelihood around what we consider \"normal\" or expected. Possible worlds are organized into concentric spheres by accessibility relationships. A claim is **necessarily true** within a sphere if true in all accessible worlds, and **possibly true** if true in some accessible worlds. We’ll add more description in the revision with the additional page.\n\n**Intuitive Explanation:** We represent claims as sets of possible worlds where they remain true. By making claims less specific (hedging), we make them compatible with more possible alternatives, reducing factual error risk while maintaining informativeness. This formulation reveals connections between claim generation and set prediction, enabling conformal prediction for risk control. We will make sure that we add more substantial explanations and illustrations to explain this idea providing additional space.\n\n## Q2: Related Work and Baseline Selection\n\nWe have a related work section in **Appendix A**. Our approach differs fundamentally from standard abstention methods by enabling partial abstention at the claim level. While existing methods refuse to answer entirely, we can respond with partial knowledge when confidence is relatively low (see Figure 3 example).\n\nFor baseline, we compare against the closest approach (Mohri and Hashimoto, 2024) in Section 4.1. While as mentioned by the reviewer there are other works that study LLM abstention and control the risk with conformal techniques, they mostly share a very similar structure to the baseline we compared to on short response QA (Yadkori et al., 2024). And from a previous study the sample-based uncertainty we adopt for our baseline is generally considered the best non-conformity score for this purpose (Cole et al., 2023). Our result clearly shows that standard abstention methods struggle under challenging settings like SimpleQA, where few prompts can be answered correctly with confidence. Also, at near-full abstention, their accuracy becomes unstable due to insufficient remaining examples.\n\n## Q3: Procedure Diagram and Implementation Details\n\n**Figure 2 Clarification:** This serves as our system diagram, with subtitles corresponding to Section 3 paragraphs. We will align subtitle naming with paragraph headers and add clearer indications of the diagram's purpose.\n\n**Belief Probing Robustness:** In general LTT remains valid as long as we have an independent and identically distributed calibration set, even if our components are not always correct. On top of this, one of the authors conducted manual verification on over 50 examples from different prompt categories to see whether a unique answer is entailed by the summary. The manual verification study shows **precision: 0.81, recall: 0.91** across different question types, indicating robustness.\n\n## Q4: Cluster Analysis with Varying K Values\n\nWhile it will be interesting to know how performance scales K, rerunning experiments with different K values requires expensive re-clustering and belief probing. However, we’d like to note that LTT remains valid with K=20. In general, the effect of adding more number of samples is that it provides more fine-grained uncertainty level that the model can fall back to, as they produce more unique clusters:\n\n| K | Avg # Clusters |\n| --- | ------------- |\n|20 | 8.676 |\n|40 | 13.735 |\n|60 | 17.806 |\n|100 | 24.461|\n\nTo summarize: Higher K provides more nuanced uncertainty control, but even K=20 maintains sufficient cluster diversity for effective calibration.\n\n## References\n\n- *Mohri, Christopher, and Tatsunori Hashimoto. \"Language models with conformal factuality guarantees.\" Proceedings of the 41st International Conference on Machine Learning. 2024.*\n- *Yadkori, Yasin Abbasi, et al. \"Mitigating llm hallucinations via conformal abstention.\" arXiv preprint arXiv:2405.01563 (2024).*\n- *Cole, Jeremy, et al. \"Selectively Answering Ambiguous Questions.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.*"}, {"review_id": "5CPOAlHOz7", "reviewer": "Reviewer_xhfo", "summary": "This paper proposes Conformal Linguistic Calibration, a claim-rewriting framework that reinterprets the uncertainty of the model as set prediction; to give a factual answer, the model tailors its specificity accordingly. For instance, if the confidence is high, the model will give a very specific response. If the confidence is rather low, the model will give a rather less specific response. \n\nThe method is applied to the task of Question Answering. First N answers are sampled from the LLM which are then clustered together by trying to find semantically unique clusters. The amount of resulting clusters can indicate the certainty of the LLM. The LLM then creates nested clusters which are after associated with a more general claim. The more general claims are then conformalized through Learn-Then-Test into a linguistically calibrated response. \n\nResults show that reducing the specificity of the answers improves the factuality while still providing conformal guarantees.", "questions": "- There is a spelling mistake on Line 270: abstention instead of abstantion", "limitations": "A  discussion of the limitations and potential negative societal impact is missing / could be made more prominent.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Well written preliminary section to make the building stones clear enough to readers less familiar with possible world semantics and conformal prediction. \n- Paper is also in general very well written and nicely structured. \n- The selection of the datasets and experimental setup (e.g., metrics used, baselines compared to) is well-motivated. It is clear why SimpleQA (and Natural Questions) is chosen in the specific dataset, due to its  difficulty but high confident responses from models. \n- The work targets an important problem in LLMs where models would still give a wrong response even when they are confident on an individual sample level. \n\n**Weaknesses**\nThere are some analyses I thought were missing and would be really useful: \n- An analysis on how the responses generated with Conformal Linguistic Calibration fare in terms of their verbalized confidence (e.g., if the answer is specific then the model is verbalized higher in confidence) would really strengthen the paper. \n- How the performance varies across different amounts of N answers sampled from the LLM would be a very interesting insight. \n- A user study would be very useful to showcase the utility of such a technique. Do users prefer being less specific to be more factually correct?", "was_revised": true, "final_justification": "The reason for my positive evaluation of the paper is due to its fresh perspective on uncertainty quantification through the lens of set prediction. This is supported by a well-motivated and suitable experimental setup. \n\nI must note that I did have to re-read the sections where all the terminology and notations were introduced but in the end I thought it was well-written. \n\nThough my question regarding different N samples was addressed, the rest of the points were left (justifiable) for future work. Given that I already had a positive score in combination with some points raised by other reviewers (e.g., the potential propagation of LLM errors throughout the pipeline and computational cost) I decided to keep my score the same.", "rebuttal": "Thank you for your encouraging review and recognition of our paper's clarity and structure. We appreciate your thoughtful questions about implementation details.\n\n## Q1: CLC Generation and Verbalized Confidence\n\nThank you for providing this interesting perspective. Through manual examination, we observe that our generated responses typically avoid explicit confidence statements. This is intended, as our formulation aims to represent uncertainty through hedging rather than verbalized confidence reports. In our implementation, our few-shot prompts in belief probing maintain structural similarity to original claims while incorporating appropriate hedging. We agree with the reviewer that it will be really interesting to see how LLM can simultaneously utilize different uncertainty communication protocols to improve user experience. **We will open-source all data and code to facilitate such investigations.**\n\n## Q2: Performance Scaling with K\n\nRerunning experiments with different K values requires expensive re-clustering and belief probing. However, we’d like to note that LTT remains valid with K=20. The effect of adding more number of samples is that it provides more fine-grained uncertainty level that the model can fall back to, as they produce more unique clusters:\n\n| K | Avg # Clusters |\n| --- | ------------- |\n|20 | 8.676 |\n|40 | 13.735 |\n|60 | 17.806 |\n|100 | 24.461|\n\nTo summarize: Higher K provides more nuanced uncertainty control, but even K=20 maintains sufficient cluster diversity for effective calibration.\n\n## Q3: Limitation Discussion and User Preferences\n\nYou raise an important point about user perception. Our current limitation discussion focuses primarily on pipeline errors and guarantee validity. **We will expand this to include discussion of how AI confidence communication impacts human decision-making, acknowledging the challenges in conducting human-based factuality trade-off evaluations due to known biases (Li et al., 2025; Ding et al., 2025).** \n\n## References\n\n- *Li, Jingshu, et al. \"As Confidence Aligns: Understanding the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making.\" Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 2025.*\n- *Ding, Yifan, et al. \"Citations and trust in llm generated responses.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 22. 2025.*"}, {"review_id": "2ZQkIGDO6L", "reviewer": "Reviewer_ayvU", "summary": "This paper introduces Conformal Linguistic Calibration (CLC), a framework that turns the intuitive act of “hedging” with linguistic qualifiers into a formal, distribution-free procedure with factual correctness $\\geq 1 - \\alpha$ while retaining as much specificity as possible.\nThe pipeline (i) samples multiple candidate answers from a base LLM, (ii) groups them into semantic clusters via a single additional LLM call, (iii) orders the clusters to form a nested sequence of answer sets, (iv) asks the LLM to rewrite each set into a single “belief summary” sentence, and (v) uses Learn-Then-Test conformal calibration to choose the most specific rewrite whose empirical error is below $\\epsilon$ at level $\\alpha$. At deployment time the system outputs that rewrite, trading specificity for coverage. Experiments on NQ and SimpleQA show that CLC provides the promised error guarantees and retains substantially more information than an abstention-based baseline.", "questions": "- Authors mention that they use an LLM to generate a list of unique answer cluster names from {a} (instead using more standard entailment models to establish equivalency relationships). Is the performance of LLM clustering analyzed or discussed? How robust is the system to the cluster generation process?\n- Please further clarify how semantic equivalency is determined in “...answers that are semantically equivalent to the corresponding cluster name...”?\n- “We observe that embedding-based similarity metrics often fail to accurately capture spatial, temporal, or numerical distances.“ --- Were embedding-based similarity metrics tried? Please provide comparison results if so.", "limitations": "Limitations are partially discussed in Section 3.1; but limitations of the pipeline brittleness or compute (API calls) budget should be discussed more in-depth.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Clearly motivates the need for partial rather than binary abstention and formalises it with conformal guarantees.\n- Careful empirical evaluation on two QA datasets; conforms to $\\alpha$-risk.\n- Fine-grained CPMI analysis; and experiments show promising results.\n\n**Weaknesses**\n- Heavy reliance on heuristic LLM clustering; no quantitative study of clustering accuracy or ablations on sphere-construction choices.\n- Baseline comparisons limited to a simple abstention baseline; perhaps could be compared to a baseline embedding-similarity or entailment-based set prediction (at least parts of the CLC pipeline could be compared to baselines in isolation).\n- Ablations or robustness of each stage of the pipeline is not really assessed.\n- Computational cost is not fully benchmarked, but it appears to be much higher than related uncertainty quantification methods.\n- Other potential weaknesses below in the \"questions\" section.\n\n---\n- It would be a great addition if you could wrap your contributions in an easy to use python package that could be plugged into open-source models or a web API.", "was_revised": true, "final_justification": "My specific questions have been adequately addressed by the authors' rebuttal, but the initially outlined limitations remain. I think this is promising work, but would still need to see additional experiments to properly contextualize these results; therefore I maintain the borderline accept suggestion.", "rebuttal": "Thank you for finding our partial abstention motivation compelling and appreciating our conformal guarantee formalization. We address your concerns about component robustness and computational costs below.\n\n## Q1: Clustering Component Evaluation\n\nWhile comprehensive evaluation is challenging without ground truth labels, we optimized each pipeline component through careful design choices:\n\n**Identifying Clusters:** We initially tested entailment-based approaches but found NLI models overly sensitive to surface forms and noisy (McCoy et al., 2019). Embedding-based approaches work poorly for numeric cases (Naik et al., 2019; Tang et al., 2025). For example, in one of our test cases “2023” is more similar to “2026” than “2024” in embedding space. Since we can’t predict if an LLM output will be numeric, number-specific methods are also hard to apply.\n\n**Estimating Answer Multiplicity (How semantic equivalence is determined):** We use embedding-based similarity (answer will be assigned to the most similar cluster, which we deem to be semantically equivalent) for answer multiplicity estimation (Section 3.1). Here using embedding-based similarity to identify semantic equivalence is more reliable as unique clusters have been identified and usually there will be one cluster that is very similar to the answer at hand. In this case, we only need to find the most similar cluster, not discover new ones. Previous research suggests that despite noisy clustering, uncertainty estimation from traditional clustering approaches remains reliable (Kuhn et al., 2023; Grewal et al., 2024). While clustering accuracy is hard to evaluate directly without ground truth label, it is indirectly evaluated together with our belief probing verification study described below.\n\n**Belief Probing:** One of the authors conducted manual verification on over 50 examples from different prompt categories to see whether a unique answer is entailed by the summary. The manual verification study shows **precision: 0.81, recall: 0.91** across different question types, indicating robustness.\n\nWe also would like to note that we use LTT for calibration, which is based on hypothesis testing and only requires an independent and identically distributed calibration set. Even if no pipeline components are perfect, the LTT guarantee remains valid, although the performance of each component does affect the optimality of the trade-off curve.\n\n## Q2: Computational Cost\n\nWe acknowledge the computational overhead of curating high-quality answer sets. To address this, our experiments demonstrate that hedging behavior can be distilled into base models for direct generation. Other relatively expensive approaches that require sampling also resort to similar ideas (Kuhn et al., 2023; Farquhar et al., 2024; Band et al., 2024).\n\n## Q3: Open Source and Technical Issues\n\nWe will fix all formatting issues, including the last line of Figure 2, and will **open source our complete dataset and step-wise creation code with the camera-ready version.**\n\n## References\n\n- *McCoy, R. Thomas, Ellie Pavlick, and Tal Linzen. \"Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.\" arXiv preprint arXiv:1902.01007 (2019).*\n- *Naik, Aakanksha, et al. \"Exploring numeracy in word embeddings.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.*\n- *Tang, Eric, Bangding Yang, and Xingyou Song. \"Understanding LLM Embeddings for Regression.\" Transactions on Machine Learning Research.*\n- *Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. \"Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. \"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.\" The Eleventh International Conference on Learning Representations.*\n- *Grewal, Yashvir S., Edwin V. Bonilla, and Thang D. Bui. \"Improving uncertainty quantification in large language models via semantic embeddings.\" arXiv preprint arXiv:2410.22685 (2024).*\n- *Farquhar, Sebastian, et al. \"Detecting hallucinations in large language models using semantic entropy.\" Nature 630.8017 (2024): 625-630.*\n- *Band, Neil, et al. \"Linguistic Calibration of Long-Form Generations.\" International Conference on Machine Learning. PMLR, 2024.*"}]

Author rebuttal:

Meta-review: As reviewers have mentioned, the problem studied in this paper is well-motivated, the ideas are interesting, empirical studies are convincing, and the paper is well-written. Given these positive elements, I recommend accept.
Decision: Accept (poster)