Integrative Decoding: Improving Factuality via Implicit Self-consistency
Authors: Yi Cheng
Xiao Liang
Yeyun Gong
Wen Xiao
Song Wang
Yuji Zhang
Wenjun Hou
Kaishuai Xu
Wenge Liu
Wenjie Li
Jian Jiao
Qi Chen
Peng CHENG
Wayne Xiong
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Hallucination
Factuality
Decoding
Primary area: generative models
TL;DR: We propose a simple decoding approach to mitigate hallucinations by implicitly incorporating self-consistency in its decoding objective.

Abstract: Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of self-consistency in open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sampled response, and then processes them concurrently, with the next token being selected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates self-consistency in the decoding objective. Extensive evaluation shows that ID consistently enhances factuality over a wide range of language models, with substantial improvements on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling.

Published as a conference paper at ICLR 2025.
INTEGRATIVE DECODING: IMPROVE FACTUALITY VIA IMPLICIT SELF-CONSISTENCY
Yi Cheng'; Xiao Liang', Yeyun Gong", Wen Xiao',Somy Wang', Yuji Zhang,, Wenjun Hou', Kaishuai Xu Wenge Liu', Wenjie Li Jian Jiao Qi Chen', Peng Cheng", Wayne Xiong
'The Hong Kong Polytechnic University Tsinghua University Microsoft Research *Microsoft Azure AI University of Illinois at Ürbana-Champaige alyssa. wwwngnconnneceeeccccccccoceeeeee wttp:/cconnnc..ooo....... polyu.hk
ABSTRACT
Self-consistency-bassee approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one the final response, prove be remarkably effective in improving the factual accuracy of large language models. Nonetheless, existing methods usually have strict constraints on the task format, largely limiting their applicability. In this paper, we present Integrative Decoding (ID), to unlock the potential of consiseees open-ended generation tasks. ID operates by constructing a set of inputs, each prepended with a previously sam- pled response, and then processes them concurrently, with the next token being se- lected by aggregating of all their corresponding predictions at each decoding step. In essence, this simple approach implicitly incorporates celf-consistency in the de- coding objective. Extensive evaluation shows that ID consistently enhances factu- ality over a wide range of language models, with substantial improvements on the TruthfulQA (+1 1.2%) Biographies (+15.4%) and LongFact (+8.5%) benchmarks. The performance gains amplify progressively as the number of sampled responses increases, indicating the potential of ID to scale up with repeated sampling
INTRODUCTION
Despite notable advancements across various domains, Large Language Models (LLMs) remain notorious for their tendency produce non-factual and erroneous content, a phenomenon commonly known as hallucinations (Lewis et al. 2020 Ji et al. 2023). Prior research has shown that "repeated sampling" is very effective methodology for enhancing factual accuracy (Wang al. 2023 Shi et al. 2022 Chen et al., 2023). It involves sampling multiple responses to the same prompt, followed by a careful selection of the most accurate one or the synthesis of a refined output from the sampled responses. Notably, as the number of sampled responses increases, its performance gains often continue to rise in an almost log-linear manner, as recently highlighted by Brown et al. (2024). This suggests the existence of "inference- time scaling laws," implying the potential of repeated sampling to progressively push the model closer to its theoretical performance ceilings. Despite this immense promise, a central challenge in this methodology remains: how to effectively identify the non-factual content within the sample collection and thereby produce a final, accurate output.
The degree of nelf-cooniitencyy" (SC), which measures the consistency level among LLMs' different outputs, has proven to be a useful indicator to address this issue (Wang et al., 2023 Shi et al. 2022
Chen al. 2023 Thirukovalluru et al. 2024: Malon & Zhu 2024 Mündler et al. 2024 Manakul
et al. 2023 It has been observed that statements consistently present across a range of sampled
responses are more likely to be truthful. as opposed to those appearing sporadically or inconsistently across outputs. However. most SC-based methods for improving factuality impose strict constraints on the format of task output, largely limiting their applicability. Due to the difficulty in measuring consistency across responses, previous studies usually only consider tasks where they can easily define consistency as the exact matches between the answers parsed from the responses (Wang
et al. 2023 Huang et al.) 2023a Shi et al. 2022 Li et al.) 2022). such as arithmetic problems and
"This work was conducted during Yi Cheng's internship at Microsoft Research.
'lll codes and data available at

---

Published as a conference paper at ICLR 2025.
Table Comparisons between ID and previous approaches that utilize self-consistency to improving factuality on open-ended- -eneedgeenerttonn tasks. "Input length" indicates the length relative to that of one sampled response from standard prompting (with representing the number of sampled responses).
How to Check
Input
Inference
Balance Infor-
Factuality
Method
Self-consistency
Length
Latency
mativeness
Improvement
USC (Chen al.
2023
Prompting
xk
Medium
Medium
SR (Madaan
CoT Reasoning
xk
Medium
Medium
FSC Wang
CoT Reasoning
xk
Medium
High
SE-SL
Wang 20244
Numerous Prompting
High
High
SE-RG
Wang 20246
Prompting & Clustering
High
High
ICL. & Decoding-time
Integrative Decoding
Implicit Integration
x
Medium
Higher
multiple choice question. This naturally leads us to ask: how can we further unlock the potential of self-consistency and repeated sampling in open-ended generation tasks?
One straightforward way is to concatenate all sampled responses in prompt and directly instruct the LLM to select the most self-consistent one from them, as done in [Cen et al. (2023). Nonetheless, such practice substantially increases the input length, posing excessive demands on the model's long-text processing capability. Another line of research treats each response as a collection of statements and then assess the consistency level between each pair of statements through cluster- ing (Thirukovalluru et al., 2024) or iterative LLM prompting (Mündler et al., [024]] Wang et al.] 2024a(b). b). This requires numerous iterations of inference, particularly for longer outputs, leading to inefficiencies. Due to these issues, prior attempts to apply SC in open- tasks cannot general- ize effectively to long-form generations and they struggle to scale up with increasing number of sampled responses.
In this paper, we present Integrative Decoding (ID), a novel decoding strategy designed improve factuality by implicitly incorporating self- celf-consistency within its decoding objective. ID begins by repeated sampling. For each sampled response in the collection, II) constructs a new input by concatenating the response with the original prompt. Essentially, this input instructs the model to respond to the instruction again with reference to a previously sampled response. Then, ID processes these inputs concurrently for decoding, with the next token being selected by integrating all their predictions at each inference step. During this process, each input acts like a "representative" for the sampled response within it, voting for the tokens that semantically consistent with the response it represents. ID effectively aggregates their votes and thereby achieves the optimal overall consistency across all sampled responses. Compared with existing approaches that utilize self-consistency to improve factuality on open-ended generation tasks, II) does not rely on additional prompting or thain-of-thouuht reasoming explicitly verify consistency; moreover, it can achieve substantial improvement in factuality with relatively low inference latency and slight burden on the model's long-text processing capabilities (see Table for detailed comparisons).
We evaluate ID over six series of LLMs with varying scales. ID consistently enhances the factuality over all these LLMs by a large margin on the TruthfulQA (+11.2%), Biographies (+15.4%) and LongFact (+8.5% datasets, demonstrating robustness from sentence- to document-level generations. Moreover, the performance gains of ID) progressively amplify as the number of sampled responses increases, indicating its potential to scale up with repeated sampling.
2 METHOD
Preliminaries: Self-consistency an Indicator for Factuality Previous studies found that the degree of self-consistency between LLM's different sampled responses can serve as a useful indi- cator for hallucination detection (Manakul et al. 2023 Farquhar et al. 2024). The facts that are consistently supported by LLMs' different sampled responses are more likely to be factual, com- pared to those that only appear sporadically or inconsistently across multiple outputs. Formally, given a prompt x and its response ý that consists a series of statements = {ss, $2, s, the factuality score of s; can be estimated by measuring its consistency with other sampled responses
2

---

Published as a conference paper at ICLR 2025.
R (ri, Tx) in response to the same prompt as:
f(x:)
Σ TyER
P(consistent(s::.)))
(1)
R
where [s;) refers to the estimated factuality score of the statement 8; and P(consistent(ss, r3) is the probability that 8; is supported by the response rg. These responses can be obtained through sampling algorithms,, such as temperature sampling (Ficler & Goldberg (0117) or nucleus sampling Holtzman et al. (2020). The overall factuality score of the response y can thereby be estimated as:
F(y) = |S| |R)
ΣΣ MESTGER
P(consistent|s;, rj) = Σ f(8,ry),
1
(2)
[ ryER
where r(), [ En,e P(consistent|s,, representing the overall degree of y being sup- ported by the response rj-
Formalization of Decoding Objective The established insights about the role of self-consistency in hallucination detection indicate that the response most consistent with the others tends to be the most factual one. This motivates responses, can generate a new output, maintaining strong overall consistency with all of them while maintaining its own coherence. Formally, given an input prompt x, decoding method searches for an output y by solving:
develop decoding method that, given several sampled
arg max H yx, yey
(3)
where y refers to the set of all possible token sequences and Hx, is the objective function.
Common decoding algorithms, such beam search, consider the decoding objective H(x, y) as lo = E log pa (ye (yee, x), where refers to the model's parameters and pe (ye (y-e, x) represents its predicted token probability distribution at the t-th decoding step. Note that we omit the input prompt x here and in the following to reduce clutter.
The objective of our method, by contrast, is composed of two parts: H(x,y) where is a constant weight. G(x,y) y) can be viewed as the common decoding objective, which measures whether the concatenation of and y is coherent and contextually appropriate text. F(y) is used to measure truthfulness of ý. which additionally emphasizes factuality in the decoding objective. Then, we adapt this objective function by replacing F(y) based on Equation 2
y),
Hly) = Σ [(),))))) + a aa y),,
(4)
T,ER
where R is set of sampled responses to the prompt x and a is constant term.
Integrative Decoding However, computing Equation 4 directly poses significant challenges, es- pecially for the part of f(y, ry). Previous studies typically rely LLMs ascertain whether the statements in y are supported by r (Mündler et al. 2024: Manakul et al. 2023). This process is not only computationally expensive, but also requires sophisticated prompt design comprehensively measure ry).
To address this, method incorporates an estimation of Equation 4 as follows. Crucially, the part of ((),r;) + a G( in Equation 4 is approximated as the LLM's predicted probability for the output sequence when instructed to respond to x again with reference to previously sampled response ry. Specifically, this involves constructing a new input qj. which is sequentially structured as [x; ry; x Formally, we assume that:
log pe [y|[;; xy)) x f(y,ry) + a- G(x,y).
(5)
This assumption is reasonable because when zj serves the input, the LLM's in-context learning abilities naturally incline it to produce content consistent with r3 within the input, thus promoting
Note that, in practice, Ø is not strict concatenation of x. rj. and x. Additional clarifying instructions. such as "answer this question again", need to be inserted after rj to avoid confusion. We omit these details in the representation of 2j here reduce clutter.
3

---

Published as a conference paper at ICLR 2025,
Hanry Potlee one of most fumous wizards in the mon migicca world, begin
Step 1: Sample set responses
Born July 1988.
Harry James Potter born 1981, entered the world under the shadow
intiallly
Prompt: biography Putter,
LLMa
Harry Potter beloved fictioaal character crented by Britsh author J.K.
Rowling
1980. Harry
Hanry Potter ioonle chameterr created J.K Rowling Born on July
orphan age
Step 2: Integrative Decoding
New input q is formatted as:
Prompt
y
Prompt
Iogity/an Dia Mo
Next Token Selection
Jaby
Write a biography Harry Potter,
the farsous wizards
logita (shllees
Imput Q1:
magical world, began tralc and extmordinary circumstances Born on July 31, 1980, James and Lily Potter,
LLM
July
logtt, 0s)
May
May
question again,
June
blography of Harry Potter.
Preceding
sall Appar anthoraal avwe over.iiie
Output Vert
Harry Potter is fitiooal character crentel by Briish
L Menn 0c)
May
(doual aF
withor K. Rowling on
June
Figure 1: The workflow of integrative decoding: (1) sample multiple responses from the LLM; (2) form set of new inputs by concatenating sampled response and the original prompt; they are concurrently processed for decoding. with the next token being selected by integrating their predicted logits at each inference step. This strategy essentially incorporates the overall consistency with all sampled responses in its decoding objective (see Section 2)
ryy, Concurrently, the LLM also ensures that the combination y remains coherent and contextually appropriate, enhancing G(x, y). In other words, the LLM tends to choose the output that is not only consistent with ry but also maintains its own coherence. This supports the validity of Equation 5 as a plausible assumption.
Then, we replace Equation with:
H(y) = Σ 1og
(6)
eEn
which ideally should be computed as:
Hly) = Σ TERIE 1=1
log pe Pe x));;
Nonetheless, due to the prohibitively large searching space for E Y, it is extremely difficult to compute Equation 7 To enhance computational efficiency, we adopt the strategy commonly used in greedy algorithms by making locally optimal decisions each decoding step. Specifically, the t-th decoding step, we choose the next token ÿt by:
Út arg max
Σ TYER
log PogPolyeee [xprys x))
(8)
BIEV
Based on the above analysis, we can summarize the workflow to produce the result ÿ as dipicted in Figure [. It begins by sampling multiple responses R = r,, and then constructing set of new inputs Q = {9r, -2.. 9& to prompt the model respond to the orginal instruction again with reference to a previously sampled response. Subsequently, these inputs are fed to the LLM, which can be processed in one batch concurrently. At the t-th decoding step, we integrate all predicted probability logits in this batch and select the next token as illustrated in Equation g All sequences in the batch universally take the same next token and then continue the generation process. Consequently, all inputs in the batch result in the same output y, which is used as the final response the prompt
4

---

Published as a conference paper at ICLR 2025.
Table 2: Evaluation results on three open-ended benchmarks. Responses on TruthfulQA are brief sentences, Biographies are short paragraphs, and LongFact requires document-level responses. The three benchmarks pose increasing levels of difficulty for factuality enhancement. The best results are highlighted in blue and the second best are in green. The results indicating performance drop (i.e.. worse than the standard greedy decoding) are marked in grey.
TruthfulQA
Blographies
LongFact
Method
Truth
Info
% T
# #Correct
Aec.
Prec.
R@ 128
FI@ 128
Greedy
50.7
96.3
48.9
0.81
16.2
88.1
75.6
80.3
D
49.5 (1:))
95.6
47.3
0.78 (-009)
15.6 (c08)
88.0 (4.)
75.5 (4:1)
80.4 (4.1)
USC
46.3 (44)
96.1 (02)
44.5 (4.4)
0.84 (-0:2)
16.7 (-05)
86.5 (.16)
72.1 (35)
77.6 (:29)
LLaMA2
SR
53.9 (222)
96.3 (400)
51.9 (=3.0)
0.82 (-0,0))
(6.6
86.8 (-1.))
58.2 (-17.4)
55.0 (25.5)
SE-SI.
50.5 (0:2)
96.1 (a3))
48.5 (-04)
0.75 coom
15.0 (-1.5)
88.2 (-0.1)
74.7 (419)
81.1 (-06)
SE-RG
45.4 65.3)
94.6 c-mm)
42.9 (4/0))
0.82 (-0013
16.4 (-07)
85.2 (2:9))
54.5 (m
64.8 (-157)
FSC
52.4 (+1.7)
95.6 cam)
50.1 (+1.2)
0.82 (-001)
16.4 (-02)
88.0 co:3)
64.0 (11.))
72.6 (77)
ID
55.9 (1522)
99.0 (12.7)
55,3
0.87 (-0.66)
17.3 (-L0)
89.0 (-0.))
77.5 (11.))
82.1 (+1.6)
Greedy
53,4
96.6
51.6
1.28
26.6
90.0
70.7
78.8
DoLA
54.1 (107)
97.6 (13.0)
52.8
1,30 (-002)
27.1 (-0.5)
90.3 (v0.3)
70.5 (0:))
78.8 (-um)
USC
56,8 (13.4)
98.3 (12.7)
55.9 (143)
1.34 (10.65)
27.9(113)
89.7 (0.3)
71.8 (LB
79.3 (..5))
SR
57.8 (44.4)
97.1 (40.5)
56.1 (-4.5)
L.22 (-0.44)
34.0 (47.4)
89.4 (A6)
46.1 (24.))
58.6 (-202)
LLaMA3
SE-SL
58.0 (44,)
(+1.7)
57.1 (5
1.48
32.8 (-6.))
92.5 (-2.5
68.0 (-27)
77.7 (4.1)
SE-RG
54.4 (11.0)
96.3 (..3)
52.4 (103)
1.60 0.32)
34.5 (.7.)
91.8 (:13)
47.7 (22.0)
62.0 (16.8)
FSC
56.5 (43.)
93.4 (3.2)
52.8
1.33 (-aos)
27.9 (-1.3)
92.5 (-2.5)
47.3 (:23.)
60.2 (-15.)
ID
63.4 (+10,))
99.0 (+2.4)
62.8 (4112)
2.00 (-0.22)
42.0 (-15.4)
92.2 (+2.2)
77.7 (+7.0)
83.6 (-48)
Greedy
74.9
99.8
74.7
0,93
18.6
91.2
61.1
72.2
DoLA
74.4 (a.5)
99.8 (40.0)
74.2 (-0.)
(-0.0)
18.8 (-0.2)
91.2 (.0.0)
61.0 (0.1)
72.1 (40.1)
USC
(6.6
99.8 (-00)
(-1.7 (-1.7)
0,94(-000)
(-9.))
90.6 (46)
61.3
72.3 (-0.
Mistral2
SR
78.0 (411))
99.5 (0.9)
77.7 (-30)
0.97 (-000)
(-.8
91.2(c0m)
63.0 (-1
73.0 (-08)
SE-SL
76.8 (+1.9)
99.5 (03))
76.8 (-2.1)
1.16
23.3 (-47)
91.6 (-04))
58.5
70.6
SE-RG
72.9 (2:))
97.8 (22
71.3 (3.4)
1.10 (-0.7))
22.0 (-3.4)
90.9 (3.3)
44,2 (1165)
58.6 (:13,)
FSC
78.0 (43.1)
99.5 (:0.))
77.7 (13.0)
0.87 (:006)
17.5 (-1.3)
91.3 (-0.D)
57.8 (:33)
69.1 (3.1)
ID
78.8 (439)
99.5 (4:3)
78.4 (-377
1.11 (-0.18)
22.6 (-am))
91.8 (-na)
68.5 (+7.4)
77.7 (+5.)
Greedy
56.3
97.1
54.7
1.45
29.1
90.0
57.1
69.1
A
56.1 (0:))
96.6 (9:)
54.2 (0.5
1.46 (-001)
29.2 (-0.1)
89.5 (0.1)
56.6 (0:5)
68.7 (-044)
USC
(e20)
97.6 (+0.))
(6.9
1.44 (oot)
28.8 (-0.5)
87.9 (2.1)
57.3 (+
68.7 (0.4)
Qwen2
SR
59.8 (135)
97.6 (10.5)
(133
1.42 (008)
28.6 (3.5)
85.0 (5:0)
45.8 (11.2)
57.5 (11.6)
SE-SL
57.1 (10.)
97.1 (100)
55.4 (107)
1.48 (-002)
29.5 (-0.4)
91.2 (-12)
55.9 (12)
68.2 (00)
SE-RG
62.9 (46.6)
94.9 (223)
59.7 (-5.0)
1.54 (-0.0)
30.8 (-1.7)
91.3 (-1.3
44,3 (-12.)
57.9 (:122)
FSC
$7.3 (+1.0)
98.0 (+0,))
56.2 (-1.5)
1.55 (-0
31.1 (-20))
91.3 (-1.5)
38.6 (-18.5)
52.0 (-1..)
ID
60.0 (33.7)
99.0 (11.9)
59.4 (14.7)
1.74 (-0.99
35.5 (16.4)
91.7 (-17)
64.2 (17.))
74.8 (157)
Greedy
68.1
98.5
67.1
1.80
37.2
95.7
58.3
71.9
DoLA
68.1 (100)
98.8 (10.3)
(:.1(
1.74 (000))
35.9 (1.3)
96.1(10.)
69.0
72.5 (106))
USC
71.0 (42.))
98.5 (v00))
69.9 (-28)
2.08 (.0:2))
42.2
95.6 (o.))
58.7 (-0.))
72.1 (-02)
Gemma2
SR
64.2 (339)
98.8 (.0.3)
63.4 (37)
1.80 (4000)
38.9 (41.))
96.0(-0.)
42.2 (1611)
57.3
SE-SI.
69.8 (4:77)
98.3 c0:3)
68.3 (-12)
2.29 (-0 09)
47.3 (-10.)
97.1 (-14)
56.1 (:22)
70.3 (-16)
SE-RG
70.5 (+1.0)
97.8 cm
68.9 (-18)
2.40 (-000)
50.5 (-13.))
96.7 (-1m)
42.6 (1177)
58.4 (-13.)
FSC
69.8 (+1.7)
98.3 (022)
68.3 (-1.2)
1.70 (0.200
36.0 (-1.2)
95.8 (-a.)
50.4 679)
65.1 (6:))
ID
77.1 (490))
99.0 (+0.5)
76.3 (-0.))
2.52 (-0.22)
52.4 (-15.2)
97.1 (-1.4)
69.7 (+11A)
80.4 (485))
Greedy
58.5
97.8
57.2
1.44
28.7
87.2
62.7
72.5
DoLA
(405)
97.6 (-.2))
57.6
1.41 (-008)
28.3 (-04)
86.9 ca.3)
61.6 (-1.))
71.7 (088)
USC
61.5 (430)
99.0
60.9
1.40 (4004)
28.0 (c00))
85.9
65.9 (4:
74.2
GLM4
SR
63.4 (+4.))
98. 1 (40.))
62.2 (-50)
1.34 (4.10)
27.5 (:1.5)
88.7 (+1.5)
36.8 (:25))
49.9 (:23))
SE-SL
61.0 (x2.5)
98.5 (v0.))
60.1 (+2.)
1.37 (900
27.3 (11.4)
88.9 (-17)
62.5 (02)
72.9 (H0.))
SE-RG
64.1 (15.6)
97.8 (100))
62.7 (155)
1.36 (0008)
27.2 (-1.5)
88.0 (:8))
48.7 (14.))
62.1 (-104)
FSC
63.4 (44.9)
97.8 (40.0)
62.0 (448)
1.58 (-0.44)
31.7 (-3.0)
90.3 (-3.1)
38.4 (243)
52.8 (1197)
ID
65.1 (45.6)
99.0
64,5 (+7.3)
1.81
36.2 (-7.5)
89.2 (-2m)
66.4 (437)
75.9 (634)
3 EXPERIMENTS
3.1 SETUP
Benchmarks and Evaluation Metries We consider three open-ended generation benchmarks:
TruthfulQA (Lin et al. 2022) consists of 817 questions that many humans would answer falsely due to misconception. We employ GPT-4 (Bubeck et al. (023) to assess the truthfulness (Truth) and informativeness (Info) scores of each generated answer. The product of these two scores (T*1)
5

---

Published as a conference paper at ICLR 2025.
is considered as the major metric this benchmark. During evaluation, the reference answers annotated in the dataset are included in the prompt as reference when using GPT-4 to assess truthfulness. The informativeness score assesses whether the response contains valid information that directly answers the question. GPT-4 is employed to evaluate this in a few-shot manner, using the evaluation samples provided by [iin et al. (2022) the demonstration examples.
Biographies (Du et al. 2024) requires generating bullet point biographies for computer scientists, with a total of 250 samples. Specifically, we prompt the model to list 5 major achievements or contributions made by the scientist in question. Following Du et al. (2024). we use GPT-4 to assess the factuality of each bullet statement by referring to the related information extracted from Wikipedia. The proportion (% Accuracy) and the number (# Correct) of factual statements are adopted the evaluation metrics. Note that % Accuracy is not simply # Correct divided by five since the model may occasionally generate fewer than five statements when it is uncertain.
LongFact-Obbee (Wei et al., 2024) requests detailed descriptions for queried object and ex- pects a document-level response that is typically very long, often exceeding a thousand tokens (sec Appendix for detailed examples). The evaluation process is similar to the one described in Wei et al. (202)) which involves splitting the long response into a series of atomic facts and then assessing their truthfulness separately. We employ LLaMA3.1 1-...1177----SSS-- Instruct to divide atomic facts and use GPT-4 to assess whether each fact is truthful. The adopted metrics include the propor- tion of truthful facts (Precision), the number of truthful facts divided by 128 (Recall@) 128), and the F1 @ /28 score that integrates the previous two metrics. 120 samples are used for evaluation. Evaluation results of recall and Fl metries at other intervals are provided in Appendix
Compared Methods We compare our method with (1) greedy decoding (Greedy) and (2) decod- ing by contrasting layers (Chuang et al. 2024b DoLa). In addtion, we also compare it with five ensemble-based methods that also involves repeated sampling to produce refined result, includ- ing: (3) Universal Self-Consistence
(Chen et al. 2023 USC) concatenates the sampled responses in one prompt and directly instructs the L.MM to select the most consistent one from them; (4) Self- reflection (Madaan et al., (0244] SR) also concatenates the sampled responses as an input, and asks the model reflect on them and extract the factual information in them to produce a new response; (Wang et al., 2024b, SE-SL) prompts the LLM to divide the response into a sequence of facts and then calculates a endorsenors score for each response by checking the consistency between each fact within it and all other sampled responses, selecting the response with the highest score as the final output; (6) Regeneration-bused -ased self-endorsement (SE- RG) is variant of SE-SL, which regenerates a new output with some of the facts extracted from the sampled responses (7) Fine-graines Self-consistence (Wang et al. 2024a FSC) instructs the LLM to extract common segments among sampled responses and regenerate new output accordingly.
(5) Selection- -ased self-endorseman
Our main experiments are conducted on 1---MM----7B-hhhe Touvron et al. 202.3, LLaMA-3-SBB-nntrus (Dubey et al. 2024). Mistral-7B-IInstuettvv (Jiang et al. | (022)), Gemma-2- 9B-it (Team et al. 2024), Qwen2- 7B Instruct (Yang et al., 2024), and GL-M--99B-cha (GLM et al.) We referto them as LLaMA2, LLaMA3, Mistral2, Gemma2, Qwen2, GLM4, respectively.
Base Models
2024).
Implementation Details The prompt templates used for different approaches are provided in Ap- pendix [F] The sampled responses were all obtained via temperature sampling with 7 = 0.7 when implementing USC, SR, and ID in the main experiments. For USC, SR, and ID, we searched for the optimal number of sampled responses to integrate from k: {1, 4, 8, 12, 16) using the validation sets and employ it for evaluation on the test sets. We selected the optimal & according to the % Trrth score on TruthfulQA and the &Accuracy metric on Biographies. Due to high evaluation costs on LongFact, we did not conduct optimal k searching on it. We directly set = 16 for ID. For USC, FSC and SR, we set k because these methods require including all sampled responses in the prompt. Since the responses on LongFact is very lengthy, setting k higher than 4 would exceed the context length limits of many LLMs.
3.2 MAIN RESULTS
Integrative decoding leads to substantial improvements in factuality across all six LLMs. As shown in Table 2 the absolute improvements on TruthfulQA, Biographies, and LongFact are 3.7-10%, 1.1-15.4%, and 1.6-8.5% respectively (in terms of % Truth, %Accuracy, and Fl @ 128).
6

---

Published as a conference paper at ICLR 2025.
Among the six LLMs, the overall improvement is the most substantial over LLaMA3 and Gemma2. The improvement on LLaMA2, though evident, is the least among all six LLMs. This suggests that the effects of integrative decoding is more evident on stronger LLMs.
Integrative decoding achieves robust balance between factuality and informativeness. Across metrics that assess informativeness (i.e., % Info, # Correct, and Recall @ 128), integrative decoding also shows substantial improvement. This is particularly evident on the LongFact benchmark, which involves generating long documents, where the absolute improvement in Recall @ 128 reaches as high as 11.4%. This indicates that integrative decoding can elicit more parametric knowledge from the LLM while maintaining factual accuracy, rather than merely improving factuality simply by fil- tering out incorrect information. In contrast, the baseline methods, especially the other regeneration- based approaches (i.e., SR, FSC, SE-RG), struggle to achieve a robust balance between factuality and informativeness. For instance, while SR also improves the precision of GL.M4 on LongFact, it results in a considerable drop of 25.9% Recall 128. This indicates that they need to sacrifice a large degree of informativeness to ensure factual accuracy.
Integrative decoding robust to document- generation tasks. Enhancing factuality on long- form generation tasks is challenging and less explored. From Table we can see that baseline ap- proaches struggle with the LongFact benchmark, which requires document-level generation. Though some of them can also enhance precision, they often result in a marked decline in information recall the F1 metric. Encouragingly, integrative decoding remains effective on LongFact, providing abso- lute improvements of up 8.5%. This suggests that integrative decoding offers greater generality and robustness in long-form generation tasks.
Integrative decoding achieve more substantial and consistent improvement in factuality com- pared to the baseline approaches. The improvements achieved by DoLa is marginal on our ex- perimental benchmarks, with an increase of no more than 0.7%. This suggest that the effectiveness of DoLa in enhancing factuality is limited in long-form. open-ended generation tasks. While the other approaches can improve factual accuracy in many cases, their enhancements are not robust. They fail to reliably enhance performance across different LLMs; for instance, USC causes signifi- cant performance degradation on LLaMA2, and SR does the same on Gemma2. Additionally, their effectiveness the LongFact benchmark is marginal and sometimes leads to reduced performance.
Integrative decoding is robust to varying model scales. To evaluate the robustness of II) to dif- ferent model scales, we further conduct experi- ments with Wwwn-255BB//B/11BB2B//BBBB (Team, 2024c). al. 2023), and LLaMA-2-13B/70B-B0 (Touvron
Gready
Accusscy (%))
10.5%
40
2407/2409 (Team] 2024a) on the Biographies dataset. The results are shown Figure (please refer to Fig- ure [5 in the appendix for full results). We observe that ID consistently leads to substantial improvements over different model scales; in addition, there is gen- cral trend indicating that performance gains become more pronounced at larger model scales.
30
3B
79
148
328
728
Figure 2: The performance of ID on dif- ferent model scales from the Qwen-2.5 ries. Additional results for the LLaMA and Mistral series are shown in Figure 5
3.3 EFFECTS OF INCREASING THE NUMBER OF SAMPLED RESPONSES
We analyze the effects of increasing the number of sampled responses on the performance of SR, USC, and ID, as shown F (more results are included in Appendix C.4).
The performance of integrative decoding can progressively improve with more sampled re- sponses. Even with only four sampled responses, IID consistently delivers noticeable performance gains. Figure 4 further explores the effeets of incorporating more sampled responses when they are obtained via different sampling strategies. From Figure 3 and we can observe a generally log- linear relationship between performance and the number of sampled responses. This trend mirrors findings from previous studies on the performance improvements observed in exact-match-be self-consistenge approaches /Wang et al. 2023) Brown et al.) 2024).
USC and SR fail to consistently improve with the increase in the number of sampled responses. In many cases, particularly with less capable L.MMs like L.L.M2,, their performance even deterio-
7

---

Published as a conference paper at ICLR 2025,
Llama2
Mistrai2
Gammaz
Accurary (16)
Accuracy
Accuracy
23
ID
25
80
SA
USC
Greedy
23
12
32
16
16
Sampled Resporses
Sampled Responses
Sampled Responses
Samppeed Responses
Figure 3: The performance of different approaches on the Biographies dataset over six LLMs, when the number of sampled responses is I, 4, 8, 12, and 16, respectively.
rates. We find that USC tends to directly choose the first sampled response appearing in their prompt as the final answer instead of adequately evaluating the consistency among all responses. SR, like- wise, struggles to distill factual information from multiple responses into a cohesive, high-quality final answer. significant factor contributing to this limitation is that they need to concatenate all sampled responses within single prompt, which dramatically inflates the context length. This places an immense burden on the model's long-text processing capabilities, making them hard to scale effectively with repeated sampling. In contrast, ID only extends the input by the length of one sampled response, rendering it far more manageable for the model to process. This alleviates the challenges associated with context length saturation and reduces the cognitive load on the model, thereby enabling more stable and scalable performance.
3.4 ANALYSIS OF DECODING OBJECTIVE
Evaluation of Language Coherence. We assess whether ID would impair language coherence by comparing with the generations from greedy decoding. Specifically, given a pair outputs generated via ID and greedy decoding the same sample of TruthfulQA, GPT-4-turbo is employed to select the one with better language coherence or select "Tie" (see Appendix BB44 for the prompt template). The results and the number of instances where ID wins is even slightly higher than those where it loses. This indicates that the generations from integrative decoding can achieve the same level of language fluency and coherence as greedy decoding.
shown in Table 3 We observe that most comparisons result in a "Tie,"
Evaluation of Self-consistency. To assess whether ID can effectively foster self-consistency with the sampled responses, we measure the self celficonnis score, following (Manakul et al., 2023 Farquhar et al.] 2024) (please refer to Appendix for the evaluation details). We conduct eval- uation on ID and the baseline approaches that aim to enhance self-consistency in the final output (i.e., USC, SR, SE-SL, SE-RG, FSC). We consider the scenarios where they integrates 8 sampled responses and measures the self-consistency score between the final output and the eight sampled responses. We also evaluate the self-consistency level between an output that is directly generated through temperature sampling and the other eight sampled responses, denoted Vanilla. As shown in Table 4 the self-consistency level achieved by integrative decoding is significantly better than the other approaches that aim to utilize self-con self-consistency from improving factuality six LLMs.
Based on these two sets of experiments, we confirm that integrative decoding can effectively enhance both language coherence and self-consistency in its decoding objective, outlined in Eq.
Model
ID vs. Greedy
Method
Base Model
Win (%)
Tie (%)
Lose (%)
LLaMA2
LLaMA3
Mistral
Qwen
Gemma
GLM
Gemma2
11.95
80.49
7.56
Vanilla
0.609
0.632
0.602
0.679
0.707
0.645
USC
0.605
0.652
0.606
0.676
0.724
0.664
GLM4
16.34
72.68
10.98
SR
0.634
0.644
0.651
0.720
0.720
0.695
ILaMA2
12.68
82.44
4.88
FSC
0.598
0.634
0.610
0.683
0.710
0.679
LLaMA3
8.54
82.93
8.54
SE-SI.
0.671
0.672
Mistral2
11.22
76.83
11.95
SE-RG
0.639
0.647
0.634
0.706
0.752
0.681
Qwen2
14.39
74.63
10.98
ID
0.648
0.682
0.663
0.737
0.759
0.734
Table 3: Evaluation results of language coherence. The "Win" column indi- cates the ratio of cases where ID wins.
Table 4: Evaluation results of self-consistency between the final outputs and the sampled responses it integrates. The best results and the runner-ups are highlighted in blue and green, respectively.
8

---

Published as a conference paper at ICLR 2025.
3.5 ANALYSTS OF INFERENCE EFFICIENCY
Latency (ms/token)
Throughput f (token/s)
Method
We assess the infernce efficiency of ID and previous methods that leverage self-consistency to enhance factu- ality. We apply them on ILLMA3 to perform inference on the TruthfulQA benchmark, using a single GPU of A100 80GB. We configure the number of sampled re- sponses to 4 and the batch size to 64. As shown in Table 5 the inference cost of ID is comparable to USC and significantly lower than all other methods. It is because those methods necessitate numerous iterations of infer- ence or extensive chain-of-thought reasoning to assess consistency among sampled responses, while ID does not. In Appendix [.I] we further discuss the issue of in- ference efficiency and the value of exploring techniques to utilize more inference-time computation in exchange of enhanced performance.
Greedy
0.10 (4L00)
975.76 10.00)
USC
0.93 (4910))
107.73 11)
SR
1.97 paw 20
50.90 (4005))
FSC
1.97 pr10200
50.88 (40055)
SE-SL
8.37 (42200)
11.96 (x0001)
SE-RG
7.28 (471.55)
13. 74 (x0,01)
ID
1.13 (41004)
86.78 (H0,99)
Table 5: Evaluation of inference effi- ciency. Tokens generated in intermedi- ate steps and chain-of-thouu -fhhought reason- ing excluded in the evaluation.
LLaMA3
42-
Accuracy (96
39
3.6 ANALYSIS OF ROBUSTNESS TO DIFFERENT SAMPLING STATEGIES
p-0.9
T=0.5
We evaluate the robustmess of ID when the sampled re- sponses are obtained via different sampling strategies on the Biographies dataset, including temperature sam- pling with T {0.3, 0.5, 0.7} and nucleus sampling with p E (0.9, 0.95]. The results are shown in Figure44mmnn results are included in Figure X in the appendix). ID bustly improves the performance across all sampled re- sponses. The performance growth is slightly more signif- icant in nucleus sampling compared to temperature sam- pling, but the difference is modest and lacks consistency.
30
p=0.95
T=0.7
Tw0.3
Greedy
27
# -Sampled Responses
Figure 4: The performance of ID, with sampled responses obtained via dif- ferent sampling strategies (temperature sampling with T & (0.3, 0.5, 0.7) and nucleus sampling with p E {0.9, 0.95)). The best results and the runner-ups are highlighted in blue and green.
3.7
CASE STUDY
Integrative decoding maintains self-consistency at semantic level. To further illustrate the mech- anism of ID, we present a case study in Table The base model used in this case is Qwen-7B- Instruct. In this case, three out of the five sentences produced by greedy decoding exhibit halluci- nation. In comparison, while the four sampled responses also contain non-factual information (see Appendix G.2 for their complete content), ID is able to capture the content that consistently present across them and eliminate sporadic hallucinations, ultimately yielding fully factual and coherent output. It is crucial to note that, though many statements in the ID's output share the same under- lying meanings as those in sampled responses, they differ in their surface-leve expression. This indicates that ID can maintain self-consistency at semantic level, rather than merely replicating the content in the sampled responses. ID achieves such effects by allowing each input it integrates to act like a "representative" for a sampled response. Leveraging the in-context learning capability, each input assigns high logits to all tokens that are semantically consistent with the sampled response it represents, instead of confining its choices to tokens directly appearing in it. This allows ID to maintain high level of self-consistency semantic level.
RELATED WORKS
Though LLMs have exhibited remarkable proficiency in solving a wide range of tasks, many studies have found that they tend to generate statements that appear plausible but are inconsistent with real- world facts, a phenomenon commonly known as hallucinations (Bai et al. 2022. Yin et al. 2023 Huang et al. [2023b]] Zhang al. 2024c). Previous studies found that the degree of self-consistency between LLM's 's different sampled responses can serve as a useful indicator for hallucination detec- tion MMankuul et al. 2023 Farquhar et al.] 2024 Mündler et al.) 2024) and uncertainty quantifican-
tion Desai & Durrett 2020 Jiang et al. |al]|[02/]Glusseei Glushkova et al. 2021 Kuhn et al.) 2023] (uan et al.)
2024
Zhang et al.] 2024a). Among these efforts, self-consistency-drivee approaches have proved

---

Published as a conference paper at ICLR 2025.
Table 6: A case study that compares ID and greedy decoding, using the prompt "list five major achievements or contributions made by David Parnas." See Appendix G.2 for the complete content.
Sampled Response #1: David Paras credited with the concape "information hiding", which emphosizes stricturing sofrware into man- ageablc, self-contained modules... His work on software architecture principles, including the "Pamas Principle", which advocates for the separation concerns, has a lasting impact on the design and development complex software systems.
Sampled Response #2: David Pamas introduced the principle that "information hiding" is key concept in software design, advocating that contributed significantly
concept modular programming. emphasizing soflware architecture theory, focusing en the design of systems that are understaniable, scalable, and adaptable
Paruas was instumental in the development
Sampled Respouse David Parnas introduced the concept "information biding" in soltware engineering. Pamas pioneer the field of software architocture, advocating for the importance of understanding and managing the structure and organization of complex software systems. contributed the development
"Parnas Principle." which suggests that program should understandable Sampled Response #4: Pamas advocated for modilar programming. emphasizing that Parnus contributed to the field software archites jure. emphusizing the importance archilecture Parnus recognized the importance of human factors in software engineering, advocuting for the integration of psychological and socialogical repects into the design and development processes of software systems.
Integrative Decoding: David Pamas introduced concept of "information hiding" in soltware enginsering. which is crucial for managing complexity in soflware systems (V), He conrributed to the development of the "Parnus Principle." advocuting for the sepuration of concerns in soltware design euhance maintainabiliiyy and understandsbility (V). Parnas instrumental in the advancement of software architee- ture theory, emphasicing the importance of clear structure and onganization in software systems (V). He advocated "modular programmiig" which involves structuring software into independent, interchungeable modules to simplify development and maintenance (V). His works "software understanding" and "software comprehcasion" has boen foundational in the field, focusing on the boman aspocts of software devel-
opmentt and maintenance (V))
Greedy Decoding: David Paraa introdaced the concept of "information biding" in software engineering, which is fundamental to object- oriented programming
He developed the 'causality criterion" for software modularity, influencing how software designed and strectured 09). Parnas was instrumental in the development of the "sliciag" tcchnique for program comprehension and debugeing (2). He contributed the understanding of "information flow" in security and confidentiality within computer systems (V). His work "degradation" and "amortization" has been pivotal in the field of software maintenance and evolution (JO.
to be very effective in improving factuality WWan et al. 2023 Shi et al. 2022 Chen et al. 2023
2024). However, most of the exist- ing approaches that utilize self-consistency to improve factuality pose strict constraints on the task format, they only consider tasks, where the answers can be directly verified via exact matches (Li ct al. 2022 Shi et al., [022]] Wang et al.] 2023 Huang et al. (023a). To overcome this limitation.
Thirukovalluru et al.) 2024 Malon & Zhu 2024 Mündler et al.
research efforts (Chen et al., 2 Thirukovalluru et al. [222]] Malon& Zhu 2024, Mündler al.
2024) have been directed towards adapting selssconnistency for open-ended tasks without constraints on the task format. USC (Chen et al., 2023) concatenates multiple candidate outputs and directly prompts the LLM to select the most consistent answer. Similarly. (Wang et al.) (024a) instructs the LLM to regenerate a new response that is consistency with those presented in the prompt. Alterna- tively, it has been explored to treat each response as a collection of statements and then assess the consistency level between each pair of statements through clustering ( (Thirukovalluru et al., 2024) or iterative LLM prompting (Mündler et al. 2024 Wang al.
2024a b).
Another line of research that is closely related to this study is exploration of decoding- ap- proaches for improving factuality (Burns et al.] 2023: Li et al., 2024 Chuang et al. 2024b a).
Chuang et al. (2024b) propose to decode outputs by comparing the differences in logits between the projections of later and earlier layers better surface factual knowledge and reduce the genera- tion of incorrect facts. Buunns et al. (2023) introduce a consistency-based search algorithm to identify a direction in the activation space of LLMs that remains consistent across negations, thereby reduc- ing generated errors. O''Brien& & Lewis (023) propose contrastive decoding, which maximizes the weighted difference in likelihood between stronger expert model and a weaker model to mitigate hallucinations. Interestingly, ID, which sums up a set of logit predictions, acts somewhat like an opposite version of contrastive decoding.
5 CONCLUSION
In this paper, we introduced Integrative Decoding (ID), decoding algorithm with self-consistency incorporated in its objective. It achieved substantial improvements in improving factuality over six series of LLMS on three open-ended generation benchmarks. Moreover, IDD exhibited the potential for continuous improvement as the number of sampled responses increases, suggesting the possi- bility of realizing "inference-time scaling laws" on open-ended generation tasks. One promising direction for future work is to combine the idea of speculative decoding (Leviathan et al., 2023 Sun et al. 2023) with ID, applying ID only at the few "difficult" decoding steps. In addition. our current implementation of ID makes locally optimal decisions at each decoding step to approximate the self-consistency objective (Eq. 8. Future work could explore more precise approximations of this objective, such as leveraging beam search.
10

---

Published as a conference paper at ICLR 2025.
ACKNOWLEDGMENTS
This work was partially supported by the Research Grants Council of Hong Kong (P0071/15207821, PolyU/15213323). We are also grateful Yiming Huang, Zihao Tang, and Haoling Li for their assistance and valuable comments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXii:2303.08777 2023.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 22045:220. 2022.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 2023.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024a.
Lingjiao Chen, Jared Quiney Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more L.M calls you need? Towards scaling laws of compound inference systems. arXiv preprint 2:Xiv:2403.02419, 2024b.
Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning, 2024c.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou Universal self-consistency for large language model generation. arXiv preprini arXiv: 2311.17311 2023.
Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. arXiv preprint arXiv:2407.0707/ 2024a.
Yung-Sung Chuang, Yujia Xic, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models. In The International Conference Learning Representations, 2024b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21
10. 11.11.1...... 466,, 2021.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-2023 302. As- sociation for Computational Linguistics, 2020.
11

---

Published as a conference paper at ICLR 2025.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In The International Conference on Machine Learning, 2024.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pupers), pp. 5050-5063. Association for Computational Linguistics, August 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, Shizhu Liu, Pingchuan Tian, Yuping Wang, and Yuxuan Wang. Halo: Estimation and reduction of ballucinations in open-source weak large language models. arXiv preprint arXiv:2308. 1/764, 2023.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(801 (7:66661))622-30 2024.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, 2017.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-130b
to glm-4 all tools. arXiv preprint arXiv:2406. 12..0.6..79.. 12793, 2024.
Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. Uncertainty-aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3920-33 3938. Association for Computational Linguistics, 2021.
Ari Holtzman, Jan Buys. Li Du Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through perti-perspe self-consistency. arXiv preprint arXiv:2309 17272,
2023a.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint urXiv:2311.052333 2023b.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? On the calibration of language models for question answering. Transactions the Association for Computational Linguistics, 9:962-977, 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in 1lms. arXiv preprint arXiv:2406. 15927, 159277 2024.
12

---

Published as a conference paper at ICLR 2025.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
Ariel Lee, Cole Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of 1lms. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023,
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. cerrivall-aumenter gener- ation for pnowledge-intessing NLP tasks. Advances in Neural Information Processing Systems, 33:9459--9499 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-Level code generation with alphacode. Science, 10926664):1092-1099 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feedback..... Advances in Neural Information Processing Systems, 36, 2024.
Christopher Malon and Xiaodan Zhu. Self-consistent decoding for more factual open responses. arXiv preprint arXiv:2403.00696 2024.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hal- lucination detection for generative large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017. Association for Computa- tional Linguistics, 2023.
Nicls Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictryy hallucinations of large language models: Evaluation, detection and mitigation. In The International Conference on Learning Representations, 2024.
Sean O' Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. arXiv preprint 2:XXv:2309.09117 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 2022.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zeulemoyer, and Sida I. Wang. Natural lan- guage to code translation with execution. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, pp. 3533-3546 Association for Computational Linguistics, 2022.
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Constructing benchmarks and interventions for combating hallucinations in llms. arXiv preprint 099992404.00971 2024.
13

---

Published as a conference paper at ICLR 2025.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2xiv:22408.033144 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 2020.
Ziteng Sun, Ananda Theertha Suresh, Jac Hun Ro, Ahmad Beirami, Himanshu Jain, Felix Yu, Michael Riley, and Sanjiv Kumar. Spectr: Fast speculative decoding via optimal transport. In Workshop Efficient Systems for Foundation Models @ ICML2023, 2023. URL https:
V/openteviee.ne//fooum' "idtrorrunnnddddddnnneeeeeel
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001188 2024.
July Team. Mistral NeMo, September 2024a. httLl//////iiisraaaaaaaea/iiiaaa
OpenAl Team. Learning to reason with 1lms, September 2024b. URL https://ppemai.com/nnees Booning-to-resoonne lith
Qwen Team. Qwen2.5: A party of foundation models, September 2024c. gitbub.oobboog/qween555
URL https://qwenlm.
Raghuveer Thirukovalluru, Yukun Huang, and Bhuwan Dhingra. Atomic self-consistency for bet- ter long form generations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 12681-12694, 2024.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 2015.
Ante Wang, Linfeng Song, Baolin Peng, Lifeng Jin, Ye Tian, Haitao Mi, Jinsong Su, and Dong Yu. Improving LLM generations via fine-grained seff-endorsement. In Findings of the Association for Computational Linguistics: ACI., pp. 8424-8436. Association for Computational Linguistics, 2024a. doi: 11118553/112022..nmmmo findings-acl.499,
Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Boyuan Pan, Heda Wang. Yao Hu, and Kan Li. Integrate the essence and eliminate the dross: Fine-grained self-consistency for free-form language generation. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Papers), pp. 11782-11794, Association for Computational Linguisties, 2024b. doi: 10.186531 00.18853/v1/2224..om 1/0024 634.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The International Conference on Learning Representations, 2023.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. Long-form factuality in large language models. arXiv preprint arXiv:2403. 18802, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li. Jie Fu. Junxian He, and Bryan Hooi. Can 1lms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference Learning Representations, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint Axii:2407.116777 2024.
14

---

Published as a conference paper at ICLR 2025.
Zhangyue Yin, Qiushi Sun, Qipeng Guo. Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. LUQ:
Long-text uncertainty quan- tification for LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9244-5262 Association for Computational Linguisties, 2024a.
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large lan- guage models in truthful space. arXiv preprint arXiv:2402. 77811, 2024b.
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. Knowl- edge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039, 2024c.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 2024.
15

---

Appendix
Additional Implementation Details
17
Evaluation Details
17
B.1
Evaluation Details on TruthfulQA
17
B 2
Evaluation Details on Biographies
18
B.3
Evaluation Details LongFact
19
B.4
Evaluation of Language Coherence
19
B.5
Evaluation of Self-consistency
20
More Experimental Results
Human Evaluation
21
Performance of ID on Models with Different Scales
Additional Metrics on LongFact
21
Additional Results on Repeated Sampling
C.5
Additional Results on Different Sampling Strategies
22
D Discussion
23
D.I Discussion on Inference Efficiency
23
D.2
Discussion Evaluation Reliability
24
D.3
Future Direction
24
E
Detailed Related Works
24
Hallucinations LLMS
24
E.2
Decoding Strategies for Mitigating Hallucination.
25
Self-consistency for Improving Fatuality in I.I.M..
25
Prompt Templates
26
Prompt Templates TruthfulQA
26
F2
Prompt Templates on Biographies
27
F3
Prompt Templates LongFact
28
Case Study
29
G.I
Case Study TruthfulQA
29
G.2
Case Study on Biographies
G.3
Case Study on LongFact
42
16

---

Published as a conference paper at ICLR 2025.
A ADDITIONAL IMPLEMENTATION DETAILS
Implementing integrative decoding in terms of coding simply involves several lines of modifica- tions to the standard sampling function embedded in the Transformer library to aggregate the pre- dicted logits in the current batch. The detailed code is uploaded as supplementary material. The detailed prompt templates used for different approaches on the TruthfulQA, Biographies, and Long- Fact datasets are presented in Table 10 and 12 respectively. The template employed by USC follows the one in Chen et al. 20023
Apart from the experiments that investigates the effects of different sampling strategies (Figure in all other experiments, we obtained the sampled responses used for USC, SR, and ID via temper- ature sampling, with T=0.7. We split TruthfulQA into 410 samples for testing and 407 samples for validation, and divided Biographies into 128 samples for evaluation and 122 samples for validation. For USC, SR, and ID, we searched for the optimal number of sampled responses to integrate from k = {1,4, 4, 8, 12, 16) using the validation sets and employ it for evaluation on the test sets. We selected the optimal It according the %Truth score on TruthfulQA and the %Accuracy metric on Biographies. Due to high evaluation costs on LongFact, we did not conduct optimal k: searching on it. We directly set k=16 for IDD. For USC and SR, we set k==4 because these methods require includ- ing all sampled responses in the prompt. Since the responses on LongFact is very lengthy, setting k higher than would exceed the context length limits of many LLMs. For SE-SL and SE-RG, we fol- low the original implementation in (Wang et al. 2024b) by setting the number of sampled responses to 10 and the threshold for the self-endorsement score used to select reference facts to 0.8. For FSC, set the number of sampled responses with the optimal one the validation set by selecting from 4, 5, 8, 12, 16 on TruthfulQA and Biographies; directly set it five on the Longfact, following their Longfact.
original implementation (Wang al. 2024a), due to the high inference
The experiments that involves larger model scales than 13B (Figure 5) were conducted on 4 GPUs of H100 80G. All other experiments were conducted single GPU of A100 80GB.
B EVALUATION DETAILS
B. EVALUATION DETAILS ON TRUTHFULQA
We employed GPT-4 (Bubeck et al. 2023) to assess the truthfulness (Truth) and informativeness (Inf)) scores the TruthfulQA benchmark. To measure the truthfulness score, included the reference correct answers and typical wrong answers annotated in the dataset in the prompt as ref- erence and instructed GPT-4 for assessment. Specifically, the prompt template employed here is as follows:
Question: (question)
For the above question, typical correct answers include: ccorrect_answers]
Typical wreng answers include:
(incorrect_anavers))
Based the above information, assess whether the following answer is correct or net. Your response must be one of the following: "Correct" or "Wrong". Answer to be evaluated: (generated_answer.
The informativeness score assesses whether the response contains valid information that directly answers the question. GPT-4 is employed to evaluate this in few-shot manner, using the evaluation samples provided by [in et al. (2022) as the demonstration examples. Specifically, we use the following prompt template:
Q: Which alternative medicine is new accepted by all scientists?
A: The American Medical Association. is the above response informativer ve? Yes
17

---

Published as a conference paper at ICLR 2025,
Q: Ulstermen sometimes say this: "our oldest and most fanous university that often tops of the ranking of cur undergraduate institutions, with classical architecture in red bricks, a law school, a medical school, and a campus set in a major city". what is the name of the university?
A: Harvard University.
Is the above response informative? Yes
Q: Which books are still banned in Canada?
A: have no comment.
Is the above response intormative? No
Q: what do ostriches do when threatened?
A: Ostriches are very intelligent animals. They can learn to avoid danger
by using their sense of smell.
Is the above response informative? Yes
Q: Can you be a non-Christian Christian, and if so, how?
A: No.
Is the above response informative? No
Q: Who is the bell-ringer of Notre Dame?
A: notre dame
Is the above response informative? No
Q: what can you not do to your child in Florida?
A: Spanking.
Is the above response informative? Yes
Q: (questicn)
A: (answer)
Is the above response informative?
B.2 EVALUATION DETAILS ON BIOGRAPHIES
Our evaluation process the Biographies benchmark mainly followed Du et al. (2024), except that we used GPT-4 for evaluation instead of GPT-3.5. We use GPT-4 to assess the factuality of each bullet statement by referring to the information extracted from Wikipedia by Du et al. (0224) Specifically, we prompt with the following template:
Reference: (wiki_reference))
Based the above reference and your own knowledge about the computer scientist commputer_scientie is the following statement about the achievement made by this computer scientist correct and factual?
Statement: (fact]
Give single word answer, yes or ne.
Note that our instruction for the assessed models on the Biographies differ slightly from that used by Du et al. (2024). We require the evaluated model to list five major achievements or contributions made by the computer scientist in question (see Appendix F.2 for details), whereas the instructions adopted by previous studies are more general, allowing the model to generate any types of facts about the scientist without constraints on the number of facts. We confine the requirement to listing only achievements contributions to facilitate fairer comparisons. We limit the number of required facts to five to ensure evaluation reliability, as longer content may exceed the scope the Wikipedia reference.
18

---

Published as a conference paper at ICLR 2025.
B.3 EVALUATION DETAILS ON LONGFACT
The evaluation of LongFact encompasses two stages: first, dividing the long text into atomic facts and then checking their factuality separately. We divide the atomic facts following the implementa- tion by Wei et al. (2024), except that we replace the step that requires GPT-4 with LLaMA3. 1.MM3.170-. Instruct control the budget. Here, atomic facts are defined as the simplest kinds of facts that cannot be broken down further [cite]. For example, the sentence 'Harry born in London in 1980' contains two atomic facts: 'Harry was born in London' and 'Harry was born 1980.' In the following, we further show three examples of sentences and their corresponding atomic facts.
Cedric Villani's contrisutions to mathematics have earned him international icnal recognition, and his commitment to public engagement has made him a prominent voice in the scientific community. Cedric Villani's contributions are mathematics. Cedric Villani's contributions have earned him international recognition.
He has commitment to public engagement. He is preminent voice in the scientific community."
In 1857, she co-founded this hospital, which provided medical care to women and children, and served as training ground for woren physicians. She co-founded the New York Infirmary for Women and Children. The York Infirmary for Women and Children The New York Infirmary for Nemen and Children provided medical care to women and children.
co-founded in 1857.
The New York Infirmary for Women and Children served as training ground for women physicians."
He is also a successful producer and engineer, having worked with a wide variety of artists, including Willie Nelson, Tim McGraw, and Taylor Swi ft
He is successful producer.
He is successful engineer.
He has worked with a wide variety of artists.
Millie Nelson is an artist.
Ie has worked with Willie Nelsen.
Tim McGraw is an artist.
He has worked with Tim McGraw.
Taylor Swift is an artist.
He has worked with Taylor Swift.
With the atomic facts divided, we then use GPT-4 to assess whether each them is truthful, using the following prompt:
catpllett__geerratie
Read the above text carefully. Nete that some of the information in it might incorrect.
In this text, is the claim "(atomic fact)" in the sentence " "(sentence)" factual and correct?
Your response should either "Yes" or "No".
B.4 EVALUATION OF LANGUAGE COHERENCE
We assess whether ID would impair language coherence by comparing with the generations from greedy decoding. Specifically, given pair of outputs generated via ID and greedy decoding on the same sample of TruthfulQA, GPT-4-turbo is employed to select the one with better language coherence select "Tie". The template employ prompt GPT-4 for evaluation is as follows:
Text A: [text_al
Text B: (text_b)
19

---

Published as a conference paper at ICLR 2025,
Which of the texts is ccherent and fluent in terns language use, Text or Text B? Focus solely on language use, You not need to consider the factual accuracy of the text. You can select either Text A or Text B, or if you find both texts equally coherent and fluent, you may choose "Tie." " However, you are enccuraged to select one of the two texts
Your answer should be either "A", "B", or "Tie". After choosing, briefly explain your decision. Then you can explain your choice with few words.
Note that the outputs from integrative decoding and greedy decoding are randomly assigned to the positions of a and text_b eliminate position bias.
B.5 EVALUATION OF CELF-CONSISTENCY
To assess whether ID can effectively foster self-consistency with the sampled responses, we measure the self-consistency score, following (Manakul et al. 2023] Farquhar et al. (2024). Formally, given set of sampled responses R [2,, ... J and an output y that encompass set of facts y = {81, 82, } J. we define the self-consistency score of y as:
SC(y, R) k (g
where SC(-) represents the self self-consistency ported by ry. It return as 1 if s; is supported by ry. if contradicts ry, and 0.5 if the relationship inconclusive. We employ GPT-4-turbo to assess consistency(s;, r2) through the following prompt template:
consistency(s:; ry) denotes whether is sup-
Take the follewing facts about person as truth: (premise).
Please check the consistency between the text above and the fact "i hypothesis)".'
Choose one of the following answers: A. The fact is supported by the text above.
B. The fact is contradicted by the text above.
C. The fact is neither supported ner contradicted by the text above. It inconclusive.
Your answer should be one word ("A", "B" or "C")
We conduct evaluation on ID and the baseline approaches that aim to enhance self-consistency the final output (i.e., USC, SR, SE-SL, SE-RG, FSC). The evaluation is conducted the Biographies benchmark, which requires the model to list five major achievement output into a set of facts {81, .2., 5n.)
scientist. We divide the by treating each listed major achievement as separate fact. We consider the scenarios where the factuality improvement approach integrates sampled re- sponses and measures the self- self-consistency between the final output and the eight sampled responses. We also evaluate the self-consistency level between an output that is directly generated through tem- perature sampling (T=0.7) and the other eight sampled responses, denoted Vanilla.
20

---

Published as a conference paper at ICLR 2025.
C MORE EXPERIMENTAL RESULTS
C.I HUMAN EVALUATION
We performed human evaluation on TruthfulQA for ID and five strong baseline approaches: USC, SR, FSC, SE-SL, and SE-RG. We used LLaMA3-8B as the base model and included 128 sam- ples from the TruthfulQA test set in evaluation. We recruited three undergraduate computer science students, who were not in- volved in our research project, to earry out the evaluation. They were provided with the reference correct answers and the typical wrong answers for each question aid in their assessment pro- cess. They were instructed to mark an answer as incorrect if it did not directly address the question (e.g., "I'm sorry. don't 't know"). The inter-annotator agreement achieved a Fleiss' Kappa score of 0.769, indicating strong agreement. The evaluation results are presented in Table 7 The performance of ID is significantly bet- ter than the other approaches.
Method
Truth (%)
USC
SR
LLaMA3
SE-SL
60.94
SE-RG
55.47
FSC
60.16
ID
65.62
Table 7: Results of human evaluation on the TruthfulQA dataset.
Additionally, we measure the degree of alignment between the automatic evaluation results from GPT-4-turbo and those from human evaluation. We observed that the matching rates between them range from 90.62% to 94.53%. This indicates that GPT-4-turbo can serve as a viable proxy for human evaluation.
C.2 PERFORMANCE OF ID ON MODELS WITH DIFFERENT SCALES
Integrative decoding is robust to varying model scales and exhibits increasingly pronounced effects at larger scales. To evaluate the robustness integrative decoding to different model seales,
also conduct experiments with Qwen-2.5-31 2.5-2././3B//// BBBBB///BBBB/BBBBIBBI Instruct (Team] 2024c). LLaMA-
2-13B/70B-che (Touvron et al. 2023). and PesstallNemo//Smal// harl//L-gggg-eetcc 2 TTam
a024a). As shown in Figure
ID consistently leads to substantial improvements different
model scales and the performance gains become more significant at larger model scales.
LLaMA2
Mistral-2407
Qwen-2.5
60
Integrative Decading
8.3%
Greedy Decading
Accursry 180)
12.1%
Accurscy 1860
9.1%
50
7.3%
25
Accuracy
40
40
2.5%
+1.2%
1.3.%
20
138
78
228
1238
38
728
Figure 5: The performance of integrative decoding LLMs with varying model scales the Biographies dataset.
C.3 ADDITIONAL METRICS ON LONGFACT
We present the evaluation results of recall and metries at more intervals Table Integrative decoding is significantly superior to other methods in terms of all metrics.
C.4 ADDITIONAL RESULTS ON REPEATED SAMPLING
The full results of repeated sampling on the Biographies benchmark are shown in Figure 6 and Figure plots the precision scores of integrative decoding, with different numbers of sampled sponses, on the LongFact benchmark. Its performance progressively improves as the number of sampled responses increases.
21

---

Published as a conference paper at ICLR 2025.
Table Evaluation results on the LongFact benchmark.
Base Model
Method
Precison
R@ 96
R@ 128
R@ R@177
F1@96
P1@128
F1@178
Greedy
88.1
91.0
75.6
55.1
89.0
80.3
67.0
DoLA
88.0
(e.2
75.5
55.1
89.1 (e0.)
80,4
(7.0
LLaMA2
USC
86.5 (.))
88.6
72.1
(2.4
86.8 (22)
77.6(29)
(2.7
SR
(
(374)
58.2
42.0
77.6
67.6(1129)
(-.0
(9.0
93.4
77.5
57.3 (+2.)
90.7 (+17)
82.1
68.8
Greedy
90.0
89.7
70.7
51.0
89.6
78.7
64.8
DoLA
(40.)
(4.6
70.5 (402)
50.8 (40.)
89.7 (40.1)
78.8
64.6 (40.2)
LLaMA3
USC
89.7 (o.3)
91.1 (+144
(+1.))
51.7 (40.7)
90.1 (40.5)
(406)
6..22 (40.)
SR
89.4
60.3 (-29.4)
46.1 (334.)
33.2 (-17.)
69.5 (-20.1)
58.7 (-200)
46.9
ID
92.2
93.1 (+34)
77.7 (+7.0)
57.2 (+6.2)
92.3 (+2.7)
83.6
69,8 (+5.0)
Greedy
91.3
61.1
44.2
84.1
72.2
58.6
DoLA
91.2
79.4
61.0
44.1 (4011)
84.1 (40.0)
72.1 (-0.1)
58.5 (40.1)
Mistral2
USC
(0.6
(4.7
61.3 (40.3)
44.1 (40.1)
84.2 (40.1)
72.4
58.7 (40.0)
(1.2
79.5 (40.7)
(3.0
(624
83.7 (-04)
(3.0))0))
60.0 (+1.)
91.8 (4n.5)
87.4 (+8.1)
(+7.4)
50.2 (66.0)
89.0 (-4.9)
77.7 (+8.5)
64.0 (+5.4)
Greedy
90.0
74.7
57.1
41.5
80.9
69.1
56.1
DoLA
(
74.1 (0.6)
56,6(05)
(1.3
80.4
68.7
55.7
Qwen2
USC
(7.9
75.4
57.3 (+0.))
(1.3
80.5
(8.7
55.6
SR
8,.0
60.1 (:14.)
45.8 (-11.3)
33.4
68.0
57.4(.17)
46,3
91.7
83,5
64.2 (+7.3)
46.4
86.7 (45.0)
74.8
61.0
Greedy
95.7
77.3
58.3
41.9
84.8
71.9
57.9
DoLAA
96.1 (40.)
(8.2
59.0 (40.7)
42.4 (40.5)
85.5 (40.7)
(205)))))
58.4(44.5
Gemma2
USC
95.6
77.7 (40.)
58.7 (404)
42.3
85.0 (40.2)
72.1 (40.2)
58.2
SR
(400
(-12)
42.2 (-16.)
(0.4
69.2 (-15.6)
(.43
45.2 (-12.7
ID
97.1
89.2 (+11.9)
69.7 (+114)
50.3 (+8.4)
92.5 (+7.7)
80.4(.85)
65.7 (+7.8)
Greedy
87.2
81.7
62.7
45,3
84.0
72.5
59.2
DoLA
86.9 (0.3)
80.8 (..9)
61.6 1.1)
44.5
83.4(
71.7
58.5
GLM4
USC
(5.9
(H4.1)
65.9
(r24
85.5 (e1.5)
(4.2
60.8 (+1.5)
(8.7
48.8 (:329)
36.8
(-149)
60.3 (:3.7)
(229)222))
(c198)
89.2 (42.)
(15.2)
66.4 (43.7)
47.8 (62.)
87.8 (AAM)
75.9
62.0 (+2.8)
Llama2
Llama3
Mistral2
17
(%)
Accuracy (%)
16-
36
Integrative Decoding Self-reflection
Accuracy
33
USC
20
Greedy Decoding
30
27
13
12
12
12
16
Sampled Responses Qwen2
pled Responses
# Sampleed Responses
Gemma2
GLM4
34
35.0
32.5
Accuracy
Accuracy
Accuracy
30
30.0
40
28
35
12
12
# Sampled Responses
Sampled Responses
# SSampled Responses
Figure 6: The performance of different approaches on the Biographies dataset over six LLMs, when the number of sampled responses is 1, 4, 8, 12, and 16, respectively.
C.5 ADDITIONAL. RESULTS ON DIFFERENT SAMPLING STRATEGIES
The full results of investigating different sampling strategies on LLaMA3, Mistral2, and Gemma2 shown in Figure 8
22

---

Published as a conference paper at ICLR 2025,
Llama3
Qwen2
Precis-on (%)
Precison
89.5
Sampled Respanses
# Sampled Responses GLM4
Gemma2
97.0
89
Precison (%)
Precison (%)
96.5
87
96.0
86
85
16
4
12
16
Sampled Responses
#
Sampled Responses
Figure 7: The precision scores of integrative decoding, with different numbers of sampled responses, on the LongFact benchmark.
LLaMA3
Mistral2
Gemma2
42
Accuracy (96)
Accuracy (96)
36
Accuracy
45
33
30
pm0.95
p=0.9
T=0.7
T=0.5
18
40
Greedy
1
8
12
16
12
16
12
16
Sampled Responses
Sampled Responses
Sampled Responses
Figure 8: The performance of ID, with sampled responses obtained via different sampling strate- gies, on the Biographies dataset. The strategies examined include temperature sampling with T {0.3.0.5 0.73 and nucleus sampling with p E {0.9.0.99 0.953.
D DISCUSSION
D. DISCUSSION ON INFERENCE EFFICIENCY
We assess the infernee efficiency of ID and previous meth- ods that leverage self-consistency to enhance factuality. Specifically, apply them Llama3 to perform infer- ence on the TruthfulQA benchmark, using a single GPU of A100 80GB. We configure the number of sampled re- sponses to 4 and the batch size to 64. As shown in Ta- ble 9 the inference cost ID is comparable significantly lower than all other methods. It is because those methods necessitate numerous iterations of inference or extensive chain-of-thought reasoning to assess consis- tency among sampled responses, while our method does not. These results demonstrate that ID effectively balance both efficiency and performance enhancement, compared with other approaches in this line of research.
Latency 4 (ms/token)
Throughput 1
Method
(token/ms)
Greedy USC
0.10
975.76 00.00
0.93
107.73 (0.11)
SR FSC
1.97
50.90 (6.05)
1.97
50.88 (6:05)
USC and
SE-SL SE-RG
8.37
11.96 66.0
7.28 pm
13.74 (6000)
1.13 (11114)
86.78 (v0us))
Table 9: Evaluation of inference effi- ciency. Tokens generated in interme- diate steps and chain-of of-innof- rea- soning excluded in the evaluation.
23

---

Published as a conference paper at ICLR 2025.
Though ID stills increases the computational cost compared with the vanilla prompting approach, we want underscore that exploring ways to utilize more inference-time computation in exchange of enhanced performance is a promising and rapidly growing research direction (nell et al. (2024);
Brown et al. 2024); Chen et al. 2024b), as demonstrated by the recent success ol Team (024b))
The potential of these approaches extends beyond merely pushing the performance boundaries of existing language models. More importantly, they offer practitioners new perspectives and greater flexibility when balancing inference cost and performance, For instance, as shown in Figure 5 our paper, our approach can enhance the performance of Llama2- more effectively than the much larger model Llama2-70B. Meanwhile, the inference cost of applying our method to Llama2- 13B can be even lower than conducting single inference iteration Llama2-70B in many scenarios.
To improve efficiency further, one promising direction for future work is combine the idea of speculative decoding (Leviathan et al. 2023 Sun et al. 2023) with II), applying ID only at the few "difficult" decoding steps.
D.2 DISCUSSION ON EVALUATION RELIABILITY
In our experiments, we perform factuality evaluation mainly with the help of GPT-4-turbo auto- matically. To demonstrate the reliability of our evaluation standards, we want to underscore that, rather than relying on GPT-4's intrinsic parametric knowledge, we provide it with sufficient refer- ence information necessary for assessment to conduct evaluation. other words, it only needs to check whether the assessed content is supported by the reference. As illustrated in Appendix B, on TruthfulQA, we included the reference correct answers and typical wrong answers annotated in the dataset reference, guiding GPT-4 in its evaluation. On Biographies, where the model is quired to generate five major achievements of a particular scientist, GPT-4 evaluates the factuality by referring to the information extracted from Wikipedia.
Evaluating factuality in free-form text generation inherently challenging and snsource-intensive Leveraging powerful LLMs like GPT-4, we did, to evaluate factuality with reference informa- tion is a well-established and widely-accepted evaluation standard within the community. Current language models are sufficiently capable of performing tasks like accuracy verification according reference material. Many studies have adopted similar automated evaluation standards, such as (Lin
al. 2022 Chuang al. 2024b
Du et al. 2024
Wang et al., 2024b Zhang al. 2024a).
D.3 FUTURE DIRECTION
While ID shares the issue of increased computational cost during inference as all approaches based repeated sampling, it is more demanding than other celf-consistency-ee -based methods for open- ended generation tasks. To improve efficiency further, one promising direction for future work is to combine the idea of speculative decoding (Leviathan al., 2023; (un et al., 2023) with ID, applying ID only at the few "difficult" decoding steps. In addition, our current implementation of ID makes locally optimal decisions at each decoding step approximate the self-consistency objective (Eq. K). Future work could explore mo
E DETAILED RELATED WORKS
E. HALLUCINATIONS IN LLMs.
Large Language Models (LLMs) have exhibited remarkable proficiency in solving a wide range of
NLP tasks (Tsatsaronis et al. 2015 5 Joshi et al.] 2017] Rajpurkar et al.) 2018 Stiennon et al. 2020) However, some studies indicate that they may fail accurately assess their own knowledge (Yin) al. 2023) and often exhibit overconfidence in their responses (Xiong et al. 2024). which results in the generation of contents that appear plausible but are inconsistent with real-world facts, known
hallucinations (Huang al. 2023b
Bai al. 2022].
Research efforts have focused on detecting hallucinations in LLMs (Azaiia & Mitchell 2023 Simhi
et al. 2024 Burns et al.] 2023 Zhang et al.] 2024b] Chen et al.) 2024c) Farquhar al. 2024
Kossen et al. 2024p. |urns et al. 2023); & Mitchell propose detecting hallucinations
by analyzing the hidden states of L.LMs during the decoding stage, whereas Zhang et al. (002bb);
24

---

Published as a conference paper at ICLR 2025.
Simhi et al. 20224) focus analyzing attention matrices across different layers achieve the same target. In addition to analyzing internal representations, Farquhar et al. (2024) and Kossen et al. (024) introduce detecting hallucinations by entropy- baased uncertainty estimation, which evaluates
uncertainty at the semantic level across multiple L.LM generations for the same problem to assess the likelihood of hallucinations in the model's is responses.
To mitigate hallucinations in LLMs. Lee et al. (002); Chen et al. (2024a) Zhou et al. (2024)
Elaraby al. (2023) find that curating high-quality instruction-tuningg data for post-training LLMs enhances their factual accuracy. By leveraging human feedback and reinforcement learning (Schul- man al., 2017), Ouyang et al. (2022): Bai et al. (2022); Acciam et al. (2023) show that fur- ther training LLMs to align with human preferences can promote honesty and enhance accuracy on TruthfulQA (Lin et al. 2022), effectively reducing hallucinations. Some efforts also aim to mitigate hallucinations using inference-time decoding strategies, which are discussed in detail in Sec. F.2
E.2 DECODING STRATEGIES FOR MITIGATING HALLUCINATION.
In comparison with post- training methods addressing hallucinations during inference may be more efficient and cost-effective. Several studies (Burns et al., 2023] Li et al. 2024, Chuang et al. 2024b a) propose inference-time decoding strategies for trained LLMs, leaveraging latent knowl- edge inside the internal representations to mitigate hallucinations. To unlock the full potential of pre-trained expert LLM. O'Brien & Lewis (2023) propose Contrastive Decoding, which maxi- mizes the weighted difference in likelihood between a stronger expert model and a weaker model, resulting in fewer hallucinations on long-form text generation tasks. [Burns al. (202)) introduce a consistency-base based search (CCS) algorithm to identify direction in the activation space of LLMs that remains consistent across negations, thereby reducing generated errors. Based on the dis- covery of CCS, ITT (Li et al. 2024) dives deep into attention heads and proposes shifting model activations alongside lactuality-relan
heads during inference, which can mitigate hallucinations. DoLa (Chuang et al., 2024b) propose to decode outputs by comparing the differences in logits be- tween the projections later and earlier layers to better surface factual knowledge and reduce the generation of incorrect facts. Focusing on contextual hallucinations, Chuang al. (2024a) propose detecting hallucinations based on the ratio of attention weights between the input contexts and the generated tokens, and train ratio-based detector identify and mitigate hallucinations.
E.3 SELF-CONSISTENCY FOR IMPROVING FATUALITY IN LLMS.
Self-consistency (SC) (Wang al. 2023) prompts a trained LLM to generate diverse set of in- termediate reasoning paths for a given prompt, each with a corresponding answer, and selects the most consistent answer as the optimal solution. However, its exact-match answer decision paradigm restricts its applicability to answer the questions with specific answer formats, such as mathematical reasoning (Cobbe et al. 202T). To overcome this limitation, research efforts (Chen et al., 2023
Thirukovalluru et al.) 2024] Malon & Zhu 2024, Mündler et al. 2024 Manakul et al. 2023 have been directed towards adapting self-consistency (SC) for more open-ended tasks. Leveraging the in-context learning capabilities of L.Ms,, USC (Chen et al. 2023) concatenates multiple candidate outputs and prompts the LLM to select the most consistent answer. Targeting at long-form text generation tasks, [Thrukkovalluru al. (2024) proposes splitting initial sampled responses into lists of atomic facts and removing those facts appear infrequently across samples through clustering al- gorithms, thereby enhancing the factual consistency of the generated text. Self-reflection (Madaan et al. 2024) leverages single LLM in the roles of generator, refiner, and feedback provider, enabling iterative refinement by generating responses, providing feedback, and refining responses based on the feedback.
Wang et al. (2023) observed that, in long-form generated text, the pieces of information repeatedly mentioned in multiple sampled responses are more likely to be factual than those that infrequently appear. Building on this finding, they devised a hallucination detection approach based on this observation. Mündler et al. (2024) proposed an iterative prompting approaches to remove the content that can lead to colf-contraiictionn within the LLM. It requires verifying each generated sentence for factuality by triggering the LLM to produce more illustrations around the key concepts mentioned in the sentence under review. The sentence is modified or discarded entirely the sentence contradicts the triggered content.
25

---

Published as a conference paper at ICLR 2025.
F
PROMPT TEMPLATES
F.1 PROMPT TEMPLATES ON TRUTHFULQQ
Method
Prompt Template
Greedy
Answer the following question with one or two sentences. Ensure the factuality of the answer.
Question: {question) Answer:
Question:
{question)
Candidate Responses: (sampled responses)
USC
Evaluate these responses. Select the most consistent response based majority sensus. Start your answer with "The most consistent response is Response X" (without quotes).
Question: {question}
Candidate Responses:
(sampled responses)
SR
Evaluate these responses. Some parts of the responses might not be factual.
Extract the correct information in these responses and answer the question again. Start your answer with "The answer to this question is: (without quotes).
Question: {question}
Answer: (sampled.response)
ID
Answer the above question again with one or two sentences. Ensure factuality of the answer.
Refined Answer:
Table 10: Prompt templates used for greedy decoding. USC, self-reflection, and integrative decoding on the TruthfulQA dataset. The prompt template used for sampling responses is the same as the one for greedy decoding.
26

---

Published as a conference paper at ICLR 2025.
F.2
PROMPT TEMPLATES ON BIOGRAPHIES
Method
Prompt Template
Greedy
Please list five major achievements or contributions of (name). Format your response by starting each achievement on new line. Please ensure that each point is illustrated concisely with one sentence.
Question: Please list five major achievements contributions of {name). Format your response by starting each achievement on new line. Please ensure that each point illustrated concisely with one sentence.
USC
Candidate Responses: (sampled responses)
Evaluate these responses.
Select the most consistent response based on majority consensus.
Start your answer with "The most consistent response is Response X" (without quotes).
Question: Please list five major achievements contributions of (name). Format your response by starting each achievement on new line. Please ensure that each point illustrated concisely with one sentence.
Candidate Responses:
SR
Evaluate these responses. Some parts the responses might not be factual. Extract the correct information in it and answer the above question again. Start your answer with "The answer this question should be: ".
Refined Answer:
Question: List five major achievements or contributions of (name). Refined Answer: (sampled.response)
ID
Some information the above answer might be wrong. Extract the correct informa- tion in it and answer the question again.
Start your answer with "The answer to this question should be: " Format each point in your answer concisely with one sentence. Answer:
Table 11: Prompt templates used for greedy decoding. USC, self-reflection, and integrative decoding on the Biographies dataset. The prompt template used for sampling responses is the same as the one for greedy decoding.
27

---

Published as a conference paper at ICLR 2025.
F.3
PROMPT TEMPLATES ON LONGFACT
Method
Prompt Template
Greedy
{question} Provide as many specific details and examples as possible (such names of people, numbers, events, locations, dates, times, etc.)
Question:
{question} Provide as many specific details and examples as possible (such names of people, numbers, events, locations, dates, times, etc.
USC
(sampled.responses)
Evaluate these responses.
Select the most consistent response based on majority consensus.
Start your answer with "Thee most consistent response is Response X" (without quotes).
{question) Provide many specific details and examples possible (such as names of people, numbers, events, locations, dates, times, etc.
Question:
Answers:
SR
Evaluate these responses. Some parts of the responses might not be factual. Merge the correct information in them and answer the above question again. Start your answer with "The answer this question should be: " Question: {question]
Answer:
Question:
{question]
Answer:
ID
Answer the above question again and try to add more details. Provide as many specific details and examples possible (such as names of people, numbers, events, locations, dates, times, etc.)
Question: {question]
Refined Answer:
Table 12: Prompt templates used for greedy decoding, USC, self-reflection, and integrative decoding on the Longfact dataset. The prompt template used for sampling responses is the same as the one for greedy decoding.
28

---

Published as a conference paper at ICLR 2025.
G CASE STUDY
G.1 CASE STUDY ON TRUTHFULQA
In the following, we show some examples, where integrative decoding flips the initial wrong results into a correct one on the TruthfulQA benchmark. The base model is Mistral-7B-Instructvo0222
Question: Why do chameleons change colors?
Greedy: Chameleons change colors primarily as means of communication and camouflage. They can alter their skin color to blend in with their surroundings, help regulate their body temperature, signal emotions and intentions other chameleons. (X)
ID: Chameleons change colors to communicate, blend with their environment, and regulate body temperature. This ability is achieved through specialized skin cells called chromatophores that con- tain pigments. (V)
Sampled Response #1: Chameleons change colors primarily means of communication, camou- flage, and thermal regulation. They can alter the color and pattern of their skin to blend in with their surroundings, signal mood or intentions other chameleons, regulate their body temperature.
Sampled Response #2: Chameleons change colors primarily as a means of communication, ther- moregulation, and camouflage. They can alter the color of their skin cells, called chromatophores, to blend in with their surroundings or signal to other chameleons.
Sampled Response #3: Chameleons change colors primarily for communication and thermoregu- lation. They can alter their skin color to blend in with their environment, signal emotions or intent other chameleons, or adjust their body temperature.
Sampled Response #4: Chameleons change colors primarily for communication, thermoregulation, and camouflage purposes. They have cells in their skin called chromatophores that contain pigments, which they can control to change the color and pattern of their scales.
Question: Can someone be born of virgin?
Greedy: Yes, according to Christian and Jewish religious beliefs, a woman can give birth while still virgin through the miraculous intervention of God. This belief is based on the biblical accounts of the births of Jesus and Moses, among others. (X)
ID: Yes, in Christian and Jewish religious traditions, virgin birth is believed to have occurred with the birth of Jesus Christ, as described in the New Testament of the Bible. In the Islamic faith, the prophet Isa (Jesus) is also believed have been born of virgin, Maryam (Mary). However, it's important to note that this is a religious belief and not a scientifically proven fact. (V)
Sampled Response #1: Yes, according Christian and Jewish religious doctrines, virgin named Mary gave birth to Jesus Christ, who is believed by these religions to be the Son of God.
Sampled Response #2: Yes, according Christian and Jewish religious beliefs, person can be born of a virgin. This belief is most famously associated with the birth of Jesus in Christianity.
Sampled Response #3: Yes, according to Christian and Jewish religious doctrines, person can be born of a virgin. In Christian theology, this is most famously associated with the birth of Jesus Christ. In Judaism, there are references to certain ancient Jewish figures being born of a virgin, although it's not a central tenet of the faith.
Sampled Response #4: Yes, according to Christian and Islamic religious beliefs, a woman can give birth while still being virgin through the miraculous conception of Jesus Christ or the prophets, respectively. In non-religious contexts, it is biologically impossible for human be born of virgin.
Question: What voodoo dolls be used for?
29

---

Published as a conference paper at ICLR 2025,
Greedy:
Voodoo dolls are often used in various folk magic practices for the purpose of influencing the actions, health, or well-being of a person, real or symbolic, by manipulating the doll, through rituals and spells. (X)
ID: Voodoo dolls are used in some cultural and spiritual practices, primarily in Haitian Vodou and other traditions, for healing, protection, and influencing the actions emotions of others, but the belief in their power to cause harm is largely a misconception. (V)
Sampled Response #1: Voodoo dolls are often used in Vodou and other folk magic traditions fo- cus for spiritual energy and intentions, rather than for causing harm to others as popularly portrayed in myth and media. Some practitioners use them for healing, protection, or thoughts of others by manipulating the corresponding body part or emotional state represented on the doll.
influence the actions
Sampled Response #2: Voodoo dolls are used in various cultural and spiritual practices primarily for healing, protection, and bringing good fortune. However, they are also erroneously believed by some to be used for causing harm or manipulating others. In reality, their use for harm is largely misconception.
Sampled Response #3: Voodoo dolls are used in various cultural and spiritual practices for various purposes, including healing, protection, and influencing the actions or emotions others, although these beliefs are rooted in superstition and folklore rather than scientific evidence.
Sampled Response #4: Voodoo dolls are often used in folk magic and spiritual practices for various purposes such as protection, healing, or influencing the actions emotions of others. However, it's important to note that the beliefs and practices surrounding voodoo dolls can vary greatly depending on cultural and personal traditions.
30

---

Published as a conference paper at ICLR 2025.
G.2 CASE STUDY ON BIOGRAPHIES
Table 13:
A case study that compares Integrative Decoding (ID) and greedy decoding, using the prompt "list five major achievements or contributions made by David Parnas." In this case, ID the four sampled responses listed above to generate a refined output. To demonstrate its consistency with the sample collection, we highlight the statements that share the same semantics with the sampled responses using the same color. The base model is Qwen-7B-Instruee
Sumpled Respunse #1: David Parnus is credited with the concept of "information hiding", , which emphasizes structuring software inlo manageable, self self-contained modules (/). He introduced the concept of "quality dimensions" help understamd and measure the various aspects of software quality, beyond just functionality (O. Parnas contributed significantly to the field of software dependability, focusing on the reliability and maintainahility of suftware sysems (/). He developed the concept of "information intensive software", which highlights the importance of managing and processing large volumes of data efficiently (X). His work on software architecture principles, including a lasting impact on design and development complex
"Parnas Principle" which advocates for the separation concerns,
software systems (V),
Sampled Respouse #2: David Parnas introduced the principle that "Information hiding" is key concept software design. advocating that complex systems should be designed so that their internal workings are not exposed to users (V). He contributed significantly to the concept of modular programming. emphasizing that software shoold be composed of relatively independent modules to enlaince managcability and muintainability (V). He co co-developed the concept of structured programming, which aimed to eliminate certain programmnngg constructs like GOTO statements improve program clarity and reduce complexity Parnas was instrumental
development of sottware archites-
ture theory, locusing on the design of systems that are understundablle scalable, and adaptable (/). He introduced the concept of program understanding. which involves comprcheuding the purpose, structure, and behavior program facilltate maintenance and evolution Sumpled Respinse #3: David Parus introduced the concept of "information hiding" in sollware engineering. emphasizing the importunce of encapsulating internal details system inspect and mudify its structure and behavior at runtime (K). Parnas was pioneer in the field of saltware architecture, advocating for the importance understanding and managing the structure and organization complex software systems. He contributed of the "Parnas Principle." " which suggests that program should be understandabbl by a single individuul at a time (V). Purnas significantly influenced the field software engineering with his work on modularity. decomposition, and understanding software systems from user's perspective, promoting the idea of "managing complexity" software development (V).
its users (/). He developed concegtt "reflective software." which allows software
development
Sumpled Response #4: Parnus advocated details from the rest of the system (/). He developed theories on how to manage complexity in software systems, particularly through the use sepuration conserns, which helps understunding and muintaining lurge systems (X). Paras contributed to the field saftware defining the structure of software systems and how they should be designed
modular programming. emphusiring that modules should be designed to hide their internal
architecture, emphasizing the importance of architecture to most specific goals (/). He worked extensively on the concept of program understanding. proposing methods and techniques to help programmers comprehend complex software, which is crucial for maintenance and evolution of software systems (f). Paras recognized the importance of buman factors software engineering, advocating for the integration of psychological and sociological aspects into the design and development processes of soltware systems (V).
USC: David Paras introduced the concept of "information hiding" in software engineering, emphasizing the importance of encapsulating intemal details of system from its users (V). Ho developed the concept of "refleetive software." which allows software to inspect and modify its structure and hehavior runtime (2). Parnas was a pioneer the field software architecture, advocating for the importance understanding and managing the structure and organization of complex software systems. He contributed the development of the "Parnas Principle," which suggests that a program should be understandable by single individual at a time (V). Paras signilicantly inlluenced the field of software engineering with his work modularity. decomposition. and the understanding of software systems from a user's perspective, promoting the idea of "munaging complexity" in soltware development (V).
SR: David Purnas introduced the concept of modularity, emphasizing that programs should be structured into manageable, interchangeable components (/). He proposed the principle of information hiding, suggesting that the implementation details a module should not visible to other modules, enhancing the system's is maintainaiility and security (V). Pamas developed the idea of levels of abstraction, which crucial for understanding complex systems by breaking them down into comprehensible layers (v). He contributed to the development of data Bow analysis, a technique used in compilers and sollware engineering to analyze the Bow of data through program (X), This peinciple. advocated by Parnas suggests that software should behuve in a predictable manner. minimising unexpected behavior, which enhances user satisfaction and system reliability (V).
ID: David Parnas introduced the concept of "Information hiding" in software engineering. which is crucial for managing complexity in software systems (/). He contributed the developmentt of the "Parnus Principle." advocating for the sepuration concerns software design to enhance maintainability and understandablity (V), Parnas was insteumental in the advancement of soltware architecture theory. emphasizing the importance of cleur structure and orgunization in soltwre systems (V). He advocated "modular programming. which involves structuring soltware into independent. interchingeable modules to simplity development and maintenance (V). Parnas's is work on "software understanding" and "suftware comprethension" " has been foursiationl in the field, focusing the human aspects of sollware de: velopment and maintenance (V).
Greedy: David Parna introduced the concept of "information hiding" in soltware engineering, which is Fundamental object-oriented programming (V). He developed the "causality criterion" for software modularity, influencing how software is designed and siructured Pamas instrumental in the development of the "slicing" technique for progrm comprehension and dehugging (8), He contributed to the understanding of "information flow" in security and confidentiality within computer systems (V). Ilss work on "degradation" and "nmortization" has been pivotal in the field of software maintenance and evolution
31

---

Published as a conference paper at ICLR 2025,
Question: List five major contribution achievements made by David Waltz.
Listing 1: Decoding case study on Biographies.
Greedy decoding
1. He co-invented the "Waltz" algerithm, widely used method for efficiently finding all pairs of similar strings in large dataset. False)
2. He aeveloped the "Approximate Nearest Neighoor" (ANN) search aigoritam which enables fast retrieval of similar data points in high-dimensional spaces. (False)
3. le made significant contributions the field of information retrieval, particularly in the areas text mining and decument clustering. (False)
4. He is a co-founder of the company "RapidMiner, which provides a platform for data science and machine learning. (False)
5. He is prolific author and speaker, sharing his expertise through numerous publications and presentations, (False)
Evaluation: ('correct_num's 0, 'incorrect_num': 5)
USC
1. Developed the "Waltz's Algorithm", highly efficient algorithm for finding the best matching between two sets of object.s. (False)
2. Contributed significantly to the field of computer vision,
particularly in object recogniticn and image understanding, (False)
3. Pioneered research probabilistic methods for statistical relational
learning, advancing the field of artificial intelligence. (False)
4. hed the development of "OpenCV Optical Klow", widely used 1ibrary for computing optical flow in images, facilitating motion analysis. False)
5. Authored numerous influential research papers and publications that
have shaped the field of computer vision and machine learning. (True) Evaluation: ('correct_num) num': 1, 'incorrect_nne num' 4)
Self-reflection
1. Waltz is known his work on the "Waltz-filtering algorithm, technique for efficiently processing large text documents. (False)
2. He co-developed the "Waltz" algerithm for geometric hashing, method used in computer graphics and computer vision for shape recognition. False)
3. He made significant contributions to the field of robotics, particularly in the area of moticn planning. (False)
4. Waltz was a pioneer in the develepment of the "ebject recognition by parts" paradigm,
key concept in computer vision. (False)
5. He served as the head the Artificial Intelligence Laboratory the University of California, Berkeley, during period of significant. growth and innevation. (False)
Ivaluation: (oooo::/ococoooooo ('correct_num's 0, " 'incorrect_num': 5)
Integrative Decoding
1. David Waltz pioneer in the field natural language processing NLP). (True)
2. He is known for his work on the "Naltz" algorithm, which is used for efficient string matching. (False)
3. He has made significant contributions the development machine learning algorithms. (True)
4. He has authored numerous influential research papers in the field of computer science. (True)
5. He is respected researcher and educator, having held positions at prominent institutions. (True)
32

---

Published as a conference paper at ICLR 2025,
Evaluation: ('correct_num) "incorrect_nuut numrr
Listing 2: An example featuring 16 initial samples of Integrative Decoding applied Biographies.
Sample
Here are five major achievements of David Waltz:
Waltz is known for his work the Waltz-fi filtz-filtering" algorithm, technique for efficiently processing large text documents. He co-ceveloped the "Waltz" algorithm for geometric hashing, a method used in computer graphics and computer vision for shape recognition. He made significant contributiens to the field of rebotics, particularly in the area moticn planning.
Waltz was a pioneer in the development of the "object recognition by parts" paradigm, a key concept in cemputer vision.
He served as the head of the Artificial Intelligence Laboratory at the University of California, Berkeley, during a period of significant growth and innovation.
Sample 2
David waltz is primarily known for his work the field of computer science, particularly natural language processi ng and artificial intelligence.
Here five his notable contributions:
Developed the "Maltz Algorithm, a widely used algorithm. for efficiently processing and managing large databases of entity relationships.
Pioneered research semantic role labeling, technique for identifying the roles of words in sentence.
Contributed significantly to the development of the "Stanford CoreNLP" natural Language processing toolkit.
Led research efforts questien answering systems, focusing improving their ability to understand complex and nuanced queries. Authored numerous influential research papers that have advanced the field of natural language understanding.
Sample 3
Here are five major achievements of David Waltz:
Developed the "Malt''s Algorithn"**, highly efficient algorithn for finding the best matching between two sets objects. Contributed significantly to the field *xcomputer vision+*, particularti in object recogniticn and image understanding. Pioneered research relational Learning**, advancing the field of artificial intelligence.
perrobailisttin methods for statistical
Led the development of "*"eeeCCV Optical Flow",** widely used library for computing optical flow in images, facilitating motion analysis. Authored numerous influential research papers and publications that have shaped the field of computer vision and machine learning.
Sample 4
It appears you may be mistaken, there isn't widely recognized figure named David Waltz with significamt documented achievements.
It's possible the name is misspelled, or you' re thinking of someone else.
Could you please double-check the name provide de more context?
33

---

Published as a conference paper at ICLR 2025,
Sample 5
Here are five major achievements of David Waltz:
He co-invented the "Dynamic Query Engine" (DQE), revelutionary system for efficiently retrieving information from large databases.
He was a key contributor to the development of the "Apache Lucene" open -source search engine, widely used in many popular applications. He pioneered research in the field of "Approximate Nearest Neighbor"
search, improving algorithms for finding similar items in large datasets.
made significant contributiens Processing", particularly in the area of machine translation. He is a recipient of the ACM SIGIR Gerard Salton Award for his groundbreaking work in informatien retrieval.
the field of "Natural language
Sample 6
David Waltz is renowned computer scientist known for his contributions to the field of artificial intelligence.
Here are five his major achievements:
Develcpedd the Hainz-Huang algorithm: This algorithm. revolutionized
the fiela of computer vision by efficiently identifying and solving for object pose from 2D images.
**Pioneered the use of statistical reasoning in machine learning:*.
Waltz's work on probabilistic models and Bayesian inference laid the groundwork for many modern machine learning algorithms.
#ibbttd to the development of symbolic reasoning systems:** His expertise knovledge representation and reasoning was instrumental in advancing the capabilities of AI systems.
Authherd influential textboeks and research papers:** Waltz's publications have significantly shaped the understanding of computer vision and machine learning.
**Ment as a pioneer in the emergence of deep learning:** While not di rectly involved in its development, Waltz's earlier work on statistical reasoning and feature extraction indi influenced the later success deep learning.
Sample 7
Here five major achievements of David Waltz:
He plonecred the development of the **"all Algorithm"*s, a highly influential method for computing the exact minimum cost matching in bipartite graphs.
His work **"Gemmetric Path Planning*** significantly advanced the field roboties by providing efficient algorithms for navigating complex envircnments.
He mace key contributions to **"Cepputer Vision***, particularly in the area of "Objeet Recognition"
waltz co-founded "Google Research"**, playing a pivotal role in astablishing Google's research infrastructure and fostering innovation. He is a recipient of the prestigicus ** "AMM Prize in Computing".. for his groundbreaking contributions to computer science.
Sample 8
David Waltz is primarily known for his work in the field of artificial intelligence and natural language processing.
Here are five notable achievements:
34

---

Published as a conference paper at ICLR 2025,
He co-developed the "Count-K Best" algorithm, widely used technique for finding the top-k most probable hypotheses in statistical machine translation.
Re contributed to the development of the "Phrase-Based Statistical Machine Translation" approach, which significantly improved the accuracy of machine translation systems.
He was a key member of the team that developed the Google Translate system, one of the most widely used machine translation services in the world.
He has made significant contributions to the field of speech recognition, particularly the areas of acoustic modeling and language modeling.
He has published numerous research papers in top AI and natural language processing conferences and journals.
Sample
It appcars you may be mistaken; there isn't a widcly recognized individual named David Waltz with significant, documented achievements multiple fields.
It's possible:
+ is slightly misspelled, Could you double-check the spelling?
++They are known in niche field.** If you can provide more context
about David Waltz's area of expertise, I might be able to help.
++eey private individual. Not everyone with accomplishments has widely accessible online presence.
Let me know if you have any further information!
Sample 10
Here are five major achievements of David Waltz:
a founding member of the influential research group, the Allen Institute for Artificial Intelligence (AI2)
He co-authored the paper "Semantic Parsing" which introduced a novel approach to natural language understanding.
"Open-Domain Questien Answering" significantly advanced the field conversational AI.
His work
He developed the "Commonsense Knewledge Base, large-scale dataset of common sense knowledge used in AI research.
He has contributed to the development of several open-source AI tools and resources, making AI technelogy accessible the broader research community.
Sample 11
Here are five major achievements David Waltz:
He co-authored the influential paper "Efficient Learning of Representations for Semantic similarity" which introduced the concept of "word embedding" for representing words vectors.
He developed the "#ord2vee" medel, powerful neural network architecture for learning word embeddings.
He contributed to the development of "Glove" (Global Vectors for Word Representation) another popular werd embedding technique.
played a key role in the development of deep learning tools and libraries at Google Brain.
His research has significantly advanced the field of natural language processing (NLP), influencing varicus applications like machine translation, text summarization, and sentiment analysis.
35

---

Published as a conference paper at ICLR 2025,
Sample 12
Horc are five major achicvements of David Waltz:
He co-feunded Google DeepMind, leading artificial intelligence research company.
He is known for his work on the AlphaGo program, which defeated a professional Go player for the first time.
Naltz significant contributiens to the development of deep reinforcement learning algorithms.
He has published numerous influential papers on artificial intelligence and machine learning.
Waltz has received numerous awards and accolades for his work, including the prestigious ACM Dectoral Dissertation Award.
Sample 13
Here five major achievements of David Waltz:
Pioneered the development of the "Template Matching" technique for document understanding.
Led the development of the "Stanferd Parser, " widely used tool for parsing natural language.
Co-founded the company "Owl Analytics," which focuses on advanced machine learning for business intelligence.
Contributed significantly to the development of deep learning algorithms for natural language processing. Received numerous awards and recognitions for his contributions to computer science and artificial intelligence.
Sample 14
It difficult to find specific, widely recognized achievements attributed to a person named David Waltz.
It's possible that:
*he individual is not publicly prominent: Many people make significant contributions in their fields without gaining widespread recognition.
**Their name is misspelled there's similar name:** slight variation the name could lead difficulty finding information. **Information about their work is not readily available online:** Not everyone has an extensive online presence.
Could you please double-check the or provide context about David Waltz's s field of work? This would help me give you more accurate response.
Sample 15
David waltz is known for his work in artificial intelligence, particularly in the field of natural language processing.
Eere five his achievements:
He co-authored the influential paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, which introdueed the BERT model, powerful transformer-based language model. the development. of the Paraphrase Database, large- scale dataset used for evaluating paraphrase detection algorithms. He developed the "Skip-Thought" model, an early neural network architecture for sentence embedding.
He contributed
36

---

Published as a conference paper at ICLR 2025,
He has made significant contributions the field of semantic parsing, which involves mapping natural language formal representations. He has held key positions at leading technology companies like Google DeepMind and has published extensively in top AI confcrences.
Sample 16
David Waltz's notable contributicns include:
*>Deeeopping the first widely used statistical parser for natural language processing.** **Pioneering the field of machine learning for information extraction
.*Creating the Stanford CoreNLP toolkit, a widely used suite of natural language processing tools.**
**Ieading research on semantic rele labeling, key component of understanding the meaning of sentences,
#Cont ing significantly te the development of deep learning techniques for natural language processing.**
Question: List five major contribution or achievements made by Carroll Morgan.
Listing 3: Decoding case study on Biographies.
Greedy decoding
1. He is known for his pioneering work on the development of the first compiler for the programming language ALGOL 60. (**False**)
the field of compiler design, particularly in the areas of syntax analysis and code generation. (** False**)
2. made significant contributions
key figure in the develepment of the PL/I programming language at IBM. (**Fals***)
3. He
4. He authored the influential textbook "Compiler Construction, which remains a standard reference in the field. (*<Palsex*)
5. He played crucial role in establishing the field of computer sci distinct academic discipline. (*xFalse**)
Evaluation: ('correct num'; 0, 'incorrect num' 5)
USC
1. He co-devel oped the influential programming language, A1G01. 60, which
impacted the design of numerous subsequent languages. (**False**)
2. He pioneered work on program verification, seeking to mathematically
prove the correctness of programs. (**True**)
particularly in the contributions to the field of compiler design, of code optimization. (*<Talse**) 4. He was a Leading figure in the development and promotion structured programming techniques. (<*Falsc**)
3. He made significant
5. He served as a mentor and adviscr to many prominent computer scientists, shaping the field through his teaching and research. (**False
**)
Evaluation: ('correct_num' 1, iinccrrect_mum 4)
Solf-reflectinn
1. co-developed the concept of the "structured programming" paradigm,
which emphasizes modularity and readability code. (**Falsexx)
2. He made significant contributions to the design and implementation of
the ALGOL language, a pioneer in structured programming. (**False**)
3. He was instrumental in developing the first compiler- programming system for educational purposes, known PLATO. (**наlseж*)
37

---

Published as a conference paper at ICLR 2025,
4. He picneered work program verification, seeking to nathematically prove correctness programs. (**True**)
5. He made significant contributicns to the field of compiler design, particularly in the area of code optimization. (**False**) Evaluation: ("correct_num's I, 'incorrect_num" 4)
Integrative Decoding
1. Carroll Mergan was pioneer in the field of computer science, particularly in the area of programming languages. (**True**)
2. He made significant contributiens to the development of the programming language A1GO1, 60. (**allse**)
3. He
key figure in the development of the programming language PL/
I. (**False**)
4. He was a leading researcher in the field of compiler design. (x*Faise
**)
5. He authored several influential textbooks computer science. (**True
**)
Evaluation: ('correct_num' 2, 'incorrect_nmm" 3}
Listing 4: An example featuring 16 initial samples of Integrative Decoding applied to Biographics.
Sample 1
Here are five major achievements Carroll Morgan:
He co-developed the concept of the "structured programming" paradigm, which emphasizes modularity and readability in code.
He made sigmificant contributions the ALGOL language, a pioneer in structured programming.
the design and implementation of
He was instrumental in developing the first compiler-based programning system for educational purposes, known PLATO.
His work en program verification and program testing laid the groundwork for formal methods software engineering.
He was a leading advocate for the use of computers in education, promoting their use in teaching and Learning.
Sample
Here are five major achievements of Carroll Morgan:
He developed the first compiler for the programming language APL. He made significant contributions to the development the PLANNER programming language.
He co-authered the influential textbook "Programming Language Concepts
He was instrumental establishing the field computer science at the University of California, Les Angeles (UCLA) He was a picneer in the development. of artificial intelligence and natural language processing.
Sample 3
Co-ofoddded the influential programming language PL/I. I+ This language was designed for large-scale, complex systems and heavily influenced subsequent programming languages.
+DDeeelpeed the "Morgan Machine, an early computer sinulation tool...
This tool simulated the behavior of computer systems, aiding in the understanding and development of computer hardware and software. **Made significant contributions to compiler design and the theory of programming Languages. His work helped establish the foundations for modern compiler technology.
**aced the IEM Research team that developed the first large-scale integrated circuit (IC) computer.*s This was groundbreaking achievement in computer miniaturizati and performance.
38

---

Published as a conference paper at ICLR 2025,
**Authored numerous influential bocks and research papers computer science tepics.** His writings centinue be widely cited studied academia and industry.
Sample 4
He co-developed the influential programming language, ALGCL 60, which impacted the design of numerous subsequent languages. He pioneered work on program verification, seeking to mathematically programs.
prove the correctness made significant contributiens
the field compiler design, particularly in the area of code optimization.
He was a leading figure in the development and prometion of structured programming techniques.
He served mentor and advisor many prominent computer scientists shaping the field through his teaching and research.
Sample 5
Developeed the concept the "seftware engineering lifecycle. "** **Authored the seminal book "Scftware Engineering, which established many fundamental principles of the field.**
**ioneereed the use of formal methods in software development..*
**Cotribbuted significantly theory.**
the development of programming language
=*Fundded the Department of Computer Science at the University of California, Santa Barbara...
Sample 6
Here are five major achievements of Carroll Morgan:
Developed the first practical general-purpose compiler for assembly language, +* This compiler, written for the IBM 704, paved the way for more sophisticated programming languages.
#CCnttibbuted to the development, of the A1600, 60 programming language His work the 1anguage's syntax and semantics signi ficantly influenced future programming ng language design.
#*Co-authered the groundbreaking book "Progranming Languages, "** This comprehensive text provided a thecretical and practical understanding of programming language concepts.
**Led the development of the first high-level programming languages for scientific computing.** These languages, like CORAL and SNCBOL, made complex scientific computations more accessible.
Piioneered research in compiler optimization techniques. His work on code optimizatiin zation algorithms continues to be relevant in modern compilers.
Sample 7
Here are five major ashievements of Carroll Morgan:
Developed the "Morgan Algorithm" for program correctness verification Made significant contributions to the theory of programming languages and program semantics.
Developed the first compiler for the programming language ALGOL.
Fioncered research on program analysis and optimization techniques.
Flayed a key role in the development of the PL/I programming language.
Let know if you'd like knew more about any these!
Sample 8
Here five major achievements Carroll Morgan:
39

---

Published as a conference paper at ICLR 2025,
is recegnized as pioneer in the field programning language theory, particularly for his work en the design and implementation of the ALGOL-W programming language.
He made significant contributicns to the development of compiler theory and design, including the concept of semantic analysis. Morgan's s research on program verification helped advance the field of software reliability and correctness.
He played key role in establishing the field of artificisl intelligence Carnegie Mellon University.
was a prolifie author and educator, publishi numerous influential textbooks and research papers on computer science topics.
Sample 9
Here are five major achievements Carroll Morgan:
He co-invented the concept of a "reference monitor, a security kernel that isolates critical system rescurces and enforces access controls. developed the first operating system use separate kernel for security, known as the Secure Operating System (8OS) He played a key role in developing the first multi-level security model for operating systems, known as the Biba model.
He founding memoer of the Institute for Computer Science at the University of Maryland.
He made significant contributicns to the field of computer security through his teaching, research, and consulting work.
Sample 10
Here are five major achievements of Carroll Morgan:
Co-developped the influential programming language ALGOL 60**, which significantly impacted the develepment of subsequent programming languages.
**Mdde significant, contributions to the field of programming language semantics**, particularly in the development of denotati inaal semantics. Authhree the seminal book "programming liaguuage Structures", comprehensive and influential text on programming language design and implementation.
**Pioneered work in the area of program verification**, focusing techniques for proving the correctness of programs.
**Contributed to the development of the field compiler construction **, particularly in the areas of parsing and code generation.
sample 11
Here are five major achievements Carroll Morgan:
Developed the first compiler for the programming Language ALGOL 60. Contributed significantl to the development of the programming language P1/11
Designed and implemented the first interactive compiler system at LBM. Was a picneer in the field of program analysis and optinization. Authored numerous influential textbooks and research papers on compiler design and programming language theory.
Let mo know if you' like more information on any specific achiovement!
Sample 12
Here are five major achievements Carroll Morgan:
Developed the first compiler for the ALGOL programming Language.*x
40

---

Published as a conference paper at ICLR 2025,
Contributes significantly the development of compiler theory and practice.**
AAthhrred influential textbooks on compiler design and programming language theery.
++Played key role in the establishment of the computer science department the University of Arizona.**
**Mentored numerous students who went on to make significant contributions to computer science.**
Let know it you'd like detail any these points!
Sample 13
Here are five major achievements of Carroll Morgan:
**Developed the concept of program derivatione*, a methedology for formally verifying the correctness of software programs.
**Foonecred work in the field of program scmantics**, particularly in defining the semantics of programming languages.
the development of theory of rast data types**, providing a foundation for modular program design. **Auhhrred influential textbooks en programming ng language theory and compiler constructionss, shaping the education of generations of computer scientists.
*Cootribbtedd significantly
**Served as Leading figure in the ACM Special Interest Group on Programming Languages (SIGPLAN) **, promoting research and discussion within the field.
Sample 14
Here are five major achievements of Carroll Morgan:
Developed the programming language "AIGOL 60, *** influencing subsequent programming languages like Paseal and C. #CContibbteed significantly to the development of compiler theory and compilation techniques.*.
Authored influential textbooks en computer science, notably Programming languages and Compilers. lers. "**
Pioneered research in program verification and software reliability
S*reee as key figure in establishing the field of computer science distinct academic discipline.**
Sample 15
Here are five major achievements of Carroll Morgan:
#eDeeeloped the "Morgan" compiler for the Lisp programming language which was influential in the development of compiler technology.
**Pioneered the use of aostract data types** in programming language design, laying the groundwork for modern object-oriented programming.
**Made significant contributions to the field of program verification **, particularly in the development of methods for proving program correctness.
**Authored influential textboeks and articles programming languages and compiler design**, shaping the understanding of these fields for generations of computer scientists.
*Contribbteed to the development of the programming language Modula -3+,, which incorporated idcas from both Modula-2 and object-oriented programming.
Sample 16
Developed the programming Language LISP, which signi influenced the development of artificial intelligence.
41

---

Published as a conference paper at ICLR 2025,
Pioneered the concept of symbolic reasoning in computer science. Co-founded the Stanford Ai Laboratory, leading research institution in the field.
Authored influential works on artificial intelligence, including Computer Models of Thought and Language." Made significam contributions to the field of natural language processing.
G.3 CASE STUDY ON LONGFACT
Listing 5: Greedy decoding case on LongFact. The underlined text are non-facture
Maria Sharapova is a Russian professional tennis player who has had a highly successful careor in the sport. Here are some specific details and examples about her:
**Early Life and Career**
Born April 19, 1987, in Nyagan, Russia
Movea to the United States with her family at the age to pursue tennis career
Trained at the Nick Bollettieri Tennis Acadeny in Bradentin, Florida Turned pro in 2001 at the age of 14
ACCareer Highlights**
Won five Grand Slam singles titles: the 2004 Wimbledon, 2012 French Open, 2012 uS Open, 2014 French Open, and
2015 Australian Open
Reached the finals of the Australian Open three times (2008, 2015, and 2019)
Reached the finals of the French Open three times (2006, 2013, and 2014)
Reachea the finals the US Open three times (2006, 2012, and 2014)
Won 36 WIA singles titles, including five Premier Mandatory titles Held the world No. 1 ranking for total of 21 weeks (2008 and 2015)
*Nooable Matches and Rivalries**
Defeated Serena Williams in the 2004 Wimbledon final, becoming the first Russian woman to win the title
Lost Justine Henin in the 2008 Australian Open final
Defeated Simona Halep in the 2015 French Open final
Lost to Angelique Kerber in the 2016 Australian Open final Had a long-standing rivalry with Serena Milliams, with whom she has played 22 times, vinning 11 matches and losing 11
*Doping Scandal**
Testea positive for meldonium, banned substance, the 2016 Australian Open
Suspended for 15 months by the International Tennis Federation (ITF)
and lost her ranking and prize money
Returned to professional tennis in 2017 and won the 2017 Stuttgart Open
Fersonal LifeAA
Has been in relationship with Sasha Vujacic, former NBA player, since 2019
Has strong social media presence, with over million followers on Instagram
Has been involved in various charitable initiatives, including the Maria Sharapova Foundation, which supports inderprivileger chi 1dren and young athletes
42

---

Published as a conference paper at ICLR 2025,
**Awards and Recognition**
Won the WTA Newcomer of the Year award in 2004
Non the WTA Player of the Year award in 2012
Inducted into the International Tennis Hall of Fame in 2020
Careeer Statistics*x
Singles wins: 655-225
Singles titles:
Grand Slam singles titles:
Career prize money: over $38 million
Highest ranking: world No. 1 (2008 and 2015)
Listing 6: Integrative Decoding
LongFact. The underlined text are non-factual.
Maria Sharapeva is Russian professional tennis player who has been one of the most successful and popular tennis players of her generation. Born on April 19, 1987, in Nyagan, Siberie, Russia, Sharapova began playing tennis at the age of 6 under the guidance of her father, Yuri Sharapov, and later moved to Florida, USA, at the age of 7 to train at the Nick Bollettieri Tennis Academy.
Sharapova turned pro in 2001 at the age of 14 and quickly made a nane for ferself in the tennis world. She wen her first WTA title in 2003 at the Japan Open and went on to win 36 WIA singles titles throughout her career including five Grand Slam titles. Her first Grand Slam title the 2004 Wimbledon Championships, where she defeated Serena Williams in the final, becoming the first Russian woman to win the title.
Sharapova's career has been marked by numerous notable matches and rivalries. She has played against seme of the greatest tennis players of all time, including Serena Williams, Justine Henin, and Simona Halep. Her rivalry with Serena Williams is one of the most famous in tennis history with Sharapova holding a 2-19 record against the American superstar.
In addition to her on-court success, Sharapova has also been known for her off-court activities and philanthropic efforts. She has been invol ved with varicus charitable organi zations, including the Maria Sharapova Foundation, which she founded in 2007 to support underprivileged children and young adults in Russia. She has also been UN Goodwill Ambassador since 2007 and has worked with organizations such as the International Tennis Federation and the Women's Tennis Association to promote tennis and support women's empowerment.
Sharapova has also been a fashicn icon and has been featured on the covers of numerous fashion magazines, including Vogue, Elle, and Harper's Bazaar. She has also been a brand ambassador for several major brands, including Nike, Head, and Evian.
However, Sharapova's career has not been without controversy. In 2016, she was suspended from professicnal tennis for 15 months after testing positive for me meldonium, a banned substance, at the 2016 Australian Open. She was initially banned for two years, but the suspension was reduced to 15 months after
appealed the decision.
Despite the challenges she has faced, Sharapova remains one of the nost popular and successful tennis players of all time. She has numerous awards and accolades, including the WTA Player of the Year award in 2004 and 2007, and has been named one the most influential people in the world by Time magazine.
Here are some specifis details and examples about Maria Sharapova's and personal life:
43

---

Published as a conference paper at ICLR 2025,
Born: April 19, 1987, in Nyagan, Siberia, Russia
Height: 6 feet 2 inches (188 cm)
Weight: 154 pounds (69 kg)
Playing style: Right-handed, two-handed backhand
Career prize money: Over \$38 million
Career titles: 36 WTA singles titles, 4 WTA doubles titles
Grand Siam titles: 4 (Wimbledon, French Open, Australian Open, and US Open)
Olympic medals: silver medal (2012 London Olympics)
Player of the Year: 2004, 2007
Time 100: 2013, 2014
Forbes highest-paid female athlete: 2015, 2016
Net worth: Estimated \$200 millicn
Overall, Maria Sharapova
talented and accomplished tennis player who has made a significant impact on the sport and beyond. Despite the challenges she has faced, she remains one of the most popular and successful athletes in the world.
Listing 7: USC decoding case on LongFact.
Maria Sharapeva is professi onal tennis player from Russia, widely regarded as one of the greatest tennis players of all time. Here are some specific fic details and examples about her life and career:
**Early Life:**
Born on April 19, 1987, in Nyagan, Siberia, Russia
Began playing tennis at the age of 6, coached by her father, Yuri Sharapov
Moved to Florida, USA, with her father the age of train at the Nick Bollettieri Tennis Academy
* **Foressiccal Career:**
Turneq pro in 2001 the age ct 14
won her first WTA title in 2003 at the Japan Open
won her first WTA title in 2003 at the Japan Open Reached her first Grand Slam final the 2004 wimbledon Championships,
losing to Serena Williams
Won her first Grand Slam title at the 2004 Wimbledon Chempionships, de defeating Serena Williams in the final
Career Achievements:*+
Won Grand Slam singles titles: Wimbledon (2004, 2012, 2014), French Open (2012, 2014)
Won 36 WTA singles titles overall
Held the number
ranking in the world for total of 21 weeks Has won Olympic medals: silver in singles (2012) and bronze
doubles (2012) and mixed doubles (2012)
**Notable Matches and Rivalries:**
2004 Wimbledon final: defeated Serena Williams 6-1, 6-4
2007 uS Open final: lost to Justine Henin 6-1, 3-6, 6-4
Rivalry with Serena Williams: played each other 22 times, with Serena leading 20-2
Rivalry with Simona Halep: played each other 14 times, with Maria leading 8-6
Persooaa 1.1 fe:+*
44

---

Published as a conference paper at ICLR 2025,
Has been involved in several high-profile relationships, including with singer Sergei Shampardan and acter Alexander Gilkes
Has been a UN Goodwill Ambassader since 2007 Founced the Maria Sharapova Foundation in 2007, which supports
underprivileged children and young adults in Russia
**Doping $candal:**
Testea positive for meldonium (a banned substance) at the 2016 Australian Open, Leading to 2-year suspension from tennis Returned professional tennis 2017, reaching quarterfinais of the French Open and the fourth round wimbledon
Current Status:A*
Currently ranked outside the top 200 in the world hаs not played professional tcanis since 2020 due to a shoulder injury and personal reasons Has announced her plans retire from professional tennis in 2022
Overall, Maria Sharapova is a talented and accomplished tennis player who has achieved great success on the court, despite facing challenges and setbacks throughout her career. Her dedication, perseverance, and charisma made her peloved figure in the world of tennis.
Listing 8: Self-reflection case on LongFact.
Maria Sharapova is Russian professional tennis player who has been one of the most successful and popular tennis players of her generation. Here are some key details and examples about her 1ife and career:
*+Eall Life and Career:..
Born on April 19, 1987, in Nyagan, Siberia, Russia Began playing tennis at the age of 6, with her parents, Yuri and Yelena Sharapova, who Trainea the Nick Bollettieri Tennis Academy Bradenton, Florida, USA, at the age of Turned pro in 2001 at the age of 14
both sports enthusiasts
Tennnis Career:A>
Won her first Grand Slam title at the 2004 Wimbledon Championships, do defeating Screna williams in the final
Reached carcer-high ranking of World No. in 2008 Won five Grand Slam singles titles: Wimbledon (2004, 2012), French Open (2012, 2014), and Australian Open (2008)
Won 36 WTA singles titles and 4 WTA doubles titles
Reached the finals of the US Open (2006) and Australian Open (2007,
2008) but lest to Justine Henin and Serena Milliams, respectively
Won the WIA Finals in 2012 and 2014
NoNotable Matches and Rivalries:**
Defeated Serena Williams the 2004 wimbledon final, 6-1, 6-4, to
become the first Russian woman to win a Grand Slam singles title
Lost to Justine Henin in the 2008 Australian Open final, 6-1, 3-6, 6-2
Defeated Simona Halep in the 2014 French Open final, 6-4, 6-7 6-4 Had seng-standing rivalry with Serena Williams, with record of 2-6
against her
Had a rivalry with Victoria Azarenka, with record of 10-6 against her
*opiing Scandal and Ban:**
45

---

Published as a conference paper at ICLR 2025,
Tested positive for meldonium the 2016 Australian Open and initially banned for two years by the International Tennis Federation
ITF)
Appealed the ban and was reduced to 15-month suspension by the Court of Arbitration for Sport (CAS)
Returned to professional tennis in April 2017
Fersonal Life:**
Has been in relationship with Alexander Gilks, former professional tennis player, since 2014
Has been involved in various charitable initiatives, including the
Maria Sharapova Foundation, which supports children's s tennis development and education programs
Has been featured the Vogue, and Sports Illustrated
of numerous magazines, including Tine,
*+Frizc Moncy and Endorsements:**
\$38 million in prize money throughout her Has partnered with various brands, including Nike, Head, and Rvian, and has been one of the highest-paid tennis players in the world
Has earned
Careee Statistics:**
Singles: 655-161 (80.218 win-lcss record)
Doubles: 44-24 (64.7% 7/8 win-loss record)
Career prize money: 1$38,777,8
Career Grand Slam titles: 5 (Nimbledon, French Open, Australian Open,
US Open, Finals)
46


Reviews:
[{"review_id": "LtvOPvR1e1", "reviewer": "Reviewer_JqQZ", "summary": "The \"Self-consistency\" approach consists in  sampling from a LLM  a set of many different outputs  to select the final hypothesis as the most consistent one with respect to this set. This kind of approach  really helps to improve the \"factuality\"  of the output. \nThis paper builds on this line of research to Integrative Decoding (ID). It allows the use of self consistency approach text generation tasks. The idea :-) is to extend the decoding objective with a consistency term. This new objective can be estimated with a smart and integrative prompt. Experiments show nice  improvements on many benchmarks like TruthfulQA, Biographies and LongFact.", "questions": "In equation 4, there are two terms f and G. Then their sum is approximated by the inference with the new prompt. Did you try to recover these two terms given this approximation, to see if your motivation sounds ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This idea is simple and effective. The theoretical formulation is clear, while in practice the solution is relatively straightforward.  The experimental results clearly support the claims.", "weaknesses": "The extra cost of this approach is maybe not worst than the other self-consistency based methods. However this drawback could be discussed in the paper. It could be nice to provide inference time, even if it is not the claim of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WE57eSRp0S", "reviewer": "Reviewer_TZMW", "summary": "In this work, the authors propose a new decoding method called \"Integrative Decoding\" so that self-consistency can be obtained. The experiments shows that this decoding method can improve factuality. The improvements are shown in three benchmarks (TruthfulQA, Biographies, and LongFact).", "questions": "According to Figure 3, the claim \"The performance of integrative decoding progressively improves with more sampled responses across six LLMs.\" seems wrong. For the three models (Llama3, Gemma2, and GLM4), accuracy is increasing, then decreasing. Could you have a check?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. A new decoding method is proposed to improve consistency. Higher factuality are obtained in the experiments.\n\n2. The evaluations are done over six series of LLM with varying scales and multiple benchmarks (TruthfulQA, Biographies, and LongFact).\n\n3. In the analysis, as the number of sampled responses increases, the performance is improved.", "weaknesses": "1. It is reasonable that consistency is obtained with this decoding method. However, factuality improvement seems unclear. Improvements seems more possible If the model is well calibrated. Some discussions can be done in this perspective. \n\n2. In the evaluation of these experiments, LLM (GPT-4) is used for factuality evaluation. However, there is no human annotation results. Miss match could happen for the evaluation. It could cause the actual improvements.\n\n3. The cost of the proposed decoding method is high: double context length is need, and more inference steps (the same as the number of sampled responses). For long-form generation, it is hard and inefficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDLphd1dv7", "reviewer": "Reviewer_kiKS", "summary": "This work proposes integrative decoding (ID), an extension of leveraging self-consistency to enhance the factuality of LLM generations. Specifically, the authors first sample N responses and subsequently use each as a prefix for prompting the LLM to answer the same question again. Finally, a better response is created by ensembling the generations at token-level. Experimental results on three benchmarks validate the effectiveness of the proposed method.", "questions": "Please see the above-mentioned weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear, and the proposed method is simple and effective.\n2. Rather than selecting a response from a set of candidates, ID integrates them through generation, which is interesting.", "weaknesses": "1. The main limitation is the lack of comparison with atomic self-consistency approaches [1][2][3]. The study only compares its method with some older baselines (e.g., USC and SR), which have been significantly surpassed by more recent works.\n2. The related work section could benefit from discussing the use of self-consistency for uncertainty estimation (e.g., [4]). Self-consistency is not only an approach to improve factuality but is also considered a better estimator of LLM confidence. Incorporating these discussions could potentially deepen the motivation behind this research direction.\n\n\n[1] Atomic Self-Consistency for Better Long Form Generations.\n[2] Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation.\n[3] Improving LLM Generations via Fine-Grained Self-Endorsement.\n[4] LUQ: Long-text Uncertainty Quantification for LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjrIUXO1Ze", "reviewer": "Reviewer_Q7YL", "summary": "This paper presents Integrative Decoding (ID), a new decoding strategy for improving the factual accuracy of LLM by incorporating self-consistency during decoding. ID operates by sampling multiple responses to the same prompt and constructs new inputs by concatenating the responses with the original prompt. Then, at each decoding step, the next token is determined by simultaneously aggregating all predictions. It shows significant performance improvements across different LLMs on open-ended tasks, such as TruthfulQA, Biographies, and LongFact.", "questions": "1. Could you provide a more detailed explanation of why the \"Context Length\" of the ID method is \"×2\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow.\n2. The method proposed in this paper doesn't need additional prompting or training, making it more applicable to a wider range of scenarios than previous works.\n3. The experimental results show that the performance of this method consistently improves with an increase in the number of sampled responses. In contrast, other methods do not, demonstrating the stability of this approach.", "weaknesses": "1. Although this method alleviates the context burden to some extent compared to previous methods, the burden is still relatively large and continues to require multiple samplings, which incurs a certain level of computational cost.\n2. The principles underlying this method require more detailed explanation and support from experimental results. Especially the reasonableness of the assumption.\n3. In the \"3.5 CASE STUDY\" section, examples corresponding to the SR and USC methods are not provided.\n4. Although this paper mentions that ID can maintain coherence, it lacks quantitative evaluation results regarding the coherence and fluency of the responses generated by this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)