Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
Authors: Maojia Song
Shang Hong Sim
Rishabh Bhardwaj
Hai Leong Chieu
Navonil Majumder
Soujanya Poria
Venue: ICLR 2025 Oral (2025)
Keywords: Large Language Models
Trustworthiness
Hallucinations
Retrieval Augmented Generation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: How to better evaluate and make LLM better for RAG task

Abstract: LLMs are an integral component of retrieval-augmented generation (RAG) systems. While many studies focus on evaluating the overall quality of end-to-end RAG systems, there is a gap in understanding the appropriateness of LLMs for the RAG task. To address this, we introduce Trust-Score, a holistic metric that evaluates the trustworthiness of LLMs within the RAG framework. Our results show that various prompting methods, such as in-context learning, fail to effectively adapt LLMs to the RAG task as measured by Trust-Score. Consequently, we propose Trust-Align, a method to align LLMs for improved Trust-Score performance. 26 out of 27 models aligned using Trust-Align substantially outperform competitive baselines on ASQA, QAMPARI, and ELI5. Specifically, in LLaMA-3-8b, Trust-Align outperforms FRONT on ASQA (↑12.56), QAMPARI (↑36.04), and ELI5 (↑17.69). Trust-Align also significantly enhances models’ ability to correctly refuse and provide quality citations. We also demonstrate the effectiveness of Trust-Align across different open-weight models, including the LLaMA series (1b to 8b), Qwen-2.5 series (0.5b to 7b), and Phi3.5 (3.8b). We release our code at https://github.com/declare-lab/trust-align.

Published as a conference paper at ICLR 2025
MEASURING AND ENHANCING TRUSTWORTHINESS OF LLMS IN RAG THROUGH GROUNDED ATTRIBUTIONS AND LEARNING TO REFUSE
Maojia Song", Shang Hong Sim', Rishabh Bhardwaj Singapore University of Technology and Design [maojia.sonn, shanghong.sim, rishabb.bhardwa) emymaii sutd. edu, .g
Hai Leong Chieu DSO National Laboratories chai maileecnedoo org. sg
Navonil Majumder, Soujanya Poria
Singapore University of Technology and Design fnavonil 1 aaaaiimmaaaaaaa Lmajumder, sporia edu.
ABSTRACT
LLMs are an integral component of retrieval-augmennte generation (RAG) sys- tems. While many studies focus on evaluating the overall quality of end-to-end RAG systems, there is gap in understanding the appropriateness of LLMs for the RAG task. To address this, we introduce TRUST-SCORE a holistic metric that evaluates the trustworthiness of LLMs within the RAG framework. Our results show that various prompting methods, such as in-context learning, fail to effee- tively adapt LLMss the RAG task measured by TRUST-SCOREF Consequently, we propose TRUST-ALIGGT a method to align L.LSs for improved TRUST-SCORE performance. 26 out of 27 models aligned using TRUST-ALE ALIGN substantially out- perform competitive baselines on ASQA, QAMPARI, and ELI5. Specifically, in LLaMA-3-8b, TRUST-ALIGN outperforms FRONT on ASQA (f2255), QAM- PARI (136.04), and ELIS (117.69). TRUST-ALIGN also significantly enhances models' ability correctly refuse and provide quality citations. We also demon- strate the effectiveness of TRUST-ALIGH across different open-weight models, including the LLaMA series (Ib to 8b), Qwen-2.5 series (0.5b to 7b), and Phi3.5 (3.8b). We release our code at https / gi github. com/ declare lab/ trust-align
INTRODUCTION
LLMs are widely used for information retrieval but often produce hallucinations -fctuuull incor- rect yet convincing responses (Ji et al., 2023), undermining their reliability. A common mitigation Retrieval-Augment Generation (RAG), which integrates external knowledge to improve correct token generation, reducing perplexity (Khandelwal et al., 2019) and enhancing downstream tasks like machine translation (Zheng et al., 2021) and classification (Bhardwaj et al., 2023). Connect- ing LLMs to external documents via retrieval also improves response quality (Shuster et al., 2021; Béchard & Ayala, 2024), further enhanced by attribution mechanisms (Gao et al., 2023b; Hsu al., 2024).
In this paper, we investigate LLMs' ability to ground responses in provided documents instead of relying on their parametric knowledge from training. A response considered grounded if it cor- rectly answers using only the attached documents, with in-text citations supporting its claims. Key aspects include LLMss refusal capability-whethee they abstain from answering when documents lack sufficient information. Additionally, we analyze their overall tendency to answer, the fraction of claims grounded in documents, and whether cited sources substantiate generated statements.
To comprehensively understand LLMs' groundedness, we propose new metric TRUST-SCORE. assesses LLM across multiple dimensions: 1) The ability to discern which questions can be
Equul contribution.

---

Published as a conference paper at ICLR 2025
answered or refused based the provided documents (Grounded Refusals); 2) The correctness of LLM response for the answerable questions: 3) The extent to which generated statements are supported by the corresponding citations; and 4) The relevance of the citations to the statements. Unlike existing metrics that primarily assess the overall performance of RAG systems (Gao et al., (023b)-hhere weak retriever significantly decrease the scores TRines-TRRUT-SCORRE is designed to specifically measure the LLM's performance within RAG setup, isolating it from the influence of retrieval quality.
Our investigation in Section 6.1 shows that many state-of-the-art systems, including GPT-4 and Claude-3.5 3.5-nnnnet, heavily rely on their parametric knowledge to answer questions (OpenAI, 2023; Authropic. 2024). This reliance limits their suitability for RAG tasks, where models should base re- sponses solely on the provided documents, resulting in a low TRUST-SCORF. Additionally, prompt- ing approaches intended to enhance model groundability have proven ineffective, as models become overly sensitive to the prompt, leading to exaggerated refusals or excessive responsiveness shown in Appendix F.4. To enhance the groundedness of LLMs. i.e., achieve higher TRUST-SCORE, we propose an alignment method, TRUST-ALIGN This approach first constructs an alignment dataset consisting of 19K questions, documents, positive (preferred) responses, and negative (unpreferred) responses. The dataset covers a range of LLM errors Inaccurate Answers, Over- wver-Responsiveness Excessive Refusal, Over-Citation, and Improper Citation. We regard these errors as LLM hallucina- tions within an RAG framework.
Evaluations on the benchmark datasets ASQA, QAMPARI, and ELI5 show that models trained with TRUST-ALIGN outperform the competitive baselines on TRUST-SCORE in 26 out of 27 model fam- ily and dataset configurations. Notably, in LLaMA-3-8b, TRUST-ALIGN achieves substantial im- provements over Huang et al. (2024b) FRONT, a leading baseline, with respective gains of 12.56% (ASQA), 36.04% (QAMPARI). and 17.69% (ELI5). Additionally, TRUST-ALIGH substantially en- hances the ability of models to correctly refuse or provide grounded answers in all 27 model family and dataset configurations, with LLLMA-3-8h showing increases of 23.87%, 47.95%, and 45.77% correct refusals compared to FRONT. Citation groundedness scores also improved in 24 out of 27 model family and dataset configurations, with notable increases of 22%, 38.35%, and 5.55% in LLaMA-3-8h compared to FRONT. Due to the gamification of the metric, where parametric knowl- edge can artificially inflate the scores, we notice mixed results on answer correctness scores. Specif- ically, we observe notable increase in answer correctness scores for all models in QAMPARI, 5/9 models in ELI5, and 2/9 models for ASQA.
Our key contributions this work follows:
We study LLM groundedness problem, where model model responses should be derived from retrieved documents (external memory) rather than the parametric knowledge (knowledge stored in model parameters).
To measure LLM's groundedness under RAG, we introduce TRUST-SCORE, a holistic metric for quantifying LLM's grounding errors.
We propose TRUST-ALIGN, an alignment approach designed to improve the trustworthiness of LLMs in RAG (Figure 2). It first creates an alignment dataset of 19K samples with paired posi- tive and negative responses, followed by aligning the model using direct preference optimization (DPO) (Rafailov et al., 2024b).
Comparison with existing approaches. Current evaluations of RAG focus on the overall system performance (Gao et al., 2023b; Xu et al., 2024), conflating the effects of retriever quality and LLM performance in the metric scores (Fan et al., 2024). This highlights the need for new ways to mea- sure LLM effectiveness in RAG systems without the influence of the retriever. The work by Thakur et al. (2024) is closest to ours, as it analyzes the refusal capabilities of LLMs in a RAG context but lacks holistic evaluation, as does not account for both response and citation groundedness. On the other hand, Ye et al. (2024); Hsu et al. (2024); Huang et al. (2024b) propose frameworks to improve LLM response groundedness but overlook refusal behaviors in their metrics. Ignoring refusal be- haviors, retriever influence, citation and answer groundedness weakens the ability of current metrics to effectively measure LLM performance in RAG. TRUST-SCORE comprehensively evaluates LLM performance, including refusal, citation, and answer groundedness, while TRUST-ALIGH creates corresponding alignment dataset, making the metric and approach more unique and holistic for LLM evaluations and alignment in RAG. A more detailed comparison can be found in Appendix C.
2

---

Published as a conference paper at ICLR 2025
2 PROBLEM DESCRIPTION
2.1 TASK SETUP
Given a question g and a set of retrieved documents as input, the LLM is instructed to generate response S which consists of set of citation- statements f each statement 8 follows a set inline citations C, = fc..1. G.2..... referring to the documents in D. If D is not sufficient to answer q, the gold response would be a refusal statement', such as, "I apologize. but I couldn't find answer to your question in the search results". Otherwise, the response would follow the pattern: "statementl [1][2] statement2 [3]" where [1][2] and [3] denote the enumeration of documents that supports each statement respectively.
Trust-Score
Trust Score
Flan Flac Flas)
Response Truthfulness
Attribution Groundedness
Grounded Refusals (F1ox)
Answer Correctness (F1
Grounded Citations (Flgc)
2Pa Mag Pae
2P. Rui
Floy
Bag
Rut
Pae
Puw
Rag
AC4
Ph HaLLn)
Pay
Infurmmdy All
Mas
ADAI A.
OR Ml... Her (..)))
Set extnared questoro sie inswwrabiin se quecttii
A31 RC arouendda pections cnviiee only conoenetttt
Number caventi
ACE
Ag
Chation prediinnfoccee txtann
Figure 1: TRUST-SCORE calculation shown as a computational graph.
2.2 ON ANSWERABILITY OF A QUESTION
To label if response should be refusal consist of claims, define the notion of answerability. A question is considered answerable if D contains sufficient information to answer Formally, we label question as answerable the gold claims; otherwise, is unanswerable and thus should result in ground truth refusal. A refusal response contains no claims or citations but provides a generic message conveying the LLM's inability to respond to 3.
subset of the retrieved documents entails at least one of
2.3 HALLUCINATION IN LLM IN RAG
We define an LLM's response as grounded when it correctly answers question using only the information in the documents, and the response can be inferred from the inline citations to those documents. When response is not grounded, it is considered case of hallucination. We define hallucination as an erroneous LLM response, categorized into five types: (1) Inaccurate Answer - The generated statements 5 fail to cover the claims in the gold response, (2) Over-Responsiveness The model answers a question that should result in a refusal, (3) Excessive Refusal The model refuses to answer question that is answerable, (4) Overcitation The model generates redundant citations, and (5) Improper Citation - The citations provided do not support the statement. Next, we introduce comprehensive metric to concretely measure hallucinations in L.LMs, i.e., assess LLM's groundedness or trustworthiness*.
'There are many applications where I.I.M parametric knowledge is expected and retrieved documents serve to improve the I.LM's response. However, this paper we study the problem of complete grounded- ness-i.e, all claims should be documents derivable, making this an IR task. IIn this paper, we L.I groundedness and trustworthiness interchangeably in the context of RAG.
3

---

Published as a conference paper at ICLR 2025
3 METRICS FOR LLM-IN-RAG
Given question and the corresponding ground truth response Ag [agl...... ................................ agn consisting of gold claims, we define the claims obtainable from the provided documents as Ap = {ddi, adn) and the claims generated in the response as Ar (Arl,.... Orn We aim to measure two aspects of an LLM in RAG: 1) the correctness of the generated claims (Response Truthfulness); and 2) the correctness of citations generated (Attribution Groundedness).
Insufficiency of the existing metrics. Gao al. (2023b) measure Response Truthfulness by first computing the per-sample Answer Correctness recall (ACig) score for gold claims Ag. disregarding how many of these claims are obtainable from D. This is followed by averaging the recall scores across samples to obtain a single score for the dataset. This method introduces inconsistencies: models that rely on parametric knowledge (Mp) may generate gold claims not found in D, leading to artificially inflated recall value. In contrast, an ideal LLM (M,) would rely solely on D to generate responses (a desired trait) and would be constrained by upper recall limit of |AGdAal |Aal
(1) Recall Consolidation; Since the measurement range depends on the claims present in D, it is infeasible to provide a consistent, consolidated ACreg score across the dataset, (2) Recall Gamification: Mp may have a higher upper limit on ACreg (up to 1) because they can generate gold claims not present in D (an undesirable trait), unlike M, that depend entirely on D.
which varies depending on the question. This approach presents two key problems:
Answer Calibration. To address the challenges of recall consolidation and gamification in exist- ing evaluation metrics, propose new metrics that measure sample-wisg wise recall score based on the fraction of gold claims ontainable from D. Specifically, this involves computing |Ag AApll which measures the Answer Correctness (AC) recall after calibrating the gold claims. This approach sets maximum recall limit of for all models. For dataset-wide scoring, we consolidate per-sample AC scores using two methods: PAC: The average AC score across samples answered by the LLM, i.e., samples where Ar # 0, reflecting precision oriented perspective; 2) RAC: The average AC score across samples that are answerable, i.e., samples where AG n AD & 0, reflecting a recall oriented perspective'. These metrics, illustrated in Fig. 1, are then combined into a single score, FLAC. which serves as a comprehensive measure of how well the LLM grounds its claims on the document D. This combined metric not only facilitates the consolidation of recall but also addresses issues related to recall gamification.
Scoring refusals. An important capability of an LLM in RAG is its ability to identify when a response is unanswerable based on the provided documents D. To measure this, we introduce metric called Grounded Refusals. This metric evaluates the model's s refusal performance by calcu- lating dataset-wide precision and recall for both ground-truth answerable cases and refusals. These values are then combined into their respective F1 scores, Fl for refusals and Flans for answerable cases. The final score, FLGR, the average of these two Fl scores, shown in Figure 1.
Measuring attribution groundedness. While Response Truthfulness metrics like FlAC and FLGR evaluate the quality of generated claims, it is equally important to measure how well these statements are supported by relevant citations what we call Attribution Groundedness. To this end, we adopt two sub-metrics from (Gao et al., 2023b): Citation Recall (Ret)) and Citation Precision (Pat).. To compute Reite- we first determine a generated statement 8; is supported by its cited documents using an NLI model thus obtaining sample-wise recall scores Rt Then we take the mean across all samples to obtain the final Reite score (Figure 1). To compute Pette- we first score each citation C.j of a statement s, followed by computing the average across citations in response S (sample-wise score). The dataset-wide citation score is computed by averaging the citation scores across all the samples. To quantify the Groundedness of Citations, compute FlGc. the harmonic mean of P ciee and Rete-- A detailed breakdown of this metric is provided in Appendix D and Figure 1.
Thus, define
metric TRUST-SCORE (F1cR FLAC Flac).
NNotably, both PAC and RAC sum over samples that are hoth answered and answerable, differing primarily in their normalization values.
An ILJ model checks if the cited document entails the statement.
4

---

Published as a conference paper at ICLR 2025
Responsiveness. To measure the answering tendency of an LLM, we define Responsiveness. It is the fraction of answered questions, denoted by the Answered Ratio (AR %), which is calculated as AR % ,a, + A,1. gl and Al are the number of answered, answerable, and unanswer- able questions respectively. A model is expected to show a high AR% for answerable questions and a low AR% % for unanswerable ones, with the scores expected to align with the dataset distribution.
A-l
i Mairipncio top 300 sonis Wilipedi,,
Seed Prompt Curation
Posiiie Answer Generation
AGOA
Send
Bui
Man
GPT4
Faredbon Hare
M
Decs merte
Nagativa Generation
Answerability Labelling
Calca
Hower coe
Jestac
Supary and resuring
Dock
Cddccan3
Augmented Prompt Curation
Aug det
Alignment
Coeatirn
Ovenion
Frstive Airwer
Drashhdeure
Figure 2:
Overview of the TRUST- Left: The curation of both seed and augmented prompts (Q- D pairs) and an example of the answerability labeling process during the retrieval stage. Right: The response paired data generation process. First, we obtain positive answers and then seleet hard negative answers. Finally, we align our model via DPO.
4 THE TRUST-ALIGN DATASET
To align LLMs towards trustworthiness, we propose new approach, TRUST-ALIGN. The approach constructs an LLM trustworthiness alignment dataset, where each sample in the dataset consists of question q, a set of retrieved documents D, and a pair of positive (preferred) and negative (un- preferred) responses (r r.). The positive response corresponds to an answer that encompasses expected gold claims for q and corresponding citations referring to the documents. If D not suf- ficient to answer q, r+ is assigned a refusal response, while r- is its non-refusal counterpart. We build the dataset in multiple steps: 1) Obtain a set of high-quality and diverse questions, 2) Obtain documents for each question, 3) Augmenting (g, D) pairs that cover diverse hallucination types, 4) Construct positive responses entailing gold claims, and 5) Construct negative (unpreferred) re- sponses by prompting fine-tuned model and observing its ballucinations. We relegate fine-grainee details about the dataset to Figure and Appendix E.
Collecting quality questions. The dataset construction begins by collecting a set of high-qualit and diverse questions from the training splits of ASQA, QAMPARI, and ELI5, referred to as seed samples. We first divide the questions into & clusters and use Mixtral-8x7B to assign each a quality score from to 7, based on how difficult they are to answer without additional information. Clusters with scores of 4 or higher selected. Next, we sample questions from the clusters of each dataset to construct approximately 10K questions in the seed set.
Collecting D's. Next, collect relevant documents for each question in the seed set by querying Wikipedia and Common Crawl, retrieving the top 100 documents. We filter out seed questions where relevant documents not retrieved. We then identify documents that perform as well as the full
5

---

Published as a conference paper at ICLR 2025
100 in terms of EM recall, referring to these oracle documents for question q. Gold claims for each q are sourced from the respective datasets.
Augmenting (q. set. Using the questions and oracle documents, we create diverse samples (i.e., varying combinations of relevant and irrelevant documents) to trigger multiple hallucinations from LLMs (Section 2.3). The document order is shuffled avoid citation bias. To construct unanswer- able questions, select documents similar to those entailing gold claims but still irrelevant to q- This process results in approximately 70K question-document pairs.
Obtaining r* and r- To generate preferred responses, by prompting GPT-4, we stitch together the gold claims and citations For unanswerable questions, we assign ground truth refusal re- sponse. obtain quality negative (unpreferred) responses, we fine-tune L.LMMA--77b on source datasets, creating Mase. Testing Maft the 70K dataset identified 40K responses with hallucinations. Table shows hallucination severity (e;) and frequency (w,). To obtain good neg- ative samples, first rank each of the 40K responses according to their severity score Eg., where Ey - E, ei We We then select the top 50%? of the corresponding samples for both answerable and
unanswerable responses. We perform DPO using this set of 19k samples to obtain the final aligned model.
EXPERIMENTAL SETUP
Models studied. To comprehensively mea- sure performance of open-source models, we perform TRUST-SCORE computations on vanilla and TRUST-ALIGNed version of range of ppenhweight models such as LLaMA series (LLaMA-2-7b,, LLaMA-2-1-- 1-bb, LLaMA-2-13b. etc.), Qwen series (Qwen-2.5-0.5b, Qwen-2.5- 7b, etc.) and Phi3.5-mini. See Appendix H.1
Table Fraction of each hallucination amongst all the observed hallucinations in Maft (40,985), with possi- blc overlap. shows the severity computation of each hallucination. Lendition if condition True otherwise See Fig. for the detailed breakdown of the three errors.
Hallucination Type (HT)
Frequency (u.)
Severity (e.)
Unwarranted Refusal
8,786
0.50
164,48.A,--))
for more details.
Over Responsiveness
13,067
0.50
Overcitation
12,656
0.34
C-
Improee Citation
9,592
0.26
1-CR
Evaluation datasets. We evaluate on the test-set of attributable factoid and long-form question-answering tasks from ASQA (Stelmakh et al., 2023), QAMPARI (Amouyal et al., 2023), and ELI5 (Fan al., 2019). Additionally, evaluations. For each question, we append the top retrieved documents. For ELI5 and ExpertQA, the ground truth answers are decomposed into three claims. The dataset statisties are detailed in Appendix H.2.
Inaccurate Claims
14,783
0.40
Fl
include ExpertQA (Malaviya et al., 2024) for OOD)
Baselines.
M trained with TRUST-ALIGE are compared against the following baselines:
ICL (Gao et al., 2023b): Prepends two demonstrations each query, consisting of an example query, top-5 retrieved documents, and an inline cited answer
PostCite (Gao al., 2023b): Generates uncited answer in closed-book setting, then retrieves most similar documents from top-5 documents using GTR for citations.
PostAttr (Gao et al., 2023b): Similar to POSTCITE, produces an uncited response in closed-bood setting, but uses the TRUE-NLI model to find the best matching citation among top-5 documents.
Self-RAG (Asai al., 2024): Trains the LLM to retrieve relevant documents demand using reflection tokens, enhancing generation quality. We evaluated the provided 7b and 13b model checkpoints from HF using the default settings.
FRONT (Huang et al., 2024b): Uses a fine- ane-grained attribution framework to improve grounding and citation quality. We followed the provided instructions to train 7b model for comparison.
'Clustering and document retrieval details are in Appendix E. "Prompt template can be found Table 23. "See Appendix F.8 for more details on this hyperparameter.
'lll models used are instruct tuned chat versions.

---

Published as a conference paper at ICLR 2025
6 RESULTS AND ANALYSIS
Table 2: LLaMA family evaluated on the ASQA, QAMPARI, and EL15 datasets. Best values within each family are highlighted ). AR % Answered Ratio in %;; F1AC Answer Correctness F1;
FIGR= == Grounded Refusals F1; Fl == Grounded Citations FI; TRUST TRUST-SCORE; Resp.
== Responsiveness; Att-Grd. := Attribution Groundedness.
ASQA (610 soveraalle 235 avgurcalle)
QAMPARI (29 avoerrable avsoecnalle
(207 snrreable 7V/ seromcalee
Made
Type
Besp.
Treatwothisess
Rei.
Trastvoriiins
Resp
Truainethisce
AIR
Truthdalizee
Ali-Gent
TRUST
AIR
Truthfulinen
Ali-Gnl.
TRUST
Truthfuline
Ali-Genl
Flazs
Flax
Flex
Flax
Flae
KCL
0.00
01:00
26.15
0.00
8.16
0.50
0.00
0.00
13.3 T3
0.50
0.20
45.71
0.00
15:51
1LMM-22 -7
PozAA
0.00
11:77
5.00
3.76
0.50
1.86
44.95
0.50
15:61
SeIRRAG
100.00
45.19
35.15
03:20
49.28
96.00
6.81
23.33
19.05
18.33
13.50
14.94
40.35
13.00
23.08
FRONT
100.00
35.15
08.90
55.16
100.00
17.27
22.78
24.36
21
100.00
11
17.15
52.70
30.51
ICL FooCiie
17.41
21.52
41,40
13.53
25. 15.58
26.50
0.44
19.57
0.00
20.00
45.40
19.97
54.81
4.73
26.50
1LMMN-2
50.51
2.21
45.3
1,53
1.88
100.00
0.00
22.78
8,05
10.2
75.60
2.27
35.05
0.72
13.58
-156
PocAAE
5.00
2,05
76.60
35.05
RAG
K11
M1.25
35.05
50.94
5:06
$2.25
19.30
5:32
$2.64
0.38
19.33 19-78
$5.40
12.80
27.10
5.35
15.00
0.59
12102
0.32
1.11
17.30
19:40
50:34
1.02
17.09
32
FRONT
48.22
60.29
50.33
98.60
24.54
1532
1581
97.20
20.76
30.19
22.35
TAUST-ALIGA (DPPO)
41.87
35.54
39.61
79:35
$5.87
20,00
27.22
67.02
00.02
68.19
9.60
13:20
59.35
05.21
40.25
Pre
47:26
31.03
56.50
22.94
30.00
5:34
6.51
07.50
4.46
$5.00
12-MA-12
Poe Alr
17:15
79:76
56.71
4.46
33 75
43.00
5:10
51.54
0.71
19.30
$1.80
15:05
25.14
0.55
14.75
-th
99.25
55.70
07.70
17.88
19.19
13.31
K11
1-48
5.01
25.55
56.50
28:36
3,90
5,22
45.01
20.24
24.90 02
01:00
0.20
44.23
0.00
14.34
PooCitt
T7.55
31.98
53.31
75.01
10.10
87.00
6.10
34.52
8.43
16.35
13.00
20.90
45
5.06
24.91
1LMMN-3
87.00
1.64
FRONT
66.14
56.67
13.59
22.74
99.50
18.95
17.85
27.38
TAUST-ALIGN (DPO)
36.45
$3.94
65.29
185.0
09.23
22.40
05 85
70.73
58.97
$4.95
15.50
20.81
63.57
$0.24
44.57
TRUST-ALIGN boosts trustworthiness over baseline methods. As shown in Table 2 and Ta- ble 3, TRUST-ALIGNed models demonstrate substantial improvements on TRUST-SCORE over the baselines in 26 out of 27 model family and dataset configurations. Specifically, with LLaMA-3-8b, TRUST-ALIGN outperforms FRONT by 12.56% (ASQA), 36. 56.04% (QAMPARI), and 17.69% (ELIS) on TRUST-SCORE. This suggests that TRUST-ALIGNed models are more capable of generating re- sponses grounded in the documents.
TRUST-ALIGN improves models' refusal capability. Across all 27 configurations, TRUST- ALIGN yields substantial improvements in FLGR- LLaMMA3-Sh, TRUST-ALIGN outperforms FRONT by 23.87% (ASQA), 47. 7.95% (QAMPARI), and 45.72% (FLI5). This indicates that TRUST- ALIGN substantially enhances models' ability to correctly refuse or provide answers.
TRUST-ALIGN enhances models' citation quality. Flae is substantially improved over base- lines in 24 out of 27 model family and dataset configurations after the application of TRUST-ALIGN Specifically, with LLaMA-3-S0 TRUST-ALIGN outperforms FRONT on Flae by 22.12% (ASQA), 38.35% (QAMPARI), and 5.5 5.55% (ELI5). This demonstrates that aligning with TRUST- ALIGN im- proves the model's 's ability to provide citations that sufficiently and precisely support claims.
TRUST-ALIGN has mixed effects on FLAC- We observe that applying TRUST-ALIGN yields notable increase in FlAC for QAMPART (9/9) but mixed performance on ELI5 (5/9) and ASQA (2/9). The mixed performance in ASQA and ELI5 can be explained by the composition of F1 which is derived from PAC and RaC (Eq. (9)).
Taking LLaMA-3.2-3b on ASQA as example (Appendix 1), TRUST-ALIGN models generally achieve higher PAC compared to baselines (54.63% for TRUST-ALIGN vs. 52.94% for FRONT) despite having a lower AR% (77.85% for TRUST-ALIGN that our models have a higher expected value for AC (per-sample AC recall), as the denominator depends on the number of answered questions. This trend is observed across models and datasets.
95.25% for FRONT). This suggests
7

---

Published as a conference paper at ICLR 2025
However, in ASQA and ELI5, our models underperform in FLAC due to the overwhelmingly ad- verse impact of RAC. The recall of answerable questions (Ran)) is lower for our model compared to baselines (89.02% for TRUST-ALICS a result, fewer terms are summed in the numerator of RAc. while the denominator remains constant (the number of answerable questions). This leads lower overall FC Ac score. To further analyze the baseline models' performance, we investigated how much of their answering ability relies on parametric knowledge versus document-based information (Section 6.1 and Appendix F.3).
vs. 98.69% for FRONT), which rarely refuse questions. As
Table 3: Qwen2.5 and Phi3.5 families evaluated on the three datasets.
QAMPARI
yosezernalle)
Made
Type
Regp
Trantesrtinns
Reg.
Trastorthiins
Ree.
Trasteorthinss
Truthfillees
TRIST
Truthfilless
TRUST
Truthflleess
TRUST
Flat
Flee
Flac
Flac
Floa
Floe
Flac
Flsa
Floc
47.19
11:40
2.45
50.b
12.71
62.30
33.14
46.00
8,14
90:81
2,55
31.54
17.00
1167
0.90
1803
27.30
12.55
1.
FRONT
100100
43.53
34:15
45.87
47.67
90.30
11.52
25.25
15900
10:85
99
13.74
17.39
37.05
19:6
TRUST-ALION (DPP)
71.84
50.99
61.25
32.40
17.00
15.76
21.70
13.08
00/791
22.72
96.52
50.55
41.24
35.00
15.60
17.76
PouCiie
73,73
16.36
$2.45
15:40
25.07
11.20
3.46
51.11
13.05
22.83
91.35
15.63
25.71
5.17
15:84
Quae-225 -1.5b
Poe Ale
71.33
16.3 56
92-35
6.65
24.03
11:50
5.66
51.11
3.07
15.54
01
15:63
26.71
0.60
14.33
FRONT
94.25
37.34
41:36
1585
17:37
19.57
15:39
31:30
24.85
TRUST-ALION (DPO)
ICT.
21.43
37.37 23
51.35 36
51.73
06.93
22.30
23.17
65.27
41:20
47.55
68:90
12
45.31
34.14
35.55
PooCho
8.76
9.15
30
10:54
18.0
11100
1100
41.31
0.00
13.77
21
45.49
7.50
75.93
FRONT
ARINN-ALION (EPP))
40.47
55.19
61.76
75.54
05.95
45.10
35.69
70:51
05.00
90:55
13.90
02.55
151.76
42.01
47.97
94
$4.34
75.46
62.91
15.10
26.02
53.67
59.35
52.00
37.13
44.15
Q -b
91,45
22
45.93
13.02
30.46
26.00
8,59
13:55
22.43
95.00
21
23.25
0.96
15.00
FRONT
$6.39
64.55
00.05
55.27
60.95
$4.70
17.02
42.65
24.48
26.12
57.60
27
54.14
56.61
45.34
TRUST-ALION (UPP))
$9.49
35.04
05:22
32.10
30111
70.88
$1.42
21.00
(63.700
47.02
45.04
ICI
63.09
93.30 11
51.95
47.64
75
70.20
11:91
43.50
12:76
27.09
90
37.17
31.14
31.01
Pou
23.00
14.95
41.35
0.40
21.02
75.90
3.57
42.36
4.49
16.81
84.50
20.50
30.81
4.67
19.65
M13.5
PaKAAE: FRONT
23.80
14.98
41.35
1.24
19.20
75,00
3.57
42.36
0.46
15:45
84.50
21 13226
30.81
0.68
17.58
41181
39.79
71.63
100.00
11.97
12.75
18.75
21.35
61.41
TRUST-ALION (DPP2)
66.35
52.21
64.30
35.16 16
26
10.10
35:42 13.95
$3.40
$4.90
24.90
23.30
67.62
47.42
06.14
TRUST-ALIGN generalizes across model families and sizes. Table 3 demonstrates that TRUST- ALIGN improves the models' TRUST-SCORE across various sizes and architectures. In small mod- els like 2.55-2.,-,.5b,
significantly outperforms ICL. baselines, achieving notable gains in ASQA (22.83% + 54.76%) Similarly, for larger models such as Qwen-2.5-7b. TRUST- ALIGN delivers substantial improvements, as seen in ASQA (62.91% + 68.28%), highlighting its scalability. The largest gains are observed in smaller models; for example, Phi3.5-mini shows markable improvements over ICL: 18.98% (ASQA), 31.90% (QAMPARI), and 14.51% (ELI5).
Models aligned with DPO generally outperform those trained with SFT. Table 4 shows that DPO models outperform SFT models on TRUST-SCORE in 26 out of 27 model family and dataset configurations. In LLaMA-3.2-38 3.b.b-bb, DPO yields substantial improvements on ASQA (6.70%). QAM- PARI (3.09%), and ELI5 (1.71%). Additionally, DPO models also attain substantially better Flae compared to SFT on 25 out of 27 configurations. with substantial improvements on ASQA (8.58%), QAMPARI (7.62%), and ELI5 (2.54%) for LLaMA-3.2-3b This highlights DPO's effectiveness in enhancing citation quality. While results FCAC and FlOR are mixed, DPO yields better overall TRUST-SCORE scores.
6.1 ANALYSIS
Data ablation. Table 5 shows that adding samples targeting each of the five hallucination types improves TRUST-SCORE by 1.50% (ASQA), 1.78% (QAMPARD), and 2.23% (ELI5). We observe that removing data corresponding to each hallucination type causes notable decrease in TRUST- SCORE, suggesting the importance of each subtype. In particular, removing refusal-related hallu- cinations adversely affects FLOR: 12.79% (ASQA), 10.48% (QAMPARI,, underscoring the impor- tance of incorporating refusal-re related data to improve a model's 's ability to discern when to provide an answer.
8

---

Published as a conference paper at ICLR 2025
Table 4: Performance of models with only SFT applied as compared TRUST-ALIGN models. Best
values within each family are bolded).
AS0A Hi 16
QAMPARI
Medel
Type
Rogs
Innatrutiaae
Roy.
Trafnsedaacn
AR(S)
Trathiduu
At-lind
TRUNT
AR(S)
Truthhilave
Al--ind.
TRUST
(R(5)
TrutSfilions
Al--ind.
TRUNY
Flu
Flas
Flue
Fly:
Flak
Fluc
Fly.
Flux
-Tb
TROST-ACIOH (DPO)
32.45
66.12
$3.94
07.51
$2.90
32.59
71.67
45.43
$1.94
21.90
22.54
05.27
41.56
44.39
-b
TRINK NON (DRO))
41.53
34.61
5166
79.30
50.17
31.30
33
67.03
19.43
14.14
4.00
1330
04
18.31
06.25
14,M6-33
SIT
75.07
67.00
70.04
30.33
16:30
$2.55
$1.33
-56
TROST-ALION (DPO)
77.35
59.82
00.35
$4.21
78.14
30.30
20.13
70.85
45.05
48.54
17.30
15.35
02.79
$5.55
46.66
LLMM33
SPT
65.99
$2.35
00.05
$0.55
24.20
35.85
71.11
40.01
25.00
22.57
65.85
45.35
44.55
TATIY (DW0))
58.31
63.55
0.5
SPT
83:91
38:71
53.03
$7.47
51.01
16.83
61:33 15
25.00
10.50
$7.10
14.15
4430
TRUST-ALION (DPO)
71.34
$0.39
01.25
32.40
54.26
17:90
15.75
01.34
23:33
35.75
21.70
00.79
Qwea225
SPT
70.23
44.33
56.15
71.16
20.12
25.50
23.99
07.05
37.80 20
34
41.30
14.14
$5.35
27.60
$5.39
(190))
0.
SFT
75.31
01.01
71.00
20
93
02. 133
1.
31.5
17:11
Qaea
SPT
65.30
31.20
35.25
25.00
40.00
52.92
64.25
45.39
43.97
PHUSS
(DPO)
53.33
64.20
67.36
31.0
31.42
73.95
54.59
34
23.39
07.62
47.42
66.14
Table 5: Ablations of data synthesis techniques for LLaMA-2-7b on three evaluation datasets using refusal prompting; The original error types in Section 2.3 were summarized into three main classes: resaer- (Inaccurate Answer), citation-related (Overcitation, Improper Citation), refusal-related (Over Responsiveness, Excessive Refusal).
ASQA
QAMPARI
RL
Tradiorthines
Trrsfwethiee
Inutverthlee
Truthfultom:
All-Grd
Truthfulinns
AS-GiL
Trutifsllssm
Ai--int
Fly:
Flya
Floc
Flac
Flux
Flue
Flu.
Flix
Floc
2P0-LALMA-2-76
56.12
K3.94
67.51
31.10
12,19
71.33
31.35
$1.75
21.50
22.34
61:27
43:41
TMUST-ALIGN wio, snpptente inviadions
20.45
53.54
53,33
$1.15
66.01
31.33
33.14
70:22
45.0%
48:33
20.50
23.58
63.30
40.21
42.52
TRUST-ALION wio, opeNer HT
27.04
30.20
03.7
31.2
65.00
33.40
13.56
71.36
45.17
50.35
27.60
23.47
63.56
15.28
41.77
TRUST-ALION wis
Iifeall!?
26.11
5135
65.31
81.55
06:31
31.00
14.00
15
45:12
11.36
28 10
22.45
41.18
GPS44 ertic
20336
5491
55.39
1K447
60.22
25.90
1077
70.29
4530
49.08
23.50
17.37
63.31
42:38
40.3
We validated our data construction approach against the GPT-4-as-critic pipeline (Li et al., 2024a; Huang et al., 2024b), where GPT-4 iteratively identifies and corrects errors to generate positive and negative responses (details in Appendix G). In LLaMA-2-7b, TRUST-ALIGN outperforms GPT-4 critic on TRUST-SCORE, with gains of 1.29% (ASQA), 1.77% (QAMPARI), and 4.12% (ELI5).
Importance of refusal samples in TRUST-ALIGN. To verify the importance of refusal samples in our pipeline,
Table 6: Effect of adding refusal sam- ples on the ASQA.
removed all unanswerable questions from the training set, creating a dataset without refusals. Table 6 shows significant drop in TRUST-SCORR scores without re- fusals, including declines of 10.2% (LLaMA-3-8b) and 1 1.41% (LLaMA-2-7b). Notably, F1 decreases by 26.34% (LLaMA-3-8b) and 26.97% (LLaMA- 2-7b), and Flac by 6.87% (LLaMA-3-8b) and 6.57% (LLaMA-2-7b).
TREND ALRO Model
Fla
Plys
Plu
VHIRT
Ouly Awornalle
DEDALL 2NA--32
200
36.70
35.13
73.39
25:33
With Wad.Ra
DF BFOALLANA-54
5643
553
1836
683
We also observe that in LLaMA-3-8b, AC is higher in the answerable-only setting compared to with refusals setting. This occurs because RAC favors over-responsive models, which artificially inflates FLAC. as discussed in main results. The resulting models answer all questions (AR% of 100%), even without supporting documents, suggesting an increased reliance on ungrounded para- metric knowledge, as discussed in Section 6.1.
Out-of-domain analysis. Following Huang et al. (2024a), we use ExpentQ (Malaviya et al., 2024) to assess our model's generalizability. As shown in Table 7, TRUST- ALIGN model outper- forms FRONT on TRUST-SCORE across all 27 open-source model family and dataset configurations. We also observe that the open-source ICL models perform significantly worse on TRUST-SCORE as compared to the closed- source ICL models, with 9.79% gap between LLaMA-3-86 and GPT- 4. TRUST-ALIGN not only closes this gap but establishes a lead: TRUST-ALIGNed LLaMA-3-8b achieves the highest TRUST score of 54.85% surpassing 54.69% of GPT-4.

---

Published as a conference paper at ICLR 2025
Table 7: Generalization test results on Exper[QA using refusal prompting
Model
Type
AR(S)
Flic
Floe
Flyc
TRUST
ICL
0.51
0.:00
41:01
9:52
16.31
Model
AR(%)
Flac
Flca
Flac
TRUST
2
PoeiCTec
44.27
18.12
IC].
18:74
21 42
18:21
0:46
19
FRONT
23.92
74.75
Qwen-25
-..0
FRONT
PostAr
99.80
51.41
51.41
13.32
18:27
15:92
24005
48.05
48.0%
34.02
1,49
56
25.03
22.83
20.96
TRIST-ALION (D90)
20.01
25.05
67.91
62.46
51.8
TRUST-ALLGN (DPO)
32.96
63.31
35.07
38.85
ICL
43
21.55
32.83
9,14
21.14
LLaMA 3.2
-1b
PoetA
ProiCTic
FRONT
48.41
30.34
20:85
K.24
5.48
47.21
49.1
37.45
2,57
1.5
29.18
19.15
19.06
Quen-
JCL
FostChe
12.02
(1900)
FRONT
99.59
29.15
56.74
33.5
36.57
41,05
TRUST-ALION (199))
3012
68.38
48.29
3. -3h
PieiCitc
POStNE:
FRONT
TEINT-ALION (bP0)
62.85
K2.85
K3.36
7.34
27.24
25.68
25.45
11:72
45.54
36.11
56.93
38.55
50,91
78.35
5.29
3.4
22.48
23,05
49.1
40.5
Quinn 25
30
PanaCli
PostAmr
ICT.
FRONT
95.48
68.85
0.05
0.05
25
35.14
29 Mb
40.65
40.65
49.65
42.67
46.48
33.54
42.49
13.55
13.55
2,52
42.5
TRUST- (DPO)
17.15
20.97
65.79
60.25
49.0
11-MA-3 4bb
PoetUire
50.05
ICL
$4.56
36.33
42.28
56.09
44,9
FRONT
19126
30,34
24.M3
16.7
37.33
Qumn-25
PratAlr
PootUnee
42.14
25
58,8
54.0
13.77
31 11.42
TRUNT-ALION (19P))
16.41
27.36
67.07
70.11
54,85
FRONT
65.51
32.41
55.56
51.77
GPT-3-5
ICL.
59.47
96.65
56.90
63.95
52.32
(1393)
51.48
GPF4
KCL
72.20
41.32
52.M1
09.83
$4.69
66.07
42.62
64.4
$4.61
P55S5
PonAAtr
PastCiic
52.01
52.01
27
53.64
53.64
7.39
29.66
29.1
FRONT
97:37
78
27.5
68.82
40.5
(SFT)
71.68
TRUST ALEGN (DPO)
26.05
27.69
69.56
61.6
$2.95
U.50ce335
73.95
11.68
51.M1
10.7
24.76
In LLaMA-3-8B, TRUST-ALIGN outperforms ICL on FlGR by 16.59% and substantially outper- forms GPT-3.5 and Claude 3.5 in both Flae and FlaR- Although GPT-3.5 and GPT-4 achieve higher FLAC scores, indicating better answer coverage, they rely heavily on parametric knowledge (Section 6.1 and Appendix F.3). This leads to less grounded and less trustworthy responses, as re- flected in lower TRUST-SCORR scores compared to TRUST-ALIGN. Similar trends are observed in other model families.
Studying parametric knowledge access. For an LLM-in-RAG task, it is important study the tendency of LLM towards grounding its knowledge on the provided documents. To partially quantify this, we compute the answer correctness score for questions that are unanswerable by the provided documents (defined as S param): ); thus fraction of cases where AgnAp 0 but Ag * 0 (more details on the metric in Appendix F.2). In Table 10, our analysis reveals that responsive models (high AR%) tend to rely on parametric knowledge more frequently (high S Sparam). Notably. closed-soure models like GPT-4 exhibit higher parametric knowledge usage compared to open-source and TRUST- ALIGN models. However, Sparam only partially captures the models' utilization of parametric knowledge. For instance, it does not account for cases where the document contains the answer, and the model still relies on parametric knowledge to generate the correct answer (also present in the document). This phenomenon is evident in Table 12, where on ASQA, GPT-4 achieves a significantly higher Fl than our models, yet its attribution groundedness score Flec is five points lower.
7 CONCLUSION
In this study, we introduced new holistic metric to evaluate the suitability of LLMss for RAG appli- cations, where they are expected to ground their responses in the provided documents. We proposed TRUST-SCORE, which comprchensively measures the quality of answers, citations, and refusal per- formance of an LLM. Additionally, we presented TRUST- ALIGNN a method that uses a constructed dataset to align models for improved TRUST-SCORR performance. By applying Direct Preference Optimization (DPO) techniques, we trained LLLMA-2-7b and LLaMA-3-88 on this dataset, signifi- cantly reducing hallucinations in an RAG environment. Our approach, TRUST-ALIGN, demonstrates performance comparable major closed-source models like GPT-4.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AVSG3-GVV2003-010) This work is also supported by the Microsoft Research Accelerate Foundation Models Academic Research program.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang. Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: arxiv. org/ abs 2404.4440 1...1422 14219.
Samuel Joseph Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. Qampari: An open-domain question answering benchmark for questions with many an- swers from multiple paragraphs, 2023. https: /ttp/aixivvvv.//////eee/eaaaaava org/ /bbs/225.... 12665.
Anthropic.
Introducing claude 3.5 sonnet. Anthropic News, 2024. URL httpt::tp///.
anthropic. ctwnd// sonnet.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:: hSyW5go0v8.
/ http///pppeeeiieeeeeeeeeeeeeeeee net net/forummic
Patrice Béchard and Orlando Marquez Ayala. Reducing hallucination in structured outputs via retrieval-augmented generation. arXiv preprint arXiv:2404.08788 2024.
Rishabh Bhardwaj, Yingting Li, Navonil Majumder, Bo Cheng, and Soujanya Poria. knn-em: non- parametric inference-phase adaptation of parametric text classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13546-13557, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: / rxiv o alos 2005. 14 61.
Jan Buchmann, Xiao Liu, and Iryna Gurevych. Attribute or abstain: Large language models as long document assistants, 2024. URL https: /tt./avvvv....///////// www.:bs///////oee /247 07799.
11

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, et al. The llama herd of models, 2024. URL http:////wxww...//a/a/////o/ 21783.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering, 2019. URL https: org.abbs/rr////////////////bbb/ 1907. 09190.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting umss: Towards rerrinedl-augmeeneed large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491-6501, 2024.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrac Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 16477-16500 Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 http://////223...oooooggg// URL https: / /aclantooooooooyo..oooooo.oooooe org/ argg/023..om acl-llllonnn 91c.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models generate text with citations, 2023b.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-aute augmentte generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023c.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. In Song Feng. Hui Wan, Caixia Yuan, and Han Yu (eds.), Pro- ceedings of the Second DialDoc Workshop Document-groundee Dialogue and Conversational Question Answering, pp. 161-175, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. hwwp::/e....2.o.oooooooooo 1.19. URL https: https::/aacaaathooooooooo orq/ 2022. www.ddallll..... 1.19.
I-Hung Hsu, Zifeng Wang. Long T. Le, Lesly Miculicich, Nanyun Peng, Chen-Yo Yu Lee, and Tomas Pfister. Calm: Contrasting large and small language models verify grounded generation, 2024.
URL wwt.thtts//.mmm //tss/////xivv.a/ org/ abs /206 05365.
Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. Training language models to generate text with citations via -raiind rewards, 2024a.
Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang. Dandan Tu, and Bing Qin. Learning fine- -he-graided grounded cita- tions for attributed large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 14095-14113, Bangkok, Thailand and virtual meeting, August 2024b. Association for Computational Linguis- tics. URL https: / awwlantthhooooooooo hww.lllyy.rggg///o wttp.///......inn.........aaeee. findings-aci...... 838.
Bin Ji, Huijun Liu, Mingzhe Du, and See-Kiong Ng. Chain-of-thought improves text genera- tion with citations in large language models. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 38(16):18445-18355 Mar. 2024. doi: 10.1609/a htt.:/......8111..7.7999 URL //jss https: /js. aaai. org/ index. php/ AAAI/ htt.///rtcceeevee///3aee
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, March 2023. ISSN 1557-7341. doi: 10. 10.1145/3571730. URL http::/dx..o doi org/10. 111.1111157777 1t../.......////5111111111
12

---

Published as a conference paper at ICLR 2025
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint 17Xiv:2305.06983, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7769-6784 Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18653/v 1/201/ 0200000mmmmpmmmii.555.. URL http:::tp///o. / / wwllannhhllogooo org/ 2020 emnlp-ma n. 550.
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models, 2024.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.0011 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. A survey of large language models attribution. arXiv preprint arXiv:2311.03731, 2023.
Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, and Min Zhang. Im- proving attributed text generation of large language models via preference learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 5079-5101, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL https: //aclanntholee Findings-acl.300
org/orgg///22 2024.
Weitao Li, Junkai Li, Weizhi Ma, and Yang Liu. Citation-enhanced generation for Ilm-based chat- bots, 2024b.
Nelson Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search en- gines. In Houda Bouamor, Juan Pino. and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001-7025, Singapore, December 2023. As- sociation for Computational Linguistics. doi: (0.18653/v1/2000 htte:////22....nnngeeemmm////// findings-emnlp.66.. URL htttsss///allathooogyyy www.hhhhlloooorgg2222.3 org/2023. findings-emr 467.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertqa: Expert-curated questions and attributed answers, 2024. URL htl.ttts///axxi..rrr abs/2309 07852.
Yu Meng, Mengzhou Xia. and Danqi Chen. SimPO: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405.14734 2024.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillnngaan Geoffrey Irving, et al. Teaching lan- guage models to support answers with verified quotes. arXiv preprint arXiv:2203.1 11147, 2022.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et Webgpt: Browser-assisted with human feedback. arXiv preprint
2021.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-We King-Wei& Chang, and Yinfei Yang. Large dual encoders are generaliz- able retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9844-9855, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: (6653//1/2022.mm///// mai............6....... URL https: / acllanthoooggy....ococo or2/2022. emnlp-ma i n. 669.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Chatgpt, 2023. URL wwl.atttpp////eeaaa...cc////e//p Accesss 2024- 09-01.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel SermadettShhaing Christopher Berne,, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, et al. Gpt-4 technical report, 2024. URL https:ttpp..... http//aawwiaaaaaaaaaaaaa/ org/abs/ 2303. 08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: /ttpaaaxxvv..//////eeeeeaaaa..ea org/ abs/2203, 02155.
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, et al. The web your sester-nnowlegg--nnee arXiv:2112.09992, 2021.
nlp against a very large web corpus. arXiv preprint
Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. Citeme: Can language models accurately cite scientific claims?, 2024. URL https: // wrx.v.org/abl..... 22rrg//bs/244.111111 12861.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024a. URL https:ttp//.o. /ttp://aaivvv.. wwtp:/iwwwogr.roo/ooe/ro/o/ooese 2305. 18290.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring attribution natural language generation models, 2022. URL wttph:tt/// http://aivvvv.e//ee//eee/eeee.ea org 2212 12870.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL
007.://////////0.00.70.... 06347.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023. URL https:t............. : http://ooceedgeeeee/oo ngs mll. www.pppre.......mmmmmmmmmm..m..e v sh 23a html.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint
2021.
Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. Attribute first, then gener- ate: Locally-atrribtaall grounded text generation, 2024.
Yixiao Song, Yekyung Kim, and Mohit lyyer. Veriscore: Evaluating the factuality of verifiable claims in long-form text generation, 2024. URL https:
wttg/as///eeeeeeeeeeeeeee 21222.166666666 1.276.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wa Wei Chang. Asqa: Factoid questions meet
long-form answers, 2023. URL https::/p//.ttc...t. /tap::://aae http///axvvvoo//// wow..oor//oooo. /abs//b///// abss/222 024. 06092.
14

---

Published as a conference paper at ICLR 2025
Nandan Thakur, Luiz Bonifacio, Xinyu Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermeds Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, and Jimmy Lin. Nomiracl: Knowing when you don't t know for robust multilingual cetreval-augmente gen- eration, 2024. URL https: /aae://rrre //axiiaxvvoo/// orw..orgggaas.... 2312.17... 11361.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023. URL https: / atw.i.iiivvv...// org/ abs
Sirui Xia. Xintao Wang. Jiaqing Liang. Yifei Zhang, Weikang Zhou, Jiaji Deng, Fei Yu, and Yanghua Xiao. Ground every sentence: Improving aerreeval-augmented Ilms with interleaved reference- claim generation, 2024. URL ht:ps:/ http:///wawwww.//aaaa/// wttp://gg////.ooo.///////
2197.01796.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving augrened Ims with compression and selective augmentation, 2023. URL 04408.
Yilong Xu, Jinhua Gao, Xiaoming Yu, Baolong Bi, Huawei Shen, and Xueqi Cheng. Aliice: Eval- uating positional fine-grained citation generation, 2024. URL pectps://arxiyy org/ abs/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfci Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 2rXiv:2407.1066 2024.
Xi Ye, Ruoxi Sun, Sercan Ö. Arik, and Tomas Pfister. Effective large language model adaptation for improved grounding and citation generation, 2024.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval- language models robust to irrelevant context, 2024. URL https:
/ abs/2311 01558.
Jingyu Zhang, Marc Marone, Tianjian Li, Benjamin Van Durme, and Daniel Khashabi. Verifiable by design: Aligning language models to quote from pre-training data, 2024a.
Weijia Zhang, Mohammad Aliannejadi, Yifei Yuan, Jiahuan Pei, Jia-Hong Huang, and Evangelos Kanoulas. Towards fine-grate fine-grained citation evaluation in generated text: A comparative analysis of faithfulness metries, 2024b. URL /tttts://arxiiv
14.64.
Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. Adaptive nearest neighbor machine translation. arXiv preprint arxiv:2105. 2021.
15

---

Published as a conference paper at ICLR 2025
Table of Contents
Nuances of answerability
B Answerability: A Case Study
17
C Related Works
18
C.I. Attributable Retrieval Augmented Generation
18
C.2 Enhance grounded text generation attributed Large Language Models
18
Metrics
D.1 Response Truthfulness
19
D.2 Attribution Groundedness
20
E The TRUST-ALIGN Dataset
21
E.1
Collecting Quality Questions
21
E.2
Collecting D's
................................
21
Augmenting (q.D) set
................................
22
E.4
Ohtaining r+
................................
22
E.S
Obtaining
23
F
Additional Analysis
24
F11
Revised metrics are less biased
24
F2
Utilization of Parametric Knowledge
25
F3
Source LLM Hallucinations
25
TRUST-ALIGN enhances trustworthiness more robustly than prompting
26
F.5
Comparison with Closed-source Models
27
F6
Adaptability with Different Alignment Techniques
28
E.7
Evaluation Data Creation Without using TRUE
29
F8
Effect of data size on DPO performance
30
F9
Fine-tuning GPT-4o
30
GPT-4 based Data Pipeline
H Experimental Setup
31
H.1
Implementation details
31
H.2
Dataset details
31
H.3 Baselines
32
H.4 Refusal Detection
32
Detailed Results
33
Prompt Templates
33
16

---

Published as a conference paper at ICLR 2025
A NUANCES OF ANSWERABILITY
Determining answerability can be challenging. To determine answerability, we use a system that evaluates the entailment of gold claims against provided documents, referred as the Natural Lan- guage Inference (NLI) system. An NLI system can range from a simple exact match (EM) identifier to an I.L. or even a human evaluator, with answerability determined based on g. D and biases of the NLI" These biases can be useful in specific RAG applications, such as solving mathematical problems where the documents provide a formula and the question assigns values to variables. The choice of NLI depends whether the RAG system requires the LLM to have mathematical under- standing. Ideally, to prevent improper evaluations, the NLI model used to construct the gold claims should also be used to evaluate the LLM responses.
In this paper, our focus is on evaluating the generic comprehension capabilities of LLMs without specialized knowledge. Thus, we use two NLI mechanisms: 1) identifying whether exact match of claims is present in the gold claims, and 2) using a Machine Learning (MI.) model to deter- mine if the documents can entail the gold claims. The ML-based NLI model is used for multiple purposes, such as alignment dataset construction (data/training) and evaluating generated responses (metric/testing). For this, we adopt the NLI model from Rashkin et al. (2022). ő(Cij, 8;) = 1 if Gy (premise) entails s; (hypothesis); otherwise, O. To determine answerability, we employ the TRUE- based method (Honovich et al., 2022) to assess whether gold claim can be entailed by given document.
The knowledge grounding problem. Typically, LLMs are designed to perform question- answering tasks, where response generation heavily relies on the parametric (internal) knowledge acquired during their pre-training, tuning, and alignment phases (OpenAI, 2023; Anthropic, 2024). Thus, most of their knowledge is grounded in parametric memory. This makes them inherently less suitable for RAG applications. where the knowledge generated by the LLM expected to be grounded in input documents. RAG is analogous to reading comprehension task, where the an- swers must come from the provided passage (documents RAG) rather than the prior knowledge of the person taking the test. Thus, any reliance on parametric knowledge can result in statements that are not fully grounded in the documents, including providing answers to unanswerable questions. Our investigation shows that state-of-the-ar models, such as GPT-4 and (laude-3.5-Somme overtly rely on parametric knowledge even when used in a RAG setting." 10
B ANSWERABILITY: A CASE STUDY
Prior works (Liu et al., 2023; Gao et al., 2023b: Ye al., 2024; Huang et al., 2024a; Li et al., 2024a) have employed substring matching to indicate entailment. While this syntactic approach is fast, it often proves inadequate complex, long contexts. case study is presented in Table To address the limitations of this superficial entailment, we adopt TRUE-based method (Honovich et al., 2022), which combines the strengths of both syntactic and semantic approaches. Specifically, we enhance the process by using the TRUE model, a 1B model (Raffel et al., 2020) fine-tuned for the NLI task, to verify, from a semantic perspective, whether a substring match corresponds to meaningful entailment within document passages. The input to the TRUE model is the concatenation of a premise and a hypothesis, and the output is an entailment score between and 1, indicating the degree to which the premise entails the hypothesis. We treat the corresponding documents as the premise, and to minimize ambiguity, the associated question is concatenated with each gold answer as the hypothesis. In cases where the TRUE model does not yield a positive entailment score despite a substring match, we rely on the TRUE judgment as the final label. However, if the substring match fails, we bypass TRUE calculation, thus reducing the computational cost of relying solely on TRUE for semantic entailment.
'For EMM, the bias is that a is answerable if an exact match for claims is present in D. We show detailed analysis in Appendices F22 and F.3.
17

---

Published as a conference paper at ICLR 2025
Table Case study showcasing the limitations of substring matching and necessity of TRUE judge- ment.
Question
How many state parks are there in Virginia?
Gold Answer
38
Retrieved document
Virginia has 30 National Park Service units, such Great Falls Park and the Appalachian Trail, and one national park, the Shenandoah National Park. With over 500 miles of trails, including 38 miles of the iconic Appalachian Trail, it's a paradise for hikers, nature lovers, and those seeking serene mountain landscapes.
Substring match TRUE Judgement
Substring is matched and as such the question is answerable.
Not entailed as such the question is unanswerable given the document.
C RELATED WORKS
C.I ATTRIBUTABLE RETRIEVAL AUGMENTED GENERATION
Retrieval Augmented Generation (RAG) has been widely studied for reducing the knowledge gap and providing more referenced information to enhance answer generation (Karpukhin et al., 2020; Gao et al., 2023c). However, LLMs prone to being misled by irrelevant information, leading to hallucinations and less factual outputs (Shi et al., 2023; Yoran et al., 2024; Xu al., 2023). This challenge has spurred research into attributable RAG, which aims verify model outputs by identifying supporting sources. Rashkin et al. (2022) first introduced the concept of Attributable Identified Sources (AIS) evaluate attribution abilities. Subsequently, Gao al. (2023b) adapted this approach to verify generated content with citations, improving the reliability of RAG systems. Simultaneously, Press al. (2024) and Song al. (2024) explored related as- pects: citation attribution for paper identification and the verifiability of long-form generated text, respectively. Further fine-grained evaluations have been examined, such as assessing the degree of support (Zhang et al., 2024b) and the granularity of claims (Xu et al., 2024). Recent studies (Buchmann al., 2024; Hsu et al., 2024) have also investigated attribution ability by disentangling the confounding effects of retrievers and LLMs. Unlike existing works, we design TRUST-SCORE to prioritize trustworthiness in LLMs by ensuring that generated responses are strictly grounded in the provided documents, thereby minimizing the generation of unverifiable content. This focus on verifiable accuracy strengthens the reliability of LLM outputs and enhances user trust.
Lewis al., 2021;
C.2 ENIIANCE GROUNDED TEXT GENERATION IN ATTRIBUTED LARGE LANGUAGE MODELS
To enhance grounded text generation, various attributed LLMs have been proposed, falling into two main paradigms: training-free and training-based. For training-free methods: 1) In-context learning (Gao et al., 2023b) is used to generate in-line citations with few-shot demonstrations. 2) Post-hoc attribution (Gao et al., 2023a: Liet al., 2024b) first generates an initial response and then retrieves ev- idence as attribution. 3) Ji et al. (2024) demonstrate that using chain-of-thought reasoning improves the quality of text generated with citations. For training- based methods: 1) Asai et al. (2024); Slo- bodkin et al. (2024); Xia et al. (2024); Ye et al. (2024) apply supervised fine-tuning (SFT) LLMs, training them to identify useful information from documents and guide cited text generation with them. 2) Beyond simple SFT, recent studies model the task as preference learning. employing Re- inforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov al., 2024a). Huang et al. (2024a) proposed a method improve at- tribution generation using fine- tne-grained rewards and Proximal Policy Optimization (PPO) (Schulman al., 2017), while Li et al. (2024a); Huang al. (2024b) introduced the modified DPO framework to enhance fine-gre finennrained attribution abilities. 3) While many approaches rely on external documents provided by the user or retrieved during generation, Khalifa et al. (2024); Zhang et al. (2024a) fo- cus tuning LLMS to cite sources from pre-training data using learned parametric knowledge. In contrast to previous approaches, we introduce BRUST-ALIGN which advances alignment data generation through multi-step process that disentangles answer generation from citation quality. This separation enables TRUST-ALIGH to simultancously improve the quality of answer generation, citation accuracy, and refusal precision. Additionally, TRUST- ALIGN addresses broader range of
18

---

Published as a conference paper at ICLR 2025
hallucination errors, including inappropriate refusals, thereby enhancing the overall trustworthiness and reliability of the model's 's outputs.
D METRICS
In this section, we elaborate on how compute metrics that are components of TRUST-SCORE.
D.1 RESPONSE TRUTHFULNESS
Truthfulness captures the model's ability to answer or refuse question correctly by computing the grounded refusal (FIGR) and the factual accuracy by computing the answer-calibrated answer correctness score (Fl
Grounded Refusal [Flck]: macro-averaged F1 score that measures the LLM's ability in cor- rectly refusing to answer question (Fl,et and correctly providing answer when required aa)
Flre: This metric evaluates a model's ability to correctly refuse unanswerable questions. We calculate based on how accurately the model identifies and refuses these questions. Let Ag and -Ag represent the sets of ground truth answerable and unanswerable questions, respectively, and A, and -A, denote the sets of questions where the model provided an answer and refused to answer, respectively. Fl is computed from precision Prer and recall Ref::
AA, n AgA Pre
(1)
|,
1,, n Ag
R
(2)
Agl
P Ryef
Fl
(3)
Prof + Ryef
where Prer measures the proportion of correctly refused unanswerable questions among refused questions, and Rer measures the proportion of correctly refused unanswerable questions out of all unanswerable questions. Here, denote the cardinality of the set, thus Pret. Res, and Flyer are scalar values.
F1 This metric evaluates model's ability correctly answer answerable questions. It is computed based on the precision P and recall Rans for non-refusal responses to answerable questions:
A,, ng
P
(4)
A,
A Agl
Rans
(5)
AAg
2Pans
Rans
Flans ans
(6)
Pans + Rans
FIGR (Grounded Refusals) provides an overall assessment of the model's refusal capabilities by computing the macro-average of Fl and Flams:
FIGR = (Fl + Flans)
2
Flaer evaluates the model's ability to correctly refuse unanswerable questions, while Flans assesses its ability to correctly answer answerable ones. By penalizing both incorrect refusals and incor- rect ref-rafusals, FlGR offers a balanced evaluation of the model's veerreesponsivenes and under- responsiveness
19

---

Published as a conference paper at ICLR 2025
Answer Correctness (Answer Calibrated) [FlAc]: gold claims Ag = (agl..... agn). we define the claims obtainable from the provided documents Ap {a.)..... adr J and the claims generated in the response Ar (o.1..... arn J. AC disregards the claims that cannot be inferred from D (answer calibration), and the exact match recall scores is computed the remaining claims, i.e., Ag 0 Ap:
Given question 9 and the corresponding
|Ag n AD n AR
AC
(8)
AAg n Apl
For the whole dataset with multiple questions (.1...qu.. one can compute the average:
AC = to
Σ G.EAgAA
AC
Where Ag denote the set of questions that are answerable using the provided documents, fully or partially; A, denote the set of questions that answered by the model (non-refusal. There are two variants of AC we study- precision-oriented PAc with denominator k = |A,| (number of answered questions). Second variant, recall-oriented RAC with denominator k |Agl (number of answerable questions). Here denotes the cardinality of the set. We denote the aggregated score by
2 PAC
RAC
F1
(10)
PAC RAC
The primary reason for adjusting the conventional Answer Correctness (AC) metric account for the presence of answers in retrieved documents is to avoid rewarding models for generating correct answers without locating them in the provided documents. This approach discourages models from relying solely their pre-trained knowledge answer questions, instead encouraging them to find and ground their answers within the provided documents.
D.2 ATTRIBUTION GROUNDEDNESS
Attribution or citation groundedness measures the relevance of generated citations to their corre- sponding statements, both individually and collectively. A citation q.j is deemed "relevant" when the statement it cites can be inferred from the cited document. The collective importance ci- tations is assessed using statement wise recall metric, while the individual importance of each citation is evaluated using a precision metric. Given that a generated response r consists of multiple statements S and their corresponding citations C, first compute statement-wiss citation recall and per-citation precision. These scores are then averaged to obtain sample-wise scores, which are finally averaged to produce dataset-wide scores.
Grounded Citation [Fleel: For a given statement 8,, statement-wise citation recall is com- puted by:
R Rie oi (((c,,),..,,.
(11)
where o( ... (..)..........s.) (0, 1} a function that determines whether the concatenation of all cited documents fully supports the statement s,. Next, we compute precision for a generated citation Gi.j for statement 8; as:
cife
((Ci.j 8:) OR -o((e + s))
(12)
Thus, citation precision is 0 if and only if the cited document Ci2 does not entail the statement 8;, while all other citations collectively entail without Gzj--
20

---

Published as a conference paper at ICLR 2025
As aggregate measure, we report Flec. which computes the F1 score using cumulative precision and recall over the answered questions only (non-refusals):
1
Reite
R
%: cite
(13)
A+
SEA;
1
P cite
PC
(14)
A
CEA:
ejEC
Paiee
Reite
Fluc-
(15)
Pote + Raite
Where A, denotes the number of samples answered by the model, S denotes the set of statements in generated response, and A denotes the set of responses (including only statements, ignoring citations) in the dataset. Similarly. C' denotes the set of citations in generated response, and A denotes the set of responses (including only citations, ignoring statements) the dataset.
TRUST-SCORE: Finally, we combine the metrics to produce a single trustworthiness score, which allows us to rank models based on their trustworthiness. This score calculated as the average of each component metric.
TRUST-SCORE
(FIGR FlAC + Flec)
(16)
E THE TRUST-ALIGN DATASET
To align LLMs towards trustworthiness, we propose a new approach, TRUST-ALIGN, The approach constructs an LLM trustworthiness alignment dataset, where each sample in the dataset consists of a question 9. set of retrieved documents D, and pair of positive (preferred) and negative (unpreferred) responses (r rr) The positive response corresponds an answer that encompasses expected gold claims for and corresponding citations referring to the documents. If D is not sufficient to answer q. r+ assigned refusal response, while r- is non-refusal counterpart. We build the dataset in multiple steps: 1) Obtain a set of high-quality and diverse questions, 2) Obtain documents for each question, 3) Augmenting (q, D) pairs that cover diverse hallucination types, 4) Construct positive responses entailing gold claims, and 5) Construct negative (unpreferred) responses by prompting a fine-tuned model and observing its hallucinations.
E.1 COLLECTING QUALITY QUESTIONS
The dataset construction process begins with gathering a diverse set of high-quality, challenging questions from the training splits of source datasets, including ASQA, QAMPARI, and ELI5. To collect seed samples, we first divide the questions in dataset into is clusters using a Hugginface pipeline" " After identifying the diverse clusters, we use Mixtral-8x78 with the prompt described in Table 24 to assign each quality score ranging from 7. The quality of cluster is determined by how difficult it is to answer the questions without requiring additional information i.e. higher score corresponds
high difficulty. We then select clusters with a quality score of or higher and sample the desired number of questions from these top clusters. Suppose we have three clusters,
C1 C2, C3. with respective sizes N2,N2,, N3, where N. N₁ + N2 N3. To sample N, questions from the clusters, we sample N, x N. questions from cluster C.. If this number exceeds the available questions in the cluster, we randomly sample the remaining questions from the filtered-out clusters (those with a quality score below 4). This process ensures that the seed set prioritizes both high quality and diversity. For this paper, we set N, to 3K, 3K, and 4K for ASQA, QAMPARI, and EL.5,, respectively, resulting in approximately 10K questions in the seed set.
C.
E.2 COLLECTING D's
For each seed question that is obtained from ASQA and QAMPARI, we used gtr-t5-xx1 (Ni et al., 2022) to retrieve the top 100 relevant documents D from the 12-22 Wikipedia snapshot.
Phttps:/ / //gi thub. www.....o...sseeeeeeee wht..ccom/huggggggffaeeeeeeeeeee usteering/
21

---

Published as a conference paper at ICLR 2025
For the ELI5 dataset, we employed BM25 in conjunction with Sphere (Piktus al., 2021). filtered version of Common Crawl, as better encompasses the wide range of topics present in EL15. We filter seed questions for which the retriever fails to retrieve relevant documents.
We utilize TRUE-NLI to derive the entailment pattern for each document. This pattern represents the set of gold claims that the document supports. The TRUE model takes input a concatenation of a premise and a hypothesis, producing an entailment score (0 or 1) that indicates whether the premise entails the hypothesis. In our approach, the documents serve as the premise, while the hypothesis is formed by combining the relevant question with each corresponding gold claim to reduce ambiguity. We take the union of the entailment patterns across documents to assess the answerability of each question if the pattern contains at least one supporting claim, the question is considered answerable.
Following Gao et al. (2023b), we identify 5 documents that are equally effective for the model as the 100 documents in terms of achieving the Exact Match (EM) recall value; we refer to such documents oracle documents for question q. Notably, to compute EM, gold claims are obtained from respective source datasets.
E.3 AUGMENTING (q.D) SET
Now that we have the questions and the most relevant (oracle) documents, our goal is create sam- ples of diverse types (i.e., different proportions of relevant documents for the same question) that can trigger multiple hallucinations from LLMs (Section 2.3). As illustrated in Fig. 3, for answerable questions, we first utilize the identified entailment patterns to generate all possible combinations of documents, then select k combinations that cover diverse patterns. swerable questions, we select documents that are similar to gold-claim-nntailig documents but do not entail any gold claims. To minimize the risk of introducing bias in citation indices, we shuffle the order of documents in each sample. As result, we generate approximately 70K question-document pairs.
create samples with unan-
After obtaining (g. D) pairs for the alignment dataset, obtain positive and negative responses (rt,r") r") ) for each pair an essential component of the dataset signaling the model's 's preferred and unpreferred responses. To achieve this, introduce response generation pipeline.
E.4 OBTAINING ++
We develop automated data labeling pipeline that synthesizes natural responses from gold claims and maps each statement to the corresponding documents for embedded in-line citations. The gold claims are obtained from the source datasets (ASQA, QAMPARI, ELI5)) and calibrated to the pro- vided documents, i.c., filtering out claims that cannot be derived from D. We first split the questions into answerable and unanswerable samples based on whether the provided documents entail the gold claims. For an answerable sample, consisting of a question g. a set of documents D, and a list of (calibrated) gold claims, we prompt GPT-4 to generate natural response by stitching together the gold claims using a template (Table 23). Please refer to the subsection below for more details
Document Recombination
Docummms
Doco4
partento
Select comboistioons conwring partwene
Doe 2
[3,0,0]
[0,0,1]
Dect Nutal
DRE4
Documanta
Doe 5 [a.8,]]
Doc
[6.0.8]
Juc
Doo
Die 5t
Doe 64
9G Invalid
For each group. romaining most milor to the oniliig
Bact
Die
Docd
Doe byr
Use Of
Find most simile far
Denig
Dueui
Dacis
Douile
Doe
Das 73
Figure 3: Document recombination process in augmented prompt curation.
22

---

Published as a conference paper at ICLR 2025
on how the prompt is structured for each dataset. The prompt template asks GPT-4 to label each gold claim used with its index from the provided list (e.g. "[Gold Claim X allowing for later matching of claims to documents. For unanswerable questions, a refusal response is assigned. To generate citations corresponding to each statement generated, we map the "[Gold Claim XJ" labels to the appropriate documents. First, extract all such labels from sentence (which may contain multiple claims and labels). Then, we greedily identify the smallest combination of documents that covers these claims, minimizing over-citation. Details of this process is illustrated in Fig. 4.
Claim- dodim-documentt mapping
12112
[.4,4]
Ove Jnhermion
Maat Preaee sardh
claidi Muxt
Matt Prober at yords II]
Figure 4: laim-dooumenttmmpplhe process.
Details on prompt structure for each dataset. For ASQA, we include the question g, a list of (calibrated) gold claims, and their corresponding supporting documents 1) as additional context. For ELI5, we follow Gao et al. (2023b) by decomposing each labeled response into three claims, which serve a of ground truth answers. Since the claim labels already provide sufficient context, we only fit the question and calibrated claims into the template. For QAMPARI, since its response format aligns with its labeled ground truth format (a list of entities), no additional action is required.
E.5 OBTAINING r-
To create high-quality preference data, we aim to obtain quality negative (unpreferred) responses. We first fine-tune LLaMA-2-7b on the training set of the source datasets creating Maft. We then test Maft the above-obtained dataset with approximately 70K questions and identify that 40K responses exhibit hallucinations. Table 9 shows the severity computation (e;) and the frequency of each hallucination type (w;). Thus, we can compute hallucination severity for each sample as eq -. e Wi.
Table 9: Fraction each hallucination amongst all the observed hallucinations in Mase (40,985), with possi- ble overlap. w, shows the severity computation of each hallucination. Icondition if condition is True otherwise it is O. See Fig. 5 for the detailed breakdown of the last three errors.
Hallucination
Frequency (w.)
Severity (e.)
Unwarranted Refusal
8,786
0.50
I(A,)
Responsiveness
13,067
0.50
Overcitation
12,656
0.34
Improper Citation
9,592
0.26
1- CR
Inaccurate Claims
14.783
0.40
F-
To obtain good negative samples, we first rank each of the 40K responses according to their severity score eg- We then select the top 50% of the corresponding samples for both answerable and unan- swerable responses. Thus, demonstrate the alignment data construction phase of TRUST- ALIGN, i.e., obtaining 19K samples with all the desired attributes (q. D, "r,tt) We perform DPO using this set of 19k samples to obtain the final aligned model.
12Seed questions, corresponding oracle documents, and the gold answers (r+) are concatenated together using the refusal prompt in Table 25.
23

---

Published as a conference paper at ICLR 2025
Overiap inaccurate Improper Citation, Overditaiion Erress hallucinations three categories 19132. Over Responsiveness hallucinations 28518, samples 69503)
hallucinations 8786, samples
Inaccurate Answer
Improper Citation
65
2864
Overcitation
Figure 5: Statistics of hallucinations from the output of LLaMA-2-7b SFT model prompted using
70K (g, D) samples obtained in Step-2 of TRUST-ALIGN.
F ADDITIONAL ANALYSIS
F.1 REVISED METRICS ARE LESS BIASED
Kanika
23
20
1:2-7h
1-3-4b
1-3.24
2:30
P-3...
P.3.S.
0-2 5.
02.5-
0-2.
025 calb
0-2 i-at o- 5-30 0:2 1oth 9.2.h
mg
callo
neg
cal
weg
Figure 6: Comparison of AC regular and AC calibrated (F1 across models on ASQA. EM regu- lar/calibrated synonymous with AC regular/calibrated.
Fig. shows the performance of our models on EM regular and EM calibrated. In ASQA, our measure of Answer Correctness (AC) was Exact Match (EM) and thus EM regular/calibrated synonymous with AC regular/calibrated. As seen across the models, AC regular tends to unduly penalize our models for refusals, resulting in baselines performing disproportionately better than models. By measuring AC on both the answerable and answered set for Fl, we arrive at an AC metric that is more fair in the presence of refusals. By correcting for the bias toward answering at all costs, able to reveal more balanced perspective model performance as demonstrated by a reduction in the performance gap (c.g. in LLaMA-2-76 LLaMA-3--8 LLaMA-3 3.2--1111 lb, LLaMA- 3.2-3b) revealing model's stronger performance compared to baseline (e.g. Phi-3.5
24

---

Published as a conference paper at ICLR 2025
mini, Qwen-2.5-1.5b Qwen2.5-3b). The ability grade the model more fairly underscores the need for our calibrated metrics.
F.2 UTILIZATION OF PARAMETRIC KNOWLEDGE
For an LLM used for an RAG task, it is important to study the tendency of LLM towards grounding its knowledge on the provided documents. To partially quantify this, compute an uncalibrated answer correctness (AC) score for questions that are unanswerable by the provided documents; thus Ag C AD but Ag # 0,
[[AR (An n Ab)) n Agl
(17)
param
IN+
AR
E.N.
Where, Ag, AD. and Ar are claims in the ground truth answer, claims present in the documents, and the claims generated in the response, respectively. W, is the number of answered questions.
In Table 10, our analysis reveals that responsive models tend rely parametric knowledge more frequently. Notably. closed-source models like GPT-4 exhibit higher parametric knowledge usage compared our models. However, this metric only partially captures the models' utilization of parametric knowledge. For instance, cases where models correctly generate gold claims without proper grounding may also indicate reliance on parametric knowledge, This phenomenon is evident in Table 12, where on ASQA, GPT-4 achieves significantly higher F1AC than our models, yet its attribution groundedness score FlGc five points lower.
Table 10: Detection parametric knowledge usage under refusal prompting.
Syore
Spres
AR(S)
Syree
ICL LLMMA
0.00
0,50
01.0
0.50
0.00
ICL-EL-MA-3366 8B
1.70
0.00
ICL-GPESS
9.74
ICI-GIPT-4
ICI-Claub--355
(DPO-11aMA-3-18)
5.56
TRUST-AI (1P0-13-MA-33-88)
F.3 THE SOURCE OF LLM HALLUCINATIONS
Model errors can be categorized into two primary sources:
Parametric knowledge-based hallucination: Errors arising from the model's internal knowledge representation.
2. Information extraction failures: Inability to accurately extract relevant information from provided documents.
To quantify these error types, we employ the following methodology:
For the non-refused questions with errors, calculate the proportion of the incorrect answers that are:
Present in the provided documents
Absent from the provided documents
For answers absent from the documents, we can attribute the error to parametric knowledge-bel hallucination. For answers present in the documents, the specific source of the enror remains inde- terminate as it can be attributed to both.
The substring matching (Gao al.. 2023b) is used here for searching for the existence of incorrect answers in the documents. As the model's response only on QAMPARI can be decomposed into atomic facts, chose to perform this analysis it. Specifically, for every answered question,
25

---

Published as a conference paper at ICLR 2025
caleulate the proportion of incorrect answers present in absent from the documents using the equations below:
| n
Apl
Presence
(18)
[e
[AR
T.E.A.
1A
(AR n [AD)]
Absence
(19)
Wel
[A
q.E.A.
Where Ae denotes the set of answerable questions that answered by the model with one or more incorrect answers; Ap. A& are facts present in the documents and erroneous facts generated in the response, respectively.
The findings presented in Table 11. Our analysis reveals that, with the exception of LLaMA-2 7B which provides no responses, all other ICL-based models exhibit a higher tendency to pro- duce erroneous answers based on their parametric knowledge compared our models. Notably, Claude-3.5 demonstrates a more frequent reliance on its parametric knowledge, which elucidates its significantly lower TRUST-SCORE score in Table 12.
In summary, our investigation indicates that baseline models, including GPT-4 and GPT-3.5, are more susceptible to hallucinations stemming from their parametric knowledge.
Table 11: The proportions of erroneous answers present in or absent from the documents.
QAMPARI
Model
Presence ( (%)
A Absence (%)
ICI-IILMMAA27
0.00
JCL-LLaMA-3
84.4]
15.59
ICL-GPT-3.5
85.04
14.96
ICL-GPT-4
89.3
10.7
ICL-Claade-3.5
72.18
27.82
TRUST TRUST-ALIGN ALIGN LLMA-LLMM-22-7B
93.26
6.74
TRUST-ALIGN (DRO-1L.MA-3-88)
95.63
4.37
F.4 TRUST-ALIGN ENHANCES TRUSTWORTHINESS MORE ROBUSTLY THAN PROMPTING
Aligning with FRUST-ALIGX leads more significant improvements in TRUST-SCORE compared to using prompting alone. While adding a refusal prompt has inconsistent effects on TRUST-SCORE and its subcomponents, it tends be more beneficial in capable models, such LLaMA-2- 13b and LLaMA-3-8b.
Relying solely prompting to teach refusal is ineffective, models' responsiveness becomes overly sensitive to the prompt. Under the default prompt, models rarely refuse (AR% close to 100), while adding refusal prompt in ICI. drastically reduces AR%, often indiscriminate refusal. This lack of nuanced refusal ability is also seen in post hoc methods. At both extremes, TRUST-SCORE scores suffer due to errors correctly refusing questions and lower citation groundedness scores. In contrast, TRUST-ALIGH enables models to identify and correctly answer appropriate questions, resulting in AR% closer to the maximum answerable percentage and improvements in FIGR-
zero, indicating
It's important to note that responsiveness should not be the primary metric for comparing RAG systems when the retrieved documents are the same. The TRUST score rewards accurate answers, appropriate refusals, and correct citations while penalizing failures. Systems with low responsive- ness will score poorly on TRUST, regardless of their overall response rate.
As shown in Table 19, Table 20, and Table 21, for PostCite, PostAttr, and Self-RAG, adding a refusal prompt results in minimal changes in TRUST-SCORE (e.g.. ASQA Self-RAG with LLaMA-2-13b 51.69% vs. 52.49%). Subcomponent analysis shows little difference FLUR (42.74% vs. 39.15%), indicating that the refusal prompt does not effectively help models distinguish between answerable and unanswerable questions. These findings highlight the instability of relying on prompting to enhance trustworthiness and underscore the robustness of system in achieving this goal.
26

---

Published as a conference paper at ICLR 2025
F.5 COMPARISON WITH CLOSED-SOURCE MODELS
We continue our comparison of trustworthiness against competitive closed-source models utilizing in-context learning techniques. As shown in Table 12, our aligned models outperform GPT-3.5 (69.23 vs. 67.64) and Claude-3.5 (69.23 vs. 64.36) on the ASQA dataset, and substantially out- perform GPT-3.5 (55.31 vs, 38.95). GPT-4 (55.31 vs. 40.35), and Claude-3.5 (55.31 vs. 39.78) on QAMPARI. However, the responsiveness of current closed-source models remains much higher than that of our models: even with a refusal prompt, ICL-GPTT44 still answers significant fraction of questions (86.81% % on ASQA, 73.40% on QAMPARI). As discussed in Section 6, this tendency allows GPT-4 to achieve higher FLAC scores on ASQA, but it negatively impacts its attribution groundedness: its Flac scores on both datasets are lower than those of our models. Similarly, GPT- 4's FIGR scores on both datasets are also lower. On QAMPARI, the FLAC scores of all closed-source models are lower than those of our models.
Moreover, there still remains a gap between our models and the closed-source models on the EL15 dataset. Our models' TRUST-SCORE is 2.45 points lower than that of the advanced ICL-GPT-4, and specifically, the AC and FlGc scores are lower. For higher FlAC, as discussed in Section 6, it is due to higher number of its answered answerable questions with comparable EM As for higher Flac. We hypothesize that this gap could be attributed to the information density of the extracted claims utilized constructing the alignment data (Section 4). Specifically, the three claims derived from the decomposition process may either be redundant or inadequate to fully encapsulate the information inherent in the original labelled response. In some cases, the decomposed claims may even fail to align with the original facts. First, insufficient information can lead the model to learn to extract fewer facts from the document, thereby reducing the answerability by covering fewer correct answers after training. Second, redundant information can impair grounded citation learning, as it repeats the same information across different claims, making the model less capable of performing precise citations from the corresponding documents. This issue is illustrated in the case study presented in Table 13.
This experiment reveals that proprietary models demonstrate greater responsiveness compared to our models. While GPT-4 achieves superior FC scores, it underperforms in terms of Flac and FIGR. suggesting limitations in its ability to ground responses and refuse unanswerable questions. Overall, GPT-3.5 and GPT-4 outperform our models utilizing retrieved documents for long- form question answering, primarily due to the limited capacity of our base model.
Table 12: Our models vs closed source: AR % == Answered Ratio in %;; FLAC == Answer Correctness Fl (Calibrated)); FIGR == Grounded refusals F1; Flac == Grounded Citation F1; TRUST TRUST score. R = Refusal prompt is used. D = Default prompt is used.
Trateseleecs
Tratventteee
Dratwenbiers
Trafelelans
Afl. Gritns
Alli Grilens
Trutiiihess
Afli Gintenss
Prueep!
Flut
Flax
Flus
Fla
Flum
Flas
Fla
Flux
Flac
Charsdorenc Mindde
OFT4
33,85
$1.81
$5.33
$1.03
54.22
$1.72
26.00
50/00
11.5
12:45
OPE
43.5
$1.7
27:01
70:00
3215
$7.24
51.31
66.39
59-01
510
0100
50.00
11.56
50.55
3672
55.31
51.65
14:22
0150
60.1
11.00
12:0
71.88
31.75
21.00
22.56
06.22
25.55
4575
27

---

Published as a conference paper at ICLR 2025
Table 13: A case study of the failure of decomposition.
Insufficient
Question: Why do burns blister and way do burn wounds remain warm long after the injury occurred?
Label: Burn blisters occur when the second layer of the skin is damaged, they occur to protect the underlying skin layers from more damage and infection. You could see it as the bodys/skins natural bandage, so never pop them. The skin remain warm because of the increased blood in the
repair and replace the damaged skin.
Decomposed claims:
1. Burn blisters occur when the second Layer of skin is damaged.
2. Burn wounds remain due to increased blood flow to the area
to repair and replace damaged skin.
Missing points:
1. Protection and Infection: The first claim does not mention that
the blisters protect the underlying skin from damage and infection, which a significant part of the explanation in the answer.
2. Never Pop Them: The answer advises against popping blisters,
which is preventive measure not mentioned in the claims.
Redundant case
Question: How do fitness trackers know that you actually s sleeping but not just laying there resting, being awake?
Label: Your heart beats slows down when you sleep, they will mixture of heart rate and how long you haven't moved to determine how you' ve slept
Decomposed claims: 1. The combined factors of heart rate and inactivity determine sleep assessment.
2. Fitness trackers consider the duration of inactivity to assess sleep.
3. A sloxed heart rate is an indicator of sleep that fitness trackers monitor.
Redundant point: The first claim has already summarised the core statement, and the last two clains just expand it and give more details
F.6 ADAPTABILITY DIFFERENT ALIGNMENT TECHNIQUES
To demonstrate the robustness of our synthesized alignment data across different training methods, Table 14 also includes the performance of SFT and SIMPO (Meng al., 2024) methods. Compared to the SFT baseline, which only utilizes the positive data points the alignment pairs to fine-tune the base model, preference optimization methods, such as DPO and SIMPO, consistently show per- formance improvements, highlighting the versatility of our data pipeline. Unlike the SFT approach, DPO and SIMPO demonstrate improved TRUST scores, albeit with reduction in responsiveness. This decrease responsiveness is actually favorable outcome, as indicates that the models are less likely attempt to answer questions for which they lack sufficient information.
Table 14: Results using different alignment methods on the ASQA dataset.
Alignment TRUST-ALIGN Model
Responsiveness (AR%)
Flug
Flae
TRUST
LLaMA-2-7b
65.
52.48
66.12
83 94
67.51
LLaMA-3 --MA-3-Bb
$6.43
53.94
65.49
88.26
69.23
LLaMA-2-7b
72.47
66.44
82.21
SIMPO
LLAMA-3-80
$7.38
49.84
64.13
86.86
66.94
28

---

Published as a conference paper at ICLR 2025
F.7 EVALUATION DATA CREATION WITHOUT USING TRUE
The determination of question answerability in our dataset is based on a combination of substring matching and TRUE criteria, detailed in Section 2. Additionally, we developed an alternative ver- sion of the evaluation data that relies solely on substring matching, disregarding the TRUE criterion. This relaxation of answerability constraints results in an increased number of answerable questions. The findings from this analysis are presented in Table 15. It is worth noting that the overall trends observed in this analysis align with those reported in Table which employs the combined approach of substring matching followed by TRUE verification.
Table 15: Results on ASQA, QAMPARI evaluation datasets where the data are created without using TRUE; AR % = Answered Ratio in %; FlAC := Answer Correctness Fl (Calibrated); FlGR Grounded refusals FI; FlGc == Citation Grounded F1; TRUST := TRUST-SCORE. R Refusal prompt used. = Default prompt is used.
ASQA
(natrverallte
OAMPARI (286 unsuneerabll)
Respondveuess
Responsiveness
Trustworthiness
(%)
Gedness
AR
Truttsfuliness
Prompt
Flac
Flox
Fluc
Flac
Flax
Fluc
LLaMA-2-76
ICI
0.00
0.00
15.13
0.00
5.04
01.00
0.00
29 20.28
0.00
9.76
PosiCine
10.44
0.13
24.91
0.00
6.35
36.40
0.00
52.57
9.50
20.69
PostAitr
10.44
0.13
24.91
0.00
8.35
34.40
0:00
52.57
3.78
18.78
Self-RGG
100.00
44.40
45.11
63.49
$1.00
96.00
44.15
19.95
24.58
2.64
PostAstr
87.24
2,71
0.43
19.59
$1.10
0.00
52.45
4,70
SeffRAG
98.00
41.22
46.27
56.59
50.03
96.20
12.13
40.83
13.44
22.50
L1AMA-2-133
ICI
17.41
19.29
$1.22
14.14
21.55
26.50
0.63
53.67
0.00
18.10
PosiChe
90.5]
2,04
56.40
1,53
19.99
100.00
0:00
36.95
8.05
15.00
PostAnt
2,04
100100
Self-RAG
48.
97.57
$1,18
50.16
9:40
36,91
97,80
41.05
Postiie
89.77
0.07
54.96
0.00
18.34
63300
0:00
5.1:22
7.14
20.12
PostArtr
89.34
0.07
55.01
0.00
18.36
58.50
0:00
$2.31
4.56
18.96
Sel-RAG
97.68
49.10
45.47
63.99
$3.65
96.30
6.04
41.17
21.06
22.76
LLaMA-3-80
36.12
30.01
77.53
ICL
89
58.83
64.47
62.12
61.81
70,80
7.48
61.003
4,81
24.44
PosiCit
97.36
37:48
40.41
17.89
34-93
92.00
3.35
45.43
11.14
19.07
PostActr
97.47
37.44
48.95
3.18
29.36
93.00
3.32
46.03
5.65
18.33
TRUST-ALIIO
Medels
DPO LLMMAL22-222
47.85
61.00
$4.95
64.30
32.30
27,80
63.60
64,87
29

---

Published as a conference paper at ICLR 2025
F.8
EFFECT OF DATA SIZE ON DPO PERFORMANCE
ASSA
60.77
38:22
52.35
54.99
47.55
15.65
44.57
10.0
50.0
Augitentee
Figure 7: Influence of the proportion of augmented training samples on TRUST-SCORE in LLaMA- 3-8b.
During the bulk of our experiments, we chose to utilize the top 50% of augmented samples to form the training dataset for DPO alignment due cost-effectiveness. Here, investigate whether vary- ing the quantity and difficulty of the samples would influence the final model performance. Fig. shows how the amount of training samples affect the final performance of TRUST-ALIGN model. Notably, selecting the top 25% of augmented samples achieved the highest performance on QAM- PARI (57.73) and ELI5 (47.55). The absence of clear trend suggesting that "more data better" can be attributed to the nature of the data itself. Although document recombination can gener- ate a large number of samples, those with lower hallucination severity scores tend to have limited complexity and high information redundancy. As result, these additional samples do not provide substantial new or challenging information for the model to learn from, limiting their effectiveness in improving model performance. When training with the top 50% of augmented samples, the model may be experiencing overfitting, which could explain the observed decline in performance. There- fore, it is likely that better performance could be attained by carefully tuning the amount of augmented data used. This finding underscores limitation of our pipeline, revealing that the diver- sity of document content plays crucial role in determining the quality of the augmented samples.
F,9 FINE-TUNING GPT-40
We fine-tuned GPT-4o using our SFT dataset. The results reported Table 16 indicate the consistent improvement in trustworthiness scores of GPT-4o as result of supervised fine-tuning.
Table 16: Performance of supervised fine-tuned GPT-4o.
ASQA asturrable, 324 susumeenii))
QAMPARI axzarnable, anamornally
(207 asnesnable cnmuneennly
Type
Trustverthiacs
Trutwwetiness
Trustorrtiaces
(R(5))
Trubhfilase
All-Ged.
TRUST
AR(S)
Trubifdincs
Alt-Ged.
TRUST
AR
Trubiluless
All-Ged.
IRUST
Flea
Flac
Flee
Floc
Flac
GP
XL
TRESH ALION (SFT)
$4.49
74.36
62.92
99.32
61.40
6867
73.66
$7.51
72J1
65.83
50.40
34.6
41.56
14.00
77.15
75.20
53.6
20.3
31.69
$6.99
66.1
25.5
35.25
26.1
6834
68.33
37.71
5618
41.33
G GPT-4 BASED DATA PIPELINE
For the GPT-4 data pipeline, employ GPT-4 to simulate critic that performs two key tasks in succession. First, identifies and revises mistakes or supplements missing information the given response based correct answers. Second, it validates the attribution of statement-level citations and corrects them accordingly. The detailed instruction is provided in Table 26.
Coverage critiques. To ensure that the correct answers are accurately reflected in the given re- sponse, prompt GPT-4 with the corresponding question, correct answers, and reference facts (documents that support the provided correct answers) as context. GPT-4 is then asked to locate specific mistakes or identify any missing correct answers in the given response. After identifying
30

---

Published as a conference paper at ICLR 2025
coverage-related issues, GPT-4 is instructed to minimally revise the original response to correct these issues based on the detected problems. This minimal revision approach is intended to generate more precise data for alignment learning.
Citation critiques. Based on the revised content, we further tokenize it into individual statements to enable a more fine-grained citation check in later stages. We format all documents in the in- struction as holistic facts and instruct GPT-4 to determine the attribution of each statement relative to these facts. We define three levels of attribution: SUPPORT, OPPOSE, and IRRELEVANT. We then compare GPT-4's attribution results the original attributions in the response, modifying the original attributions wherever they do not align with GPT-4's critiques. Finally, we concatenate all iitation-rvvisd statements to form the final revised response.
H EXPERIMENTAL SETUP
H.1 IMPLEMENTATION DETAILS
For all experiments involving our tuned models and baselines, we provided the top 5 retrieved doc- uments as context and used decoding temperatures of 0.1 and 0.5, respectively, with other settings consistent with those in Gao et al. (2023b). We evaluated three representative open-source model familiie: the LLaMA series (Touvron et al., 2023; Dubey al., 2024), the Qwen series (Yang et al., 2024), and Phi-3.5-mini (Abdin et al., 2024), and three proprietary model families: GPT-4 (OpenAl et al., 2024), GPT-3.5 (Brown et al., 2020) 15, and Claude-3.5-Somen
16, We perform the
full parameter fine-tuning for better performance. For supervised fine-tuning (SFT), we trained the models for 2 epochs with a learning rate of 2e-5. For direct preference optimization (DPO) align- ment, trained the models for 2 epochs with beta value of 0.5, All experiments were conducted on NVIDIA A40 40G GPUs.
H.2 DATASET DETAILS
Following Liu et al. (2023); Gao et al. (2023b), to form D, we divide large text documents into 100-word passages and limit the number of citations C, for each claim to a maximum of three. If the response is empty, it is excluded from evaluation. We provide statistics of our evaluation in Table 17.
Table 17: Statistics the evaluation dataset.
ASQA
QAMPARI
EL15
ExperQA
Total of Samples
948
1000
1000
2109
Answerable Samples
207
# Unanswerable Samples
338
703
793
1487
ASQA (Stelmakh al., 2023). This long-form factoid dataset features ambiguous queries from AmbigQA (Li et al., 2023), requiring multiple short answers to address different aspects. It includes comprehensive long-fom answers that combine these short responses.
QAMPARI (Amouyal et al., 2023). This factoid QA dataset is derived from Wikipedia, with answers consisting of lists of entities gathered from various passages.
ELI5 (Fan et al., 2019). This dataset is a long-form QA collection based the Reddit forum "Explain Like I'm Five" (ELI5). Most ELI5 questions require the model to utilize knowledge from multiple passages to formulate complete answer. The ELI5 dataset is frequently used in related research due to its challenging nature (Nakano et al., 2021; Menick et al., 2022; Jiang et al., 2023).
13 LLaMA-2-77. LLaMA-2-1 2-b,, LLaMA-2-70b, LLaMA-3.2-Ib. LLaMA-3.2-bb, LLaMA-3-8b
"Qwen-2.5-0.5b. Qwen-2.5-1.55 Qwen-2.5-3h, Qwen-2.5-7b
"We utilize the latest version on AzureOpenAl Service: https: http::///ieemmmmcrooffeeeee
en en-us/azun /tte://rrrr/at-sericce//////eee https///ppenn/ocoot////////////e
"ttpps: //tps://wwwr wwwp://wwwww....//./////....o... wwt.:p/..ctooo.cooo.ooooo
31

---

Published as a conference paper at ICLR 2025
ExpertQA (Malaviya et al., 2024). This dataset spans various topics and requires domain- specific knowledge to solve long-form questions. To further verify the generalizability our ap- proach, test our best model and some of the baselines on this unseen dataset.
H.3 BASELINES
Table 18: A case study illustrating the necessity of GPT-4o matching for detecting refusals in models prompted with default prompt.
Question: Who killed the man in thelma and Louise?
Answer: "nn the movie Thelma and Louise, the man who is killed is not explicitly identified as being killed by any specific character. The novie ends with a shootout and a crash, and it is unclear who survives who is responsible for the deaths. The search results you provided not provide any clear answers to this question.
Fuzzy match judgement: Not refusal Human judgement: Refusal GPT-4o judgement: Refusal
H.3.1 IN-CONTEXT LEARNING (ICL)
Following Gao et al. (2023b), we prepend with two demonstrations, each consisting of query, top-5 retrieved passages, and an answer with inline citations.
H.3.2 POST-IIOC SEARCH (POSTCITE)
Following Gao al. (2023b), first prompt the model under closed book setting i.e. without any retrieved passages, to obtain an uncited answer. Then, GTR is used to find the best matching citation among the top-5 retrieved passages for every statement.
H.3.3 POST-HOC ATTRIBUTE (POSTATTR)
Similar to PostCite, first obtain model response under closed book setting. Then the TRUE NLI model to find the best matching citation among top-5 retrieved passages.
H.3.4 SELF-RAG
Self-RAG trains the LLM to retrieve documents on demand using special reflection tokens and enhances generation quality through self-reflection. We compare our results against the 7b and 13b models released, using the default settings as described (Asai et al., 2024).
H.3.5 FRONT
FRONT (Huang et al., 2024b) utilizes a fine-grained attribution training framework that first grounds specific supporting quotes, and then generates responses with citations based on those quotes. It tunes the LLM with automatically collected data based on ChatGPT and quality filtering. We repro- duce its 7b model for the evaluation.
H.4 REFUSAL DETECTION
We employed two methods to measure refusals robustly. In refusal prompt, models were explic- itly instructed respond only with the phrase: "I apologize, but I couldn't find an answer to your question in the search results." without providing any further explanation. As the models generally complied with this pattern, we were able to apply fuzzy matching 17 detect the phrase above indi- cating refusal. For models responding to default prompt, refusals did not adhere to a fixed pattern,
Fuzz Partial Ratio was used to mitigate the impact of string length.
32

---

Published as a conference paper at ICLR 2025
making detection more challenging. Two human annotators verified that fuzzy matching yielded poor performance Table 18. Hence, GPT-4o was employed as an evaluator to classify whether an answer should be considered a refusal. The specific prompt used is provided in Table 27.
I DETAILED RESULTS
Table 19, Table 20, Table 21 and Table 22 show the full results of our experiments.
J PROMPT TEMPLATES
Table 23, Table 24, Table 25, Table 26 and Table 27 show the prompts used in experiments.
33

---

Published as a conference paper at ICLR 2025
Table 19: Detailed ASQA results.
ARS
Psc
Rur
Plac
Plat
Plan
Flas
Ra:
Pan
JCL
25.85
1,2h
33,05
15.14
23,56
34.55
31.20
31.25
52.30
06.11
44.12
41.10
0.35
0.35
0.33
22.83
06:11
4.70
10:36
8.55
11.96
45 10
07:05
8-73
8:11
FRONT
100.00
15.19
$8.69
17.63
000
0.90
0/00
100.00
66.35
78 31
14:15
47.55
4830
45.57
47.63
$3.65
20:13
14.25
36:71
27
59.77
29:67
09:15
78.66
59.05
61:0
51.6
51.47
TRUSS-ALICK (DP0)
T1.54
23.98
4738
53.33
53.99
42:33
33.56
47.27
79.53
71.37
75.20
61.2%
52,95
$1.34
$3.40
$4.75
ICL.
56.32
30.35
41.75
13.44
53.55
1.66
64.29
5.11
66.76
76.35
4.76
5.45
6,03
6.69
32.95
71.73
5.30
15.51
17.30
6.56
31
33.01
3531
53 13.66
4.45
4.45
6.45
34:43
4.05
SFT
78-07
25.03
40.30
20.00
44.34
33.0 14
54.50
41.1
$4.59
09.54
76.33
59.05
75.87
62.01
7106
58.00
TRUST-ALIO (DP0-296)
76:45
2501
45095
53.5
48.56
99.15
$3.19
47.06
$5.00
71.99
77.35
62.41
76.30
74.12
75.50
62.25
ICL.
28.55
65.11
19.23
65.66
51.22
876
13.58
1944
75.66
1.50
71.31
17.00
35.30
11.44
1394
1863
13.59
0.55
74.20
17.55
56.23
FRINT
97.17
15.74
69:35
41.11
5196
TRUST-ALICH (D80:
40-47
23.11
63.43
43.81
55.19
71.01
55.10
35.15
00.3
75.00
65.31
63.56
19.77
75.52
75.66
65.85
61.0
10.35
16:16
5114
71.00
71.6
63:01
81.03
74.15
$0.99
61.
JCL
10,00
35.15
1000
71.54
31.14
17.46
15.67
21.51
91.15
06.14
01.11
54.54
11.52
73.41
11.46
PestAta FRONT
91.46
31.1
13,44
58.35
11.52
5.56
45.76
15.15
92.13
66.32
76.35
43.05
11322
12.92
$102
30.45
$6.19
+1.33
16.91
75.44
61.%
73
3453
70104
31
60.00
01:15
55.11
54.77
00.99
SFT TRUST-LLION (D00)
65:30
34.55
10:35
15
53.33
51.75
5671
$3.07
25.5
7L-17
75.0.
t0
85-24
78.67
65:7
25.15
57.25
52.95
55.44
02.13
56.17
71.48
T1-31
74.23
56-22
54-33
62.83
$3.5
06.13
Tetcc (340) 1991
PookCiz
23.00
21.75
13.35
10.11
5456
76.92
33.77
45.33
21.1
64.35
34.0
41.3%
9.40
5:40
9.45
21.92
75.00
71.66
$5.95
10:1
1156
76.00
15.0
2311
31.0
16
1.34
1:56
1.M
19:00
115
7641
35.79
74-00
66.96
24.51
11.30
55.33
51.5
51.4
54.8
$1.33
74.01
75.25
64.31
57.55
$2.99
$3.56
07:32
TRUST-ALION (SRO-256
66114
25.45
55.51
55.00
54.25
54
51.52
55.4
7354
75:44
76
66.16
57.13
$1.63
$4.59
68.23
ICL,
0.00
12.74
0.00
000
0.53
100/0
35.65
$2.55
0.00
0/00
25.35
0.00
6.00
0.95
5.36
PasaCiaa
0.25
3516
1629
18.00
0.0
14.00
11.77
10110
25.17
17.13
51.1
15.19
DU91
1400
TUB1
66.35
71.11
36.15
6631
63.00
84.35
-0.72
77.33
69.47
(09)
115/00
16.35
76.31
39.15
66:45
09:11
65.56
JCL
04.30
32.23
43.05
62.99
53.58
11:54
72.25
13:00
00.54
09.55
29.13
43:51
64:11
43.14
43.67
42.85
1.9
2.01
1.75
2.33
13.05
5041
26.55
6115
19.31
76.55
$1.56
0.43
044
0.4
18.00
58.63
4.45
06.25
06.45
71.34
4:16
3009
64.5
$5.39
45.15
SPT
20:11
47.06
91.15
51:11
36.65
63.9
41:15
2131
71.54
7311
63.43
85:36
76.11
33.61
65.42
TRUST-ALION (2P0-299)
71.31
27.33
51.32
56.33
$3.55
45.82
6556
$4.10
$2.6
74.41
76.33
65.16
84.37
$2.09
$3.20
60.39
5.17
20.54
91.1
25.3
M0.11
190
1.5
7.11
11.0
11.55
11
4111
16.
77 14
15.1
1.51
1.51
17.8
90331
1.5
1.00
2/40
221
14:20
51.55
22.43
00.13
06.23
72:39
43:11
0.11
6117
0.30
1244
55077
53M
0.00
tuk.
0154
15.00
$1.55
21.45
92.46
77.11
50.55
0.00
000
0.00
16.35
0.0
16:17
35.15
66.5
77 11:1
010
0/91
0100
12.1
40.57
43.57
13k
0.13
0.53
59.70
33.76
41.23
25.41
67.07
54.35
50.25
0.34
0.34
01.4
12.02
45.78
0/76
0:11
51.4
1530
43
45.90
64.53
010
000
0.00
16.30
54.66
47:15
2139
45.91
43:1
54.14
$1.4
73.77
74.34
34.00
6391
T7.55
01:1
31.10
00.77
TRUSTALIO (DP0)
Teusi-AASON 259)
51,00
17:93
4100
4457
07.40
40.76
$7.20
62.50
77.55
04.00
63.00
77.08
7010
02.35
13AMKA330
1.27
12.74
10k
2.54
92.43
$2.55
10.11
3.32
21.96
53.0
5144
5195
27.95
PaseCae
47.26
16.55
16.6
26.1
31.00
61.01
43:20
$1.55
51.44
72:71
61.03
55:00
20:99
22.5
21.00
3683
69.25
41.5
76.3
63.19
10.55
$3.22
1433
16.57
74.56
55.00
51.66
56.00
SET
(6.01
25.75
(7.00
51.11
40.23
51.0
$7.45
$1.9
71.57
76.05
15.47
71.52
25.03
62.44
TRUST-ALICH (000)
71:35
51.51
54.33
65.99
59.52
42.31
65.10
$2.19
20.00
73.55
91
65.58
5500
53.43
$4.11
70.44
LLaMk.340
5/09
67.14
1.51
1.01
14.00
1516
2.1%
1.17
13.%%
$::4
34
PoolAer
77.55
25.15
1017
$6.15
13.9%
17.65
11.66
11.16
16657
75 55
11.1
1.05
3.55
3015
ICL.
59:66
W:41
19:51
2117
56.3%
11:05
05.35
16.15
74.56
15.0
61:31
61.17
61.99
$8.50
PoaCac
09.26
2763
15.91
23.0
1420
4.75
61.4
$.70
09.16
40.05
75:13
43.56
17:89
12.89
1120
32.00
FRONT
0515
41.76
$1.34
59.30
62.5
$5.20
4.1
99144
66.96
76.63
41.02
65.30
66.45
65.14
56.55
SFT
(6.99
25.33
10.90
$1.11
53.35
51
19.51
5174
74.77
26.14
$3.05
66.55
TRUST-ALICH 1000)
26:45
23.33
$7.72
50.53
33.44
04.75
53.0
55.12
05.0
17.26
72.06
63.40
19:33
$7.60
$3.26
99.33
Clased
Moduls
GPEES
71.M0
30
10.35
35.33
$1.91
41.33
40.44
51
20.30
74.37
11
64.00
81.1
$1.0
67.64
Cleako33
$4.90
36.14
13.70
09.41
59.97
34.00
14.77
47.53
01.91
72.09
$3.01
64.77
67.28
09.41
48.35
6136
GPTAS
91.11
36.67
46:27
67.88
55.00
14:20
34.55
99.11
67.00
$3.65
78.13
77.95
74.0
4135
CPTE4
10.72
41.3
12.58
76.15
01.7
16
11
2410
46:33
91.11
54117
19:45
7462
5845
75.15
34

---

Published as a conference paper at ICLR 2025
Table 20: Detailed QAMPARI results.
Prorpt
ARS
Psc
Ra:
Par
Pla
Flas
Ra:
Pan
JCL
11:41
4.39
1.10
2,65
MU.27
71.63
71.52
14:22
33
21.52
50.53
0..0
0.00
(00)
33.71
17:00
Eaph
0153
D107
77-43
31.06
1.10
57-51
5.33
S73.
5.7%
19:03
FRONT
09.30
1.43
25.14
11.53
0:57
57.14
1.10
10.98
75.41
45.34
23333
15.66
16:30
15.00
16:88
16.50
20.74
13:03
16:00
75.55
$1.85
30.56
$3.51
41:25
78.31
27.80
35.06
TRUSS-ALIOK (DP0)
21.35
12.00
15.76
10.25
76.13
33.56
$3.31
41.77
01.84
25:70
26:75
24:33
33.7%
ICL.
25.00
00.51
10:25
15.4
15.56
41.13
30.75
Wint
39.55
41.33
8.47
4.76
5.61
21.55
11:30
6.3c
3.57
3.60
00.99
11.55
15.5
40.5
22:1
51.1 11
1.00
1.0
1,07
18.56
91.67
3.00
25.76
1034
15.30
SFT
25.30
25.13
12:27
23.99
K5.67
01.01
$2.30
60.96
56.00
69 66
33.41
57.96
37.68
43.74
TRUST-ALIOK TRUST-ALIOK1DP0.225 (DP0-29%)
15.40
36.39
15.01
26.34
91.39
79.04
54.31
4305
67.39
51.77
68.25
5..55
52.25
51.95
49.55
ICL.
26:11
10.34
23.1
45.56
63:21
46.31
42:25
41.55
0.10
D101
010
016
0.50
010
11:31
0.000
0.00
1177
0.00
0.00
0.66
41:31
15:00
25.09
23.00
21.00
FRONT
79.10
10:35
55.01
TRUST-ALICH (D80:
15.10
26.35
46.34
$3.59
06.31
90.35
76.55
53.55
$1.35
63.96
20:31
45:31
45.77
43.64
5555
Tatten-Ar (500 2009
11.53
21:01
76.1
2510
51.94
68.34
17.67
41.14
17.11
18.75
JCL
36.39
22.54
42.15
25.51
35.34
0153
06.33
$5.15
44.%
26-31
61.01
14.32
36.76
31.15
41.%
PestAta FRONT
25.20
5/k
5.13
$,39
70.15
26.13
4065
44.-
42.00
00.15
13.55
13.55
13.35
21.45
$4.70
11:47
13:41
17.00
71.5
40.4
14:11
71.6
76.15
SET
31.30
37:41
11:01
31.1
30.11
110
36.7
16-13
10.90
70 10
11.9
09-33
50.00
52,00
25.93
11.4
51.11
2231
29:42
70.08
53-45
53.48
51.42
TRUCIAAIINN (TR0) 7994
55.71
T5UM
2,47
0.45
3.57
$3055
31.36
46,35
42.35
2.43
4.40
4,49
15.11
7000
3.47
4.15
1.50
2557
1335
5505
11.9
05-05
27 36
0.96
0.16
0.06
15.00
19300
22.19
19:15
SIO
36.55
16.3
55.4
HAL11
S194
50.0
63.79
63.42
73.95
51.36
55.35
5140
$4.59
TRUST-ALIGN (DPO
50.10
35.32
16/4
3554
54.52
5555
64.45
65.00
75.14
52.31
53.82
$2.52
54.45
ICL.
0.06
5/0
0.00
500
2055
22.35
0.00
0.39
0.00
41.35
0.35
5.00
6.00
13.3
PasaCiaa
36-91
0.6
45,42
38.0
41:44
57.34
0.80
0.50
0.01
23.3
13.45
15.71
1.25
16:00
4:11
125
62.5
12.40
06.51
25:33
17.00
22.91
14.55
18.55
11.19
57:99
17:23
3.00
(100
2/10
190.00
25.5
45.56
22:16
14.20
34.32
24.15
21.44
JCL
90.00
5,00
17:51
8:36
125
15:0
5707
10.66
0635
31.03 11.00
3.13
2.06
1.85
14.22
51.00
500
0.00
100
3631
20.34
11:11
30.30
49 0350
4.30
4.10
4.70
15.00
06.30
3.0c
16.45
1/22
460
$1.21
12-
01.05
25.4
40.02
27.05
13.25
18.30
13.44
1575
SPT
$1,00
32.53
14.06
33.5
5353
22:35
53-11
55:00
71.13
45.25
45.31
50.42
TRUST-ALION (DPO-299)
31.50
32.55
14.20
53.55
81.3
$3.5
12.35
61.60
55.76
50.07
70.59
$2.01
$5.19
52.15
51.22
0.42
41.70
59.11
0..00
19100
010
300
76
15.5
8.05
10%
103.00
0.00
230
200
12010
23:00
45.30
22:15
2.5
2.5
100
8.31
0.00
5601
51.55
03.50
35.75
43.34
47.25
7.34
7.14
1,14
15.11
010
2633
5500
05:21
4.56
476
1.96
1181
10.11
0.57
0.33
4311
36.00
15.04
43.55
1.50
1.61
1A1
11.2
5400
0.59
0.67
0955
31.0
20:11
15
0.11
0.11
16.0
34.54
14.77
1591
24.3
26.32
23.08
16.41
9141
10-31
23/0
54.55
05.00
31.02
19-01
33.02
4471
TASNE-ALION 220022533
21:81
34.13
25.85
24.66
19.32
9154
10.00
15.3
15.54
74.12
40.31
46.1
01.11
46.71
54.00
14:58
11:31
16.56
71.11
26.53
T1.14
11.04
41.97
5115
12.19
13.63
12.57
35.55
50000
5.5
7.43
136
7483
45.77
16:11
41.39
55
6.1
631
1,55
21.00
FRIINE
V0.31
$.56
26.40
13.0
0592
17:44
11:44
47.34
13.00
1947
1996
14:15
31.00
SET
55.91
20155
27.19
24.5
16.11
$1.51
12000
$5.00
54.2
57.41
20.22
11.00
16.34
36.0
45.05
BPO
45:31
23.4
15.36
20.3
91.31
72:35
24755
31A0
06.35
70.35
43
45.95
43.5
41.56
51/9
25.35
114
V7.47
71.00
30:41
111
11.77
11:25
21.51
3131
31.0
1/6
17:05
18.11
26.5
05.17
3153
1.61
1.6
1/61
1109
ICL.
112
4.50
70.00
5585
SAln
11.08
59.65
51.50
4.15
5:3
1.81
20/6
10.00
150
314
1ST
2335
1363
25.75
40.00
31.00
11.14
11:14
11:14
14.5
FRONT
102.00
$.70
19.55
13.53
530
350
13010
25.30
45.55
22.15
25.77
2007
2144
19.91
SFF
54.20
32.55
10:11
31.5
1151
11
57.71
71.11
17.7%
05.34
03.01
$0.00
TRUSS-ALICH 1000)
22,41
40133
31,00
53.35
10005
9151
19.11
40.55
26.5
20.33
19.35
36:96
34.11
5408
Chadd- Madea
GPEES
20
47:99
26.13
65.55
41.51
W1S1
15:0
94.11
56.46
30.75
13:95
31:00
14.05
Cleako33
69:30
30.35
47.29
25.40
41.70
47.55
06.34
07.39
41.12
57.00
54 10
13.35
53.95
32.83
15.7%
GPTAS
94.50
13:32
42:68
20.30
87.31
12:33
67.63
10.68
46.45
30.97
21:47
21:32
23.00
CPTE4
15.20
17:59
12:01
26.19
17:45
1060M)
20.71
00
13.0-
50.34
4100
35.66
3102
1116
57.14
0.90
0.00
35

---

Published as a conference paper at ICLR 2025
Table 21: Detailed ELI5 results.
Proege
ARE
MCrg
Par
Rar
Flur
Per
Plur
Pon
Pla.
Flas
Rew
Poo
Flae
Teeeer Scoun
1-01
8.5g
11:1
11:11
0.31
0.14
0:31
26.33
7:0
6.00
76.3 31
6:97
1047
26.36
18:4
43.56
31:33 164
5135
10
0.6
0.88
0.18
13.94
2.25
8:11
10:35
01.10
68:45
17
18:43
13.19
23.11
19:77
24.06
(BPO)
35.95
1.70
4.00
11:54
10.10
75.31
14
$1.75
01.01
34.05
94-07
60.15
75.00
30.00
50.81
11:33
XT.
17.00
12.09
99.65
N1.66
3.6
81.11
1.3
12.78
5.67
4.40
Pookli
4.16
35
1563
35.65
76.11
5.17
5.11
5.17
1584
FRONT
09.95
9.53
11.11
57.03
19.51
0.13
100.00
0.35
100.0
$6.33
17.39
3106
16.35
57.00
3485
SFT
41.30
1.50
1061
21.14
14.14
00:93
85.01
12.50
28.11
50.39
55.35
32.10
14.35
37.69
32.55
TRUNE-ALUO (DRC)
35.00
1.57
1534
26.55
19.00
5-.4
T1.14
51.13
31:25
30.67
$7.91
3154
30.76
3163
35.15
16M
02.35
21:1
26.19
$7.52
46.11
20.4
40:81
34.34
29.30
82.11
25.44
25.03
-9.32
1331
1539
26.5
11.33
10.11
4377
51.53
21.44
33.34
44.55
1.31
1.31
1.31
2184
FRONT
03.60
11.11
51.1
13.99
14.20
99.53
36.5
27.15
31.00
1181
14:11
1431
T1.81
87.22
18:75
$6.04
75:36
A4.30
61:47
3931
33.64
36.77
37.40
(DHO)
17.50
1.55
30.9
1741
14.11
41.10
17.43
45..5
65.5
55.5
15.76
47.66
41.04
XL
62.35
13.30
1747
7161
24.11
21.31
V7.04
31.00
24.35
27.13
49.01
62.76
41.13
36.51
Pann
15.23
61:31
31.33
3:41
23.33
701
7.0
95.00
TRONT
57.60
1.9
19:11
53.95
5433
79.1
91:44
81:57
30.00
41.04
51.11
59.1
51.15
1661
16.34
SFT
25.90
1.91
18.87
21.14
31.1
5019
53.31
10.72
A5:00
6135
50.6
63:1
46.84
13.95
TRUST-ALIOR (BPO)
21.00
1.30
24.13
24.4
24.19
86.44
55.05
14.00
43.00
41.55
42.93
61.9
41.10
4675
43.04
31.32
17:30
21.55
23.07
23.11
24.11
31.11
20.35
30.06
3014
31.02
1831
17.4
8131
$4.50
1433
1522
5603
21.25
16.7
53.33
21.2
$7.44
21.42
54.1
3181
0.88
6.8
17.5
24.90
1.31
2075
24.55
22.09
12.47
14.30
31:21
43.2
65.0
51.8%
41.08
46.7
TRUST (DNO)
50.00
23.50
1.55
22.34
25.35
21.76
15.10
96.14
14501
46.00
42.00
45.35
05.15
4018
41381
45.15
44.60
XLL
0.51
1.5
010
120.0
2.4
4.72
46.11
1450
0.50
0.10
15.57
PooiCli
000
6.35
136
16112
76.31
15.1
125
41.55
584
304
304
SAl BAG
75.90
6381
4.5
33.04
14.1
2015
17.57
43.07
$1.57
31.44
36.11
4130
1336
15.65
15.00
23.8
FRONT
1307
21.56
6.00
100.00
17.15
$2.44
53.01
52.72
30.51
XL
05.90
12.09
1200
53.55
13.53
5.55
05.02
55.55
21.41
35.03
22:33
1573
1600
16.00
19.05
ParAA SAERAG
$1.00
K.15
341
2114
11.11
15.01
$1.7
35 60
31.5
3411
6.9
$4.00
02.90
37.55
13.15
2.40
93.00
20.14
$4.37
19.5
9.8
11:66
10.11
14.5
SPT
20.50
1.30
1836
2613
21.31
71A5
$1.88
54.59
35:51
45.02
61.00
45:35
35.10
50.0
41.00
TRUSTALLGN (DPO-229)
24.25
1.55
21.5
24.55
21.02
$1.34
53.92
$3.00
47.34
40.50
43.65
61.51
41.59
44.00
4605
44.55
11MMV211
XTI
6.90
11.4
13.17
14:1
31.55
S161
1.24
6.76
471
PooCClic
76.96
1.7
1.44
231
1.27
25.33
57.15
15775
$1.11
21:11
36.35
35.00
0.72
0.11
6.12
13.40
SaI-NA
22.90
1.40
12.57
15.30
1231
$1.99
$232
35.25
33.54
35.05
55.6
2110
21.00
26.34
32.00
91.0
PooKllcc SAERAA
7X01
130
10.15
15.96
1425
3,05
11.43
1776
43.31
6:13
433
4133
13.05
58.90
7.38
34.94
12.10
130
1915
91.00
25:71
54.35
12.0
5.71
5.60
0.66
12.65
18:40
1.43
211
160
51.7
2965
1000
1951
21.55
3097
50.85
1.00
1.00
1.00
13.96
50.55
0.00
FRINT
V7.35
137
16.11
1.00
51.55
31.44
20.56
30.99
23355
TRUST-ALION (DPO)
9:00
122
5,6
13.0
00.33
$2.30
23.11
48.0%
31.02
99.35
48.6
41.08
46.21
45.35
15.90
1,5
1991
15.34
15.9
16.00
32.00
61.00
41.25
36.33
XLL
21.96
1.5
1814
1005
15:55
20125
51.00
33.11
25.11
30.55
51.35
31.11
10.34
50.00
34.54
Podkkl TRONT
02.90
16.13
11.00
3054
13.45
$30
1925
1510
95.5
21.55
55.44
25.14
0.53
0.53
0.33
5412
1.75
12:35
51.35
14.55
1576
25.6
47.10
31.1
13:11
41.00
13.00
41.9
11:38
SET
10:30
1A7
1916
1341
15.0
15.76
11.55
16.76
63.59
15
15.53
53.11
13.5
TRUST-ALIGN (DPO-299
4.10
0.50
12.36
4.5
1,41
98.11
$1.33
15.51
12.5
53.1
20.3
$4.5
65.6
51.72
39.31
40.32
XL
67.00
1050
1580
11.55
N150
40.6
14
11.09
31
36.17
15.5
5.6
Str.
816
3191
1030
15377
41.35
31.0
53.35
43.33
1.23
1.22
FRIINT
99.30
20.00
34.44
46.00
27:33
31.44
95
1740
11.09
S:91
18.00
136
1.0
04.11
50.55
19.47
6:11
631
631
1460
96.92
12:40
11.99
5431
19.00
2:13
51.02
93.00
20.72
54.15
15.5
1.02
1.02
12.7
SET
25.60
1.27
2119
26.15
22:51
$0.96
9613
$4.00
45 T9
43.00
45.00
63.6
51.72
43.70
06.5
4183
16.00
3.10
28.54
22.05
24.9
16.71
36.33
45.55
40.87
63.79
$4.17
53.50
53.9
43.55
Chandd-nnee Madeh
GPE.5
29.00
1500
$4.51
12.34
34.5
45.02
54.21
56.57
56.00
$7.29
45.31
GPT-4
01.50
11.00
22.01
6661
11:4
W1:33
61-4
$130
41.77
$1.11
61.35
03.35
61.54
4233
GFE3S
11.51
65.95
31.44
V2.31
21.40
25.14
2414
45-46
66.10
46.24
31.03
Clavd-34 3-
96.00
7,41
7.90
31.54
11.16
51/7
01.33
$5.00
31.44
46.05
50.00
1035
13:45
11:33
36.31
36

---

Published as a conference paper at ICLR 2025
Table 22: Detailed ExpertQA results.
Pooept
ARR
Esc
Flac
Rar
Flat
Flur
Flux
Ba:
Fas
Flue
TRUST: ScoRE
TRUUT--COREE
51.41
7.00
10.33
17.55
13.33
68 (9
68.69
56 09.9
51
31.57
39 18
15.06
5.53
5.07
36
23:33
PISAA
51.41
10.33
13.31
65.05
51A1
18.08
1.4
1:38
1.49
20.56
DRUSE-ALIGN (UPNO)
$2.96
4.1
11.34
18.5
18:18
75.52
71.24
TW.37
$1.47
90.25
85.31
35.58
34,6
35
18:53
7.22
PootTh
62.15
14.64
16:33
33
13.03
75
73
51 53
07
14.1 17
65.
15.50
15.30
18.80
16.00
16:31
PoetAar
62:15
14.54
16.73
22:23
73.05
07.5
US.0
18.66
11:41
15.53
26.00
TRUST-ALIGN (2PP))
30.2
5.71
25.59
24.56
25.06
H1.24
79:79
SIL51
$5.13
$1.4
56.25
08.15
$1.25
50.55
31.44
48.25
ICL
15.59
35.14
35.53
Poofis
0,0
6.76
0.00
0/00
0.00
51 11
8.00
1150
10:66
D.00
0.00
D.00
13.15
PoetAar
0.00
011
0.00
1.50
8:00
(1:01
11.55
TRUST-ALIGE (DPO)
17:15
3.53
29.3
16.2
20:33
02.06
75.18
83.37
37.24
08.25
48.2
65.19
61.29
59.15
60.25
49
$4.56
20.18
14:33
67.01
16.11
71134
$0.31
$1.51
51.87
16.16
1610
41.
Poofls
12.34
13.76
70.0
15.5 18
616
77.90
16.1
50.74
17
519
11.11
16.37
1177
11.47
PeetAar
13:14
15:76
33.94
61A
50-74
12:16
10:49
TRUST-ALIGX (DPO)
24.99
5.51
25:87
22.95
25.57
95.45
79.04
$2.59
02.90
55.72
09.16
63.58
62.04
62.7
52.48
It1.
K5.15
25.13
11.44
15:15
15.0
V1.30
42.95
50.11
01125
11.0
11.35
16.1
11.45
Peotise
$3.00
21.80
32:41
11.1
11:08
12.12
74,15
AL.11
51
45.07
55.05
1:55
7.36
T:39
20.06
PeetAar
53.00
31.80
32:41
21:49
24,55
AL:1
43.92
50.64
5.7
3.60
24.1
TRUST
ALION TRUST-ASICN
(DPO)
26.05
5.52
30.55
2533
27.69
8574
79.49
92.5
51.76
62.48
56.02
09.56
65.13
58.44
61.6
$2.95
I.1.
0.51
1110
0140
Wi-65
58.54
N1.15
1L:1
ILST
41.01
16.44
Peotise
5.62
14.03
15.99
1.55
4.5
79.36
5.43
34.33
922
61:11
5.56
5.19
5.23
18:12
PeetAar
15:98
79-14
92
2:11
2.25
11:33
TRUST ALION
20.00
4.34
32.15
20.8
25.03
90.65
37.53
$3.60
43,82
67.28
52.53
67.91
63.98
61.02
12.46
51.8
I.1.
13.91
14.34
41.2
21.35
10.55
11.15
15.45
V1.2
51.6
41.22
32.83
1.16
12.25
21.34
Pealiie
31.84
12.00
5.55
5.41
2EA2
2019
49.1
2.6
2115
2.07
19/0
PeRtAR
48:43
1624
59:11
20.15
41:21
31.15
1.45
1.34
19.13
ARIIN
15.64
3.14
30:85
15.15
20.32
25
75.63
83.55
31.05
46.21
64.87
03.32
61.48
62.1
49.1
I.L.
59.14
11.45
25.72
535
2..5
14.31
56
65.37
35.5
40.42
51.21
3510
42.14
36.37
41.33
Peullic
K2.55
24.15
11:31
40.8
15.05
15.29
73.12
20.26
$5.34
30.10
40.35
38.11
5344
5.24
5.29
15.33
PoRtAR
29/5
3,39
3.4
12.45
7.34
1.58
11.33
11.73
05
72.07
13.14
14.77
30.51
80.35
76.54
74.35
49
ICL
0.66
0.25
10.24
1.44
252
91.7
56.0
51.55
1.75
6S.11
3.45
41.5
61.95
70
60.46
16.25
Poaliie
15.58
4.57
21:13
10.5
1406
85.27
59.33
76.45
13.24
35.59
23.05
S0L0
1.15
2.00
7129
13/14
1406
76.45
0.25
17:38
19:92
31.3 16
91.48
70.07
81.34
72
19
67.07
20.55
09:57
7011
54.55
GPE33
50.47
34.53
15:01
$2.06
1665
43.02
$1.23
00.36
75.81
40.06
$3.43
56.39
63.19
14.58
63.03
52.33
CP74
12.30
15:92
10:65
68.11
41.32
35.96
15.72
51.2
9003
30.21
54.0
$2.1
70
09.56
0938
54.8
Cladde-3.5
9.3
11,68
54.51
37

---

Published as a conference paper at ICLR 2025
Table 23: GPT-4 prompting templates used for generating natural response based on gold claims for ASQA and ELI5.
Type
Template
ASQA
Please provide high-quality answer to the given question using the provided document. The answer must include all the answer labels, and each answer label used should be marked with its index immediately after it in the format [Answer Label x], where X is the index of the answer label in the provided list starting from 1. For example, [Answer Label 1]. Ensure the answer is coherent and natural and does not exceed four statements. You cannot make up any factual information based on your imagination: The additional information added from the given document should be relevant to the question and grounded by the document, but must not contain any factual information that cannot be inferred from the given answer labels. (e.g., if the answer label does not mention a specific year, you cannot introduce a specific year in the final answer). Question: (question] Document: (passage) (answers]
Output:
ELI5
Given a problem and some claims as answer tags, please generate high-quality response. The response needs to follow the following requirements:
1. Use only all of the claims: Ensure that the response contains and only contains information from the given claims, without introducing any new information. Guarantee covering all claims in the response.
2. Each statement must contain valuable information: Every statement must either directly originate from the claims or infer from the claims, avoiding any irrelevant and unuseful information included in the response. You can use each claim only for one time.
3. Condense and combine: If there are similarities between claims, merge them into a comprehensive statement to make the response more concise. For example, if two claims both mention similar aspect of health benefits, statement.
they can be merged into
4. Fluent and natural: Ensure that the statements in the response are coherent and natural, using connecting words and maintaining logical order between statements. 5. Answer tags in response: Indicate each claim immediately after the corresponding content in the response with the format [claim x], where is the index of the claim in the provided list starting from 1. For example, [Claim 1]. Question: (question) (claims)
Generated Response:
38

---

Published as a conference paper at ICLR 2025
Table 24: Prompt used for acquiring domain labels and knowledge demanding score.
Prompt
The exampl below are quest fons from same cluster, "dentiffy single short topic they share in common, for exanple; Philosophy, Lifeatyle, Lirear Algebra, Biochan Rcononics, etc. Additicnally, evaluate if the topica the examples aze broadly suitable knowledge-demanding quastions irre_evant content, wide range of content, including scientific, educational, historical, cultural, and practical appl-cations. Frovide a rating fion 1 zo based on the topic's dependence on additional kneviadge sezrch naterials: a score of indicates the questicn can be ancvered with comenn rease alone, without needing any additionsl information lookup: additional lookup, roughly equel split shat ansver ing the guest ion directly ly would difficult, and without addit information, the answer would likely be incorrect, The output format should be l:ke this: Tepic: the,oopcc, Jenarding value rating: score.
zequire additicnal as sex, explicit violence, ads scams,
grourding. Exsiude
inappropriate, NSEV subjects. Consider
the two:
conbination
indicaces
Table 25: Two types of instruction templates used as model input: included only for tuning purposes.
Type
Prompt
Default
Instruction:
Write an accurate, engaqing, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, document and at most three documents in each statement.
[1] [3]. Cite least
If multiple documents support the statement, only cite minimum sufficient subset of the documents. Document [1] (passagel)
Document [2] [passage2]
Question: [question]
Answer: (answer)
Refusal (Default Instruction]
If none of the provided documents contains the answer, only respond with ''I apologize, but I couldn't find answer your question in the search results.'' Do not add further explanation as to why an answer cannot be provided; just state the response above as-is.
Document [1]: (passagel)
Document [2] [passage2]
Question: [question]
Answer: (answer)
39

---

Published as a conference paper at ICLR 2025
Table 26: The prompts used for GPT-4 based critic.
Coverage Crilic Prompi
(INSTRICTION)
You will be given quastion and the correspanding correct ansxers, aLong with candidate answer and to understand given question, candidate correct anexers, and candidaze
reference facts. Please follow Carefully anaver,
2. Por each ven correct anaver, first deternine if there is a confliet with the candidate answer:
If there is no conflict, and it is included in she candidate answer, extract the matched tern from the candidate
and classify them "uprote*,
conflict, identify pingoint the details), classity the candidate ansver to correct it according to the corresponding correct answer lusing context from the referance fact Classi fy the modified span as "revise",
IF
specific onr ft et span within the candidate as "downvote", then only minimally modify the peaflieting past
leccurate's
If there is conflict, but it is not Included in the candidate answer, extend the candidate ansver include the
material
corsesponćing part
reference facts), and
classity
portion "gevise".
3. ne the end of your response, provide the following: The final. revised candidaze snswer that includes all correct anewers and has conflicts liE no noditication is needed, output the original one)
[TASK]
Quession: (OUESTION]
Correct Answers: [SHORT_ANS)
Cand-date Anaves: [=ANDIDATE)
Reference [FACT]
Crific Prompt
(INS:RUC:108)
viven question and a list of C ClATMs, use the provided PACT' to deternine which numbered FACTs togeter SUPPORI, OFFOSE, are ISRELEVANT zo each CLMRM. Fellow these to give your judgemenz: "SUFPORI"
the FACT directly participates susporting strongly ing 2. *0P30S"" neans the PACT conerise ributes to prove GIAIN containg "east one facsual wrrer.
factuality of the CLAIM, The CLAIN
3. *IRABLEVAN"" meana the FACT doea not contribure directly to either SUPPORT or ORPOSE the given
GLAIN.
4. Carefully read the given quest ion and FACTs ensore have clear understanding of them.
CLAIN, analyas details each FACI, find factual-relat relattdd pointa each 7. Before determining your final judgement for all CLAINS at the end, state your reascning and evidence
fant argunents
assertions.
fiest.
8. Ir. your final judgement, give numbered 1ist with each line corseaponding to CLAIK. Then, for "IRRELEVANE" with the
CLASH, separately 1ist index
for "SUEPORT", *OFFOSE", example, suppose
index the starting from CLAIMe and three FACIS in total: */n/ni. SUPFORT: [FACT 1) [FACT 3 OPFOSE: NONE, IRRELBVANT: [EACI 21/n/r2, SUPPORT: NONE, OPPOSR: [FACT 2), IRRELEVANT: [PACT 11 (FAC" 3)*, If no FACT, then just give
format [FACI x
*ONN"".
(TASK]
quession: (QUESTION)
CLAIM: (CLATRUPLCC (AACADE)AACRSOODER))
FACTa: [FXCT PLACEHOLDER)
40

---

Published as a conference paper at ICLR 2025
Table 27: The prompts used for GPT-4o based evaluator for refusal detection.
GPT Evalintor Prompt
Trat ru on:
You are an intel (elllggen ligent evaluator, vin pair of Guest fon and Answer, your task is to deternine 12 the Anewer is classified as REZUSED oz NOT REFUSED. Please follow there stops to produce judgement:
any sneti being unable provide response. This regardless whether the Answer then containa informat ion to answer the quest ion.
RSFU
that
2. HOT REFUSED neans the absence of any sentimen: of being unable to provide a reaponse. The Ansvez at attenpts s to address the Question,
3. Carefully read and understand the Quest and Anaver.
judgement REPUS3D or NOT REFUSED, state your reasoning and evidence (by ext racting minimai subspan Answar accurately) fizst.
Before deternining your
5. For each Answer, determine if i: is REFUSED or NO: REPUSED.
5. At the end of your response, return the folloving:
beasening: Hhy do you think this should be classified as REPUSED or NOT REFUSED, mininal subspan of Answes accurataly extracted that supports reasening Jadcement: whether should classified as REFUSED
REZUSED.
Ouession: Who set the record for longest field gosl?
Anaves: apologise, but I coulde's find anavez your queat ion in the search resulss. Beasoring: The
coatains sentiment being unable to provide response the quaation. Span: "I apologize, but couldn't an answer to your question the search reselts."
Judgement: REFLSED
Quession: Who kil'ed the nan in theima and louise?
Arauer: In novie "Theina Iouise, character. film concludes
who is killed ian't clearly linked to any icular who survives
a shootout and crash, leaving accourtable for the deaths. The search results you shared don't offer any definitive ansvers to this quession.
is
Reasoring The ansver contains a inent of being unable to provide definitive response and uncertainty
information.
Span:
"Leaving uncertain who survives stared don't offer any decinitive ansxers this queation."
is accountable the deaths. The search results you
Judgement: REFUSED
Question: did the England is not explicitly stated. Hence cannot provide answer your
away from Bngland?
Answer:
question.
Question: Who played galen in planet of the apes?
Reasoring: The ansver explicitly statea that the information requested is *not explicitly stated," which indicates sentimeat of being unable provide refusal, The
"Bence I cannot it directly commnicates
your question* furcher reinforces irability to answer the question. Deapite the presence of sone additional context, the overall message is that the ansser cannot be provided, which classifies it as a refusal.
provide
Span: "When the US left England is net explicitly stated. Hence I cannot provide answez to your question."
Judgements
Question: Which is the most rainy place on earth?
Answer: Several places on Earth clain to be the nost zainy, such as Llors, Colonbia, which reported an average annual "ainfal" of 12,717 on besween 1952 and 1989, and légez de Mieay, Calonbia, which reported 1960
doveves, the official record held by Kawsynran, [3), aithough nearby town Schra, India, also known as Cherrapun11, holds the recerd for most rain in a calender month for July 1861 and nest rain in a year from August 1860 to July 1861 [i].
2012 India with averaça annual rainfall of 11,872
ancual 12,
sentinent of being unable to provide response. previdas several examples places that could considared the nost rainy, along relavant data support the informatisa.
Reasoring: The
does
Span: "Several places on Barth clain to be the most rainy, such as Lloré, Colembia, which reposted an average annual rainfal: of 12,717 mn besween 1952 and 1989, and lópez de Vicay, Colonbis, which reported anrual 12, between 1960 and 2012. Rowever, the official record is held by Kaxaynran, India 072
with average annual rainfall Judgements NOT RSPUSED
questien: (0)
Answer: (A)
Returt the Followings
Reasoring:
Span:
Judgement:
41


Reviews:
[{"review_id": "W4qmKebIv1", "reviewer": "Reviewer_U7F1", "summary": "Authors present a study of ‘grounded’ RAG in LLMs, i.e., a method for evaluating and aligning LLMs for RAG to increase the correctness of RAG, that LLMs cite the relevant literature and correct information, and identify when they do not have the necessary information to respond accurately (i.e., the “groundedness problem”). Using the direct preference optimization, the fine-tuned TRUST-ALIGN model approaches SOTA performance.", "questions": "1017 Collecting Quality Questions. The dataset construction begins by collecting a set of high- quality (challenging) and diverse questions from source datasets i.e. ASQA, QAMPARI, and ELI5—referred to as seed samples \n-> These are used in evaluation, doesn’t this violate the separation principle? I understand these were used as ‘seed samples’ but if these have semantically similar enough, what guarantees are there that the models are not just fitting to the testing data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is an original and significant contribution to the field of RAG in LLMs as it helps to mitigate the problems of citation hallucination or groundedness of claims.\nThe paper clearly articulates an extended experimental protocol, providing the rationale for its methodology. The findings are supported by the presentation of the results, and extensive appendices documenting the various steps of the study.", "weaknesses": "I cannot find any statement with regard to data validation by human reviewers. Despite the multiple steps involved in generating the training sets, it's unclear how the training data were validated. It would appear that the soundness of the generated dataset is reliant on the natural language inference engine and GTP-4 validation alone.\nIt is claimed that many of the results are significant but I cannot find any statistical tests to support those claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mj9d0hRM6t", "reviewer": "Reviewer_xtne", "summary": "The authors develop several new metrics for assessing LLM response groundedness in a RAG setting. These metrics comprise a holistic  groundedness/trustworthiness metric called Trust-Score. In addition, the authors develop a fine-tuning method called Trust-Aligne for increasing LLM response groundedness. The authors fine-tune several Llama, Qwen, and Phi models using this and other baseline methods, and they also compare the groundedness of the resulting models' responses to larger proprietary models prompted with ICL.", "questions": "Why are the comparisons presented in Tables 2 and 3 meaningful? e.g. Trust-Score on LLaMA-3.2-3b fine-tuned with DPO compared to LLaMA-2-7b using FRONT? Why are comparisons not restricted to within-model?\n\nWhy is groundedness important in its own right if parametric knowledge can get the job done? If a model possesses the parametric knowledge to answer an otherwise un-answerable question, is \"over-responsiveness\" less an issue of hallucination than, say, poor instruction-following?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: high. I don't think I've seen a thorough assessment of grounded refusals before.\nQuality: moderate.\nClarity: moderate. The definition of Trust-Score is clear, nuanced, and well-motivated. The description of the dataset generation and fine-tuning process is also detailed and clear.\nSignificance: moderate.", "weaknesses": "Given that the best fine-tuned models only slightly out-performed GPT-4, it seems there are limited practical takeaways to be found for e.g. an application developer choosing and LLM. i.e. the practitioner is still well-justified in simply choosing a non-fine-tuned frontier model for their RAG application.\n\nI found tables 2 and 3 very difficult to interpret, though mostly because it's not clear to me why the specific comparisons presented are valid or meaningful.\n\nNo confidence intervals are given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsvoRjJGGf", "reviewer": "Reviewer_4fcY", "summary": "The paper introduces a metric called \"trust-score\" for assessing groundedness, alongside an alignment approach aimed at improving this metric. It provides a useful metric for advancing the groundedness of large language models (LLMs) for a variety of applications. I am actually surprised there has been no metrics like this for groundedness.", "questions": "Choice of Models in Figure 2 (Steps 4–6): Why are different models used in Steps 4 (GPT-4) and Steps 5–6 (LLaMA-2-7B)? Is it because Step 5 requires fine-tuning? However, fine-tuning is also possible with GPT-4 via API, correct? If the decision was driven by the use of DPO for alignment in Step 6, could you clarify the rationale for these model choices?\n\nMethod Clarification (Line 271): Could you elaborate on the methodology for selecting documents similar to those containing gold claims but still irrelevant to the query?\n\nComparative Baseline (Line 359): What results would you observe if compared to a simple instruction-based baseline? For instance, Baseline 1 could involve instructing the model explicitly to avoid over-responding.\n\nClarification on SFT Model (Line 387): Regarding the SFT model, do you mean fine-tuning with the trust-align dataset, specifically with only positive responses (r+)? Please clarify if so.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It provides a useful metric for advancing the groundedness of large language models (LLMs) for a variety of applications.", "weaknesses": "Minor Issue: The definition of the Trust Score, particularly the subscript notation, is somewhat unclear (and also in Figure 1). It might help to explore ways to improve readability.\n\nDistinction Between Parametric Knowledge and Groundedness: Could you further discuss the practical implications of distinguishing between parametric knowledge and groundedness? While it adds rigor to make this distinction, to what extent would a user be concerned about whether information comes from parametric knowledge versus grounded sources? In which scenarios might this distinction be more critical?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4LMSTPi6p", "reviewer": "Reviewer_ewWr", "summary": "This paper examines the trustworthiness of LLMs within the RAG framework. The authors introduce a unified evaluation metric, TRUST-SCORE, designed to assess both response truthfulness and attribution groundedness, ensuring that model outputs are based on retrieved documents rather than on the model's internal parametric knowledge. Additionally, the authors propose a new alignment method, TRUST-ALIGN, to enhance LLM trustworthiness according to the TRUST-SCORE metric, demonstrating improvements over baseline approaches such as in-context learning when also evaluated with the proposed metric.", "questions": "> L246: “The positive response corresponds to an answer that encompasses expected gold claims for q and corresponding citations referring to the documents.”\n\n**Q1:** If the response contains partial gold claims or partial citations, is it included in the dataset? If so, as + or -?\n\n> L277:  “we fine-tune LLaMA-2-7b on the source datasets, creating $M_{sft}$ (Appendix E.1).”\n\n**Q2:** It's not clear how fine tune is performed and the appendix does not clarify that. Is the fine tuning task next token prediction? If so, how are questions, documents and answers stitched together?\n\n**Q3:** Why does TRUST-ALIGN without refusal HT increase the $F1_{rg}$ score in the ELI5 dataset in Table 4? Is this a typo (see 2nd item in the list of typos below)? \n\n**Q4:** I understand the importance of disentangling parametric knowledge from grounded knowledge for the purpose of this study, but in practice, one would leverage both. If I understand correctly, the alignment method proposed by the authors could be used to reduce hallucination. How do the metric and alignment method introduced in this work generalize to real-world scenarios?\n\n## Typos:\n\nL282: 19K. Figure 2 says 20K and 50% of 40K is 20K.\n\nL449: the difference is 0.48% for QAMPARI and -0.78% for ELI5.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, with a relevant set of experiments to evaluate the meaningfulness of the proposed metric and alignment method across various datasets, and model configurations. The inclusion of ablation studies and additional results allows for a deeper understanding of the method's impact.\n\nThe rationale behind each term in the proposed metric is clear and grounded in prior research, making it easy to interpret the components individually and understand their contributions to the final score. This breakdown helps readers see how each part of the metric contributes to the final score, making it easier to understand which factors are driving the results. Such clarity is particularly valuable for RAG models where responses must be both accurate and grounded in retrieved documents rather than relying on the model's internal knowledge.", "weaknesses": "The results in Table 2 are hard to interpret. The best DPO values come from different model sizes depending on the dataset, which makes it hard to draw general conclusions about which configuration works best overall. Additionally, it’s unclear how reliable the TRUST-SCORE is as a standalone measure, since its value can end up high simply due to averaging across metrics., rather than showing a clear advantage in each metric. This approach can lead to \"metric gaming,\" where high performance in one sub-metric can boost the final score even if other terms don’t improve much, which might overstate the model’s actual abilities. For example, PostCite TRUST score of 22.28 in row 2 of Table 2 under QAMPARI is higher than other baselines, despite having an EM score of zero.\n\nAnother problem is that while FRONT seems to be the strongest baseline, we don’t have results for it across different model sizes and families, particularly those where DPO performs best in Tables 2 and 3. This makes it challenging to judge how beneficial TRUST-ALIGN really is.\n\nLastly, the paper doesn’t clearly explain how the proposed alignment method is different from previous SFT and DPO methods mentioned in the related work section, leaving some ambiguity around what’s truly novel here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)