Teaching Language Models to Hallucinate Less with Synthetic Tasks
Authors: Erik Jones
Hamid Palangi
Clarisse Simões Ribeiro
Varun Chandrasekaran
Subhabrata Mukherjee
Arindam Mitra
Ahmed Hassan Awadallah
Ece Kamar
Venue: ICLR 2024 poster (2024)
Keywords: hallucination
large language models
synthetic data
prefix tuning
safety
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We use synthetic data to tractably optimize language models to hallucinate less

Abstract: Large language models (LLMs) frequently hallucinate on abstractive summarization tasks such as document-based question-answering, meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing to make LLMs hallucinate less is challenging, as hallucination is hard to efficiently, cheaply, and reliably evaluate at each optimization step. In this work, we show that reducing hallucination on a _synthetic task_ can also reduce hallucination on real-world downstream tasks. Our method, SynTra, first designs a synthetic task where hallucinations are easy to elicit and measure. It next optimizes the LLM's system message via prefix tuning on the synthetic task, then uses the system message on realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, we reduce hallucination for two 13B-parameter LLMs using supervision signal from only a synthetic retrieval task. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively _increase_ hallucination. Overall, SynTra demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice.

Published as a conference paper at ICLR 2024
TEACHING LANGUAGE MODELS TO HALLUCINATE LESS WITH SYNTHETIC TASKS
Erik Jones, 14 Hamid Palangi, Clarisse Simões, Varun Chandrasekaran, 34 Subhabrata Mukherjee,* 4 Arindam Mitra, Ahmed Awadallah, Ece Kamar² UC Berkeley Microsoft Research UIUC Hippocratic Al
ABSTRACT
Large language models (LLMs) frequently hallucinate on abstractive summariza- tion tasks such as document-basse question-answering -anwerrnng meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing LLMs hallucinate less these tasks is chal- lenging, as hallucination is hard to efficiently evaluate at each optimization step. In this work, we show that reducing hallucination on a synthetic task can also re- duce hallucination designs a synthetic task where hallucinations are easy to elicit and measure. next optimizes the LLM's ' system message via prefix-tuning on the synthetic task, and finally transfers the system message to realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, SYNTRA reduces hallucination for two 13B 13B-paramen parameter LLMs using only a synthetic retrieval task for supervision. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination. Overall, SYNTRA demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice,
weal-world downstream tasks. Our method, SYNTRA, first
INTRODUCTION
Large language models (LLMs) are prone to hallucinate-i.e.. fabricate entities, details, or other content-when generating responses to queries, even when all salient information is included in context. For example, LLMs can make up citations (Liu et al. 2023), add titles when generating biographies (Min et al. 2023), invent new product attributes when advertising (Koto et al. 2022 To confidently deploy LLMs, we need methods to monitor and reduce these hallucinations.
Unfortunately, directly reducing L.M hallucinations on real-world tasks is challenging, in part be- cause we cannot scalably evaluate hallucination during optimization. To exhibit this evaluation challenge, suppose the LLM generates the fictional noun "fixed liability response" when summa- rizing a meeting. Cheap-to-run rule-based manual verifiers would struggle to (i) decide that this term is worth checking, and (ii) identify whether the term real or fabricated. On the other hand, using humans or LLMs to detect hallucination in long- form outputs is slow, expensive, and error- prone (Guerreiro al. 2023). It is thus difficult to directly optimize against LLM hallucinations with gradient descent or reinforcement learning, as we cannot efficiently evaluate the loss or reward.
In response, we introduce SYNTRA, method that uses synthetic data reduce LLM hallucinations (Figure I. SYNTRA first designs synthetic task where hallucination can be efficiently and tractably evaluated. then exploits this tractability by optimizing the LLM system message on the synthetic task via prefix-tuning (Li & Liang 2021), and finally transfers the system message to realistic tasks.
The core component of SYNTRA the design of the synthetic task, which provides the only direct supervision signal that captures hallucination. For this supervision signal to be sufficient, argue that the synthetic task must at least satisfy two properties: (i) LLMs should hallucinate frequently on the task, and (ii) hallucination ensures that optimizing on the synthetic task teaches the model to hallucinate less, while the latter
be cheaply and automatically evaluated the task. The former
"Corresondencee to er
jojjees@berkee bewseeebrreee...... edd edu and hpalang @mi crosoft.com
Work done Microsoft Research.

---

Published as a conference paper at ICLR 2024
Design synthetic task intractable to evaluate
Optimize synthetic task system message postfix
Transfer system message
Tractability evaluating hallu.
Use learned postlix for realistic tasks
Systen
Relyful Asshtart
Prompt:
Prompt: Summorize Meeting Transcript
Prompt: Generate Divinal Report
Promp:: Given Documents
The followingg is list of names: Elss Smothh, Joe Jomes, Eilo Towns Return all
LLM
with
Tractable to evailaate
Synthetic
Task
LAM Outgat: Ells Smith JonorJJames, Lllo Towns
Meeting
Tasks
Clinical report generation
summarization
Figure I: Overview of the SYNTRA framework. We first define a synthetic task where hallucination easy to tractably evaluate. Next, optimize the LLM system message this task by learning continuous postfix via prefix-tuning. We then transfer the learned system message across real tasks.
makes optimization tractable. Throughout this work, use the names retrieval task as the synthetic task: given a list of random names, we prompt the L.M to retrieve the first n names on the list that start with some letter. We say the model hallucinates if it generates name that is not the list.
SYNTRA then optimizes the LLM to reduce hallucination on the synthetic task. To do so, we optimize the LLM system message (e.g.. "You are a helpful Al assistant") by appending a continuous postfix to it, then optimizing the postfix. We optimize the system message rather than the whole model learn high-level instructions for how to hallucinate less, which we expect to transfer well. We additionally optimize the LLM to keep its output constant on a set of reference prompts, so the LLM does not latch onto spurious attributes of the synthetic task.
We find that SYNTRA consistently reduces hallucination across models, tasks, and metrics. We evaluate Vieuna v1.1 (Chiang et al., 2023) and Orea (Mukherjee al. 2023 on three realistic tasks: search-and-retrievee meeting summarization, and clinical report generation. Following, Yue et al. (023),, we measure hallucination with GPT-4, and find that optimizing the system message consistently reduces hallucination: on Orca, SYNTRA reduces the hallucination rate by over 7 points on average and 16 points on specific tasks. In contrast, fine-tuning the whole model can increase the hallucination rate. We also test whether SYNTRA reduces hallucination according other metries, and find its outputs (i) overlap more with references and (ii) contain fewer ungrounded entities.
SYNTRA is not without limitations; it requires designing a synthetic task, and reduces hallucination on some models more than others. Nevertheless our work provides encouraging evidence that we can use synthetic data to isolate and optimize against undesired LLM behaviors.
THE SYNTRA PIPELINE
We describe SYNTRA (Synthetic Transfer), our method reduce hallucination on abstractive sum- marization tasks using synthetic data. We outline the setup (Section 2. introduce synthetic tasks (Section 2.2). then describe how SYNTRA reduces hallucination on realistic tasks (Section 2.3
2.1
SETUP
We study LLMs that take prompts (c.g., "Write clinical report about the following dialog...") ") and produce long-form outputs (e.g., "Overview: Patient has..."). The behavior of the LLM is modulated by two features: system message that provides high-level instructions (e.g.. "You and honest Al assistant)), and the raw model weights. We thus represent the LLM as LLM a. a composition of the pystemfunction Sy that appends the prompt to the system message & and the raw LLM so that generates text based on weights 0. Formally, LLM2.0 So o 84. This decomposition into system messages and prompts is standard for many instruction-tuned L.LMs (OpenAl) 2023).
helpful
We measure hallucination abstractive summarization tasks, where models respond to queries given some context. We model an abstractive summarization task T (e.g., generate clinical reports
While the system prompt ø is usually text, this is challenging to optimize. Following Li & Liang 2021 we append continuous embeddings to the text, which be optimized via gradient descent.
2

---

Published as a conference paper at ICLR 2024
from dialogues) as a distribution over prompts D,, where each prompt p can be decomposed into query that is appended to context c, i.e., p = c 9. For example, the context c might be a dialog between patient and doctor, and the query might be "Write clinical report containing assume that the context e always has all required information to correctly respond to query q.
We
Hallucinations in the abstractive summarization setting are grounding errors: errors where the output contains some information that is not supported by the context. We define the hallucination function h, which takes in a prompt and output and returns if the output is supported by the prompt, and otherwise. The hallucination rate Eaal on task T for model parameters and system prompt ø is
(o, 7) = -EpDD h(p, LLM
where the expectation taken over the distribution D, and sampling during LLM decoding.
The hallucination function h is at best expensive to evaluate for many different prompts and out- puts, and at worse intractable. This is especially true for abstractive summarization, as even humans struggle read and remember long contexts while evaluating generated summaries (Guerreiro) et al. 2023). To measure hallucination in general, we rely on proxy metrics /prox that can use refer- ence oulputs or external databases to approximate h. Existing work has considered many possible proxy metrics including comparing to ground truth outputs (Nan et al. [021). decomposing outputs into atomic facts and evaluating them separately (Min et al.) 2023), using entailment models (Roit et al. 2023), or using LLM judges Yue et al. 2023
Gao et al. (023a).
2.2 SYNTHETIC TASKS
natural approach to reduce hallucination is to optimize against the hallucination rate haa over all tasks simultaneously, but this is intractable; the hallucination function h is expensive to evaluate at scale, which rules out direct optimization. Optimizing against proxy metrics hprox also problem- atic, as they may (i) unfaithfully capture hallucination and (ii) still be expensive to optimize.
Rather than trying to optimize a proxy in general, we instead optimize the true hallucination rate exactly carefully constructed synthetic task. Concretely, such that the hallucination function restricted to this task, h p is easy to automatically evaluate and optimize. For example, suppose the prompts contain random first and last names (context) and ask the LLM retrieve names that start with some letter (query). We can test if the LLM hallucinates by checking whether the each name it generates is in the prompt, and optimize to reduce the corresponding hallucination rate directly
construct synthetic task Tsyn
We seek synthetic tasks Tay that satisfy two desiderata: (i) the LLM hallucinates frequently on the task, and (ii) we can test for hallucination on the task efficiently, The former means optimizing on the synthetic task teaches the model to hallucinate less, while the latter makes optimization tractable.
2.3 REDUCING HALLUUINAATIN WITH SYNTRA
We now present the entire SYNTRA pipeline. SYNTRA defines a synthetic task (Section 2.2 opti- mizes either the system message or model weights the synthetic task (with regularization), then transfers the learned system message or model weights to realistic tasks. We detail each step below.
Optimization. To reduce the hallucination rate the synthetic task, SYNTRA either optimizes system message & or model weights 0.
Optimizing the system message á is challenging since the of possible system messages is discrete, precluding continuous optimization with gradient-descent. To circumvent the discreteness of the system messages, we adapt the prefix-tuning method from (Li & Liang 202I] to learn a continuous postlix to the system message. Specifically, we exploit the fact that while prompts are discrete, the LLM maps them to sequences of continuous embeddings during inference. To optimize the system message, we append a continuous postfix to the embedded discrete system message, then optimize this postfix with gradient descent. Intuitively, the optimized system message provides "high-level instructions" that we expect to extrapolate better. See Appendix A for details.
In this case, we add constraints until there is unique non-hallucien hallluclliaeddd output (e.g.. names are outputted in order comma-separated list) to avoid averaging over the combinatorially many non-hallucinated outputs. We could efficiently optimize other definitions (c.g., g... any name not on the list) with zeroth order methods.
3

---

Published as a conference paper at ICLR 2024
Optimizing the model weights 0 is easy since they are continuous; standard fine-tuning
Regularizing with reference data. Reducing the hallucination rate on the synthetic task may trans- fer well out-of-the-bos but could potentially pick up task-specifie spurious that only appear in hallucinated outputs in the synthetic task, but can appear in correct outputs in general. For example, models may only output newlines when they hallucinate on the synthetic task, but frequently output newlines general on realistic tasks. Optimizing on the synthetic task may lead the model never output a newline, which can compromise performance.
To mitigate the effect of these spurious attributes, we optimize the model to preserve its outputs over reference distribution. Specifically, for reference distribution Dier of prompts, original system message Pog. model weights Pog. and KL-divergence f, we define the reference loss fet as
Lee( 0; Dref Oog: Oog) == DD- (LLMs (p), LLM Mug (p)).
Training on the reference data helps combat spurious attributes that can take new values on that data. While LLMs may hallucinate on the reference data (and potentially learn that the true hallucination features are spurious), we find empirically that this is not an issue (Section 3.2). Optimizing with reference data resembles the PPO-ptx method from Bai et al. (2022) which mixes in pretraining gradients when optimizing reward model.
Full loss function. We optimize convex combinations of the two losses, i.c.
Eapt 0; Tayn: Oog, Dref, a) == atfaa (Φ, 0; Tayn) + (1 - arrr Drefi Oog Oog),
(3)
where [0, 1] is a hyperparameter; setting a 1 optimizes on the synthetic task, while ar optimizes to preserve the original model. We then use the learned parameters on realistic tasks.
3 EVALUATING SYNTRA
We next present an empirical validation of SYNTRA. We describe the setup (Section 3. show how SYNTRA reduces the hallucination rate realistic tasks (Section 3.2 test output quality and other hallucination metrics (Section 3.3 and isolate the impact of the reference data (Section 3.4.
3.1 SETUP
LLMs. We primarily evaluate SYNTRA on two LLMs: 13B-parameter Vicuna 1.1 (Chiang et al. 2023 and 13B Orca (Mukherjee et al., 2023). Both are fine-tuned from Llama 13B Toovrro et al., We also run subset of our experiments on Llama-2 (Touvron et al.) 20236) Appendix(B.8)
2023a).
Optimization details. We optimize the system message using prefix tuning (Li & Liang 2021) and the entire LLM with standard fine-tuning. When optimizing with reference data (denoted Synth. Ref), we set the factor in Equation (3) to a 0.5, and set = 0 for just the synthetic data. We only hyperparameter tune Orea on separate MS MARCO validation data, and reuse the hyperparameters all other Orca tasks and all Vicuna tasks; see Appendix B. for compute and hyperparameter details.
3.1.1 REALISTIC TASKS
We study how well SYNTRA reduces hallucination on three realistic LLM use-cases: search-and- retrieve (MS MARCO), meeting summarization (QMSum), and automated clinical report generation (ACI-Bench). Further details for each dataset and full prompts are in Appendix B.2
Search and retrieve (MS MARCO). We first study hallucination in search-and-retriee applica- tions, where the L.M must generate an answer from retrieved documents. We use MS MARCO as a source of examples (Nguyen et al. 2016), where the task to answer real-user Bing queries given 10 retrieved passages. For computational tractability, select 1000 random queries from the MS MARCO validation set that require a long-form response (as labeled in the original dataset).
Meeting summarization (QMSum). We next study hallucination in meeting summarization ap- plications, where LLMs are given a meeting transcript and asked to summarize aspects it. We the QMSum dataset as source of examples (Zhong et al. 2021). QMSum contains meeting transcripts and questions (e.g., "What did the group discuss about budget balancing?" We filter the QMSum train set for entries that fit in the L.M context window, for a total of 852 examples.
4

---

Published as a conference paper at ICLR 2024
Orca
Vicuna
Hallucination rate
1.0
0.8
0.6
0.4
0.2
0.0
Inexact match
New name
Inexact match
New name
Original
Sys., synth.
5ys., synth. ref.
Model, synth.
Model, synth.
Figure 2: Hallucination rate the names retrieval task the original LLM (Original) when op- timizing the system message (Sys.) or full LLM weights (Model) on either just the synthetic data (synth.) mixture of synthetic and reference data (synth. + ref.). We measure the hallucination rate with respect to the exact correct output (inexact match) and the names on the list (new name).
Automated clinical report generation (ACI-Bench). Finally, study hallucination in automated clinical report generation. We use ACI-Bench (Yim et al.] (202)) as source of examples, which contains dialogs between doctors and patients. Given dialog. the task is to generate clinical report with four specific headings. We use all 207 examples from the published dataset.
3.1.2 SYNTHETIC TASKS
Synthetic task (names retrieval). We define the names retrieval task, where the LLM needs to retrieve certain names from given list. For example, we might prompt the model with the following:
Prompt:
The following is list of names [Names]
List the first 5 names where the first starts with E in the order that they appear. Include both the first and last name in the response. If there are not 5 names that start with E, return all of the names in the list that start with in the order that they appear.
We generate dataset of 100,000 examples and test for hallucination by checking whether any name in the output does not appear in the original list. When optimizing to reduce hallucination, to use first-order methods, we optimize the log-likelihood of the unique output allowable by the prompt; in Section 3.2 show that doing so also reduces the rate of hallucinated names even among outputs that are not exact matches. See Appendix B.3 for details on the dataset construction.
Other synthetic tasks. Before trying the names synthetic task, tried reversing the words in a sentence, and "splicing" pairs of sentences by alternating words from each. We found that while LLMs performed poorly these, they produced outputs that did not match intuitive definition of hallucination (e.g.. when splicing sentences, LLMs frequently just concatenate the two sentences). We did try other synthetic tasks; there are likely others that lead greater gains.
Reference data. For the reference data Dref, we use SQuAD (Rajpurkar et al. 2016) as a source of 50000 prompts. For each passage, ask the LLM to respond the associated query, and for half we ask it to explain its reasoning or think step by step. We include details in Appendix B.4
3.2 REDUCING HALLUCINATION WITH SYNTRA
In this section, measure whether SYNTRA teaches LLMs to hallucinate less. We first verify that SYNTRA reduces the hallucination rate in-domain on the synthetic task, then measure whether reduces the hallucination rate oft-of-domain on realistic tasks.
Testing in-domain hallucination reduction. We first want to verify that our names retrieval task satisfies desiderata for synthetic task, and that optimizer works. To do so, want verify
5

---

Published as a conference paper at ICLR 2024
Model
Parameters
Data
MS MARCO
QMSum
ACI-Bench
Average
Original Orca
12.2 + 1.0
18.1 1.3
47.0 + 3.5
25.8
Synthetic
23.7 + 1.4
24.8 + 1.5
45.4 + 3.6
31.3
Orca
Full model
Synth. Ref.
1
28.0 1.5
47.0 3.5
31.5
Synthetic
9.9
1.3
44.8 3.5
23.6
Sys. message
Synth. Ref.
10.5 1.5:1.0 1.0
15.8 1.3
28.6 3.3
18.3
Original Vicuna
26.2 + 1.4
31.4 1.6
50.2 + 3.5
35.9
Synthetic
51.3 + 1.6
56.9 + 1.7
06.3
58.2
Vicuna
Full model
Synth. + Ref.
21.1 + 1.3
29.4 1.6
3.5
33.7
Synthetic
26.2 + 1.4
35.7 + 1.6
83.1 + 2.6
48.3
Sys. message
Synth. Ref.
24.6 + 1.4
33.3 = 1.6
4 33.4
Table 1:
Hallucination rate (%) measured by GPT-4 across all models, optimized parameters, and tasks (lower better). We compare against the original model, and optimize the full model or system message, using just synthetic data, or both the synthetic and reference data (SYNTRA).
that the unmodified LLM hallucinates frequently on the names retrieval task, and that optimizing on the names retrieval task reduces hallucination new in-distribution instances.
In order to test hallucination on the names retrieval task, we measure whether (i) the generated answer matches the unique correct answer, and (ii) the LLM only generates names on the list.
We report full results in Figure 2 and find that the names retrieval task satisfies our desiderata. Both Vicuna and Orca hallucinate frequently only generate names on the list less than 17% of the time. Moreover, our optimizer works in-domain; optimizing either the system message or the entire LLM reduces hallucination across all measures.
this task; they exactly correct less than 1% time, and
Testing hallucination reduction on realistic tasks. We next test whether reducing hallucination on the names retrieval task reduces hallucination on the realistic tasks from Section 3.I.1 We also study how important the reference data and optimization parameters are for transfer performance.
To measure hallucination the realistic tasks, following Yue et al. (2023;; Gao et al. (2023a) [ We prompt GPT-4 with the context, query. and LLM-generated output and ask whether the factual information in the output is grounded up to paraphrases. GPT-4 retrieves any spans that are not grounded, then scores the groundedness of the response from 0 10. We say the LLM hallucinates if the returned score is 10, i.e., there exists some ungrounded span and thus some fabricated content. See Appendix B.5 for further details.
GPT-4
evaluator OOpenAl 2023)
We find that for every realistic task and LLM, SYNTRA is able to reduce hallucination by optimizing on a mixture of the names and reference data (Table D). On average, optimizing the system message decreases the hallucination rate by 7.5 points on Orca (a 29% reduction), and 2.5 points on Vicuna (a 7% reduction). For specific applications, the decrease can be larger; for example, SYNTRA reduces the hallucination rate on ACI-Bench by 16 points on Orca, and 8 points on Vicuna.
Additional analysis. We next discuss which components are necessary for SYNTRA to transfer to realistic tasks, including the optimization parameters, reference, and L.M,, in further detail.
System message LLM weights. Our results demonstrate that optimizing the system message in- stead of the whole LLM sometimes necessary to reduce hallucination, even though fine-tuning is strictly better in-domain. On Orca, fine-tuning the full LLM actually increases the hallucination rate across most tasks, while optimizing the system message produces consistent reductions. On Vicuna, fine-tuning reduces hallucination on two out of the three tasks, but is slightly worse than optimizing the system message on average We hypothesize that this gap exists part because fine-tuning latches onto more spurious attributes of the synthetic task, and provide evidence in Section 3.4
'GPT-4 is tractable use for our evaluation, hut would expensive to use optimization directly. 'Fine-tuned Vicuna reduces hallucination by removing many non-hallucinated entities in addition to hallu- cinated ones, which reduces the fidelity of the outputs; Table Section 3.3 for details.

---

Published as a conference paper at ICLR 2024
Model
Parameters
Data
MS MARCO
QMSum
BLEU
R-1
R-2
R-L
BLEU
R-1
R-2
R-L
Original Orea
10.5
29.4
18.6
25.3
6.2
34.3
11.1
22.6
Synthetic
14.4
35.0
23.2
30.9
6.1
34.0
11.2
22.9
Orca
Sys. message
Synth. Ref.
13.8
33.9
22.6
29.7
6.3
34.4
11.5
22.9
Synthetic
13.9
31.6
20.4
28.8
4.0
24.0
7.8
17.1
Full model
Synth. Ref.
11.4
30.1
19.0
26.3
5.9
33.4
11.1
22.7
Original Vicuna
8.7
26.0
15.5
22.
5.9
98.8
11.0
22.6
Synthetic
8.5
26.6
15.4
22.8
5.5
33.1
10.8
22.4
Vicuna
Sys. message
Synth. + Ref.
10.2
27.8
17.0
23.8
5.7
33.4
11.1
22.6
Synthetic
4.9
17.7
7.9
16.3
1.8
14.8
5.0
11.4
Full model
Synth. Ref.
11.2
29.6
18.7
25.7
5.8
33.4
10.8
22.3
Table 2: Comparison between outputs generated using SYNTRA and human-written reference out- puts. We abbreviate ROUGE-I, ROUGE-2, and ROUGE-L with R-1, R-2, and R-L. We find that SYNTRA is consistently closer to the reference summaries on MS MARCO than the original model, and is comparable on QMSum. For all metrics, higher is better.
Regularizing with reference data. Regularizing with reference data is also critical to reduce the hal- lucination rate; when optimizing the system message, for 5 out of the task-LLM pairs, adding the reference data reduces the hallucination rate. While the reference data does not significantly im- pact the hallucination rate when fine-tuning Orca, it reduces the hallucination rate by over 24 points on average when fine-tuning Vicuna. The reference data helps helps break task-specific spurious attributes, which we conjecture is responsible for the improvement (see Section 3.4)
SYNTRA reduces the hallucination rate more on Orea than Vicuna. Finally, SYNTRA consistently reduces the hallucination rate more on Orca than Vicuna, and the gains of optimizing the system message instead of the full L.M weights are larger for Orca. This partly because we only hy- perparameter tune using Orca; hyperparameter tuning separately on Vicuna was intractable due to computational constraints, but would likely lead further gains. This could also be due to differ- ence in the quality of the underlying LLMs; the original Orca hallucinates much less than Vicuna, and thus might better leverage optimized instructions, without requiring changes to model weights.
3.3 ASSESSING HOW SYNTRA REDUCES HALLUCINATION
We next aim to test how SYNTRA reduces hallucination, and particular verify that SYNTRA does not simply exploit shortcomings of the GPT-4 evaluation. To do so, we test whether the SYNTRA- generated outputs are (i) lower-quality and (ii) contain fewer specific details, relative to original outputs. These tests also provide further evidence that SYNTRA reduces the true hallucination rate.
Testing output quality. We first aim to test whether SYNTRA reduces output quality by measuring whether its outputs drift away from reference outputs. To do so, we compute the BLEU score (Pap- ineni et al. 2002). and ROUGE-1, -2, and -L scores (Lin & Rey 2004). which compare the 7--ram overlap between the LLM generated output and the reference output. We compute these metrics for MS MARCO and QMSum, as they provide high-quality reference outputs.
We include full results Table 2 and find that SYNTRA's outputs do not drift away from refer- ence outputs. In contrast, when optimizing the system message with reference data, SYNTRA has comparable scores across all metrics and L.LMs on QMSum, and actually increases all metrics on MS MARCO. These metrics also reveal that outputs GPT-4 labels as hallucinated are indeed lower quality; Vicuna fine-tuned on only synthetic data, which hallucinates the most, has the lowest scores.
What does this say about hallucination? These metrics provide an indirect signal on whether LLMs hallucinate less; outputs with hallucinated content should be less similar to the fully-grounded ref- erence outputs 5 Thus, SYNTRA's consistent improvement MS MARCO across all automated metries for both LLMs provides further evidence that it reduces the true hallucination rate.
Since hallucinated content does not appear in reference outputs, hallucination tends decrease similarity.

---

Published as a conference paper at ICLR 2024
Model
Parameters
Data
Ungrounded (J)
Grounded (f)
% Ungrounded (1)
% Grounded (0)
Original Orca Synthetic
9.9
17.2
7.8
16.0
21.4%
6.8%
Sys. message
Orca
Syath. + Ref.
8.0
16.9
19.2%
1.4%
Synthetic
2.0
5.8
79.4%
666.2%
Full model
Synth. + Ref.
6.3
15.1
36.5%
12.2%
Original Vicuna
9.4
17.6
Synthetic
4.7
11.2
49.3%
36.5%
Vicuna
Sys. message
Synth. Ref.
6.3
15.0
32.3%
14.8%
Synthetic
1.4
0.2
84.9%
99.0%
Full model
Synth. Ref.
5,3
14.5
43,2%
17.6%
Table 3: Entity evaluation between the prompts and outputs in ACI-Bench. Using NER model, we measure the number of entities that are the output that not appear the prompt (Ungrounded, lower is better), and that do appear in the prompt (Grounded, higher is better), along with the per- centage decrease in ungrounded and grounded entities relative to the original output.
Testing for detail removal. We next aim to test whether the LLM avoids hallucinating by gener- ating fewer details. To do so, we use a commercial-grann healthcare (Appendix [B5) compute all entities in the output and context on ACI-Bench. We then test for details by measuring the number of grounded entities, i.e., entities in the output that are also in the context. To test for hallucination directly, we also measure the number of ungrounded entities, i.c. entities in the output that do not appear in the context (and thus could be hallucinated),
named entity model that is optimized for
We include full results in Table 3 and find that SYNTRA does significantly reduce the number of grounded entities; it decreases the number of grounded entities that Orca generates by 1.4%, and by 14.8% for Vicuna. However, SYNTRA decreases the number of ungrounded entities by much more: by 19.2% for Orca and 36.5% for Vicuna. Fine-tuning and training without the reference data eliminate more grounded and ungrounded entities. See Appendix B.7 for further analysis.
What does this say about hallucination? These results provide a direct signal that SYNTRA reduces the hallucination rate by decreasing the number of ungrounded entities across all tested methods.
3.4 REFERENCE DATA COMBATS SPURIOUS ATTRIBUTES
We next aim identify whether optimizing on the reference data combats task dask-specifer spurious attributes, which can drive up the hallucination rate. To do evaluate spurious attribute. In the names retrieval task, all correct answers do not have newlines, so LLMs may learn that newlines are associated with hallucinations. However, never outputting newline could lead to errors when transferring; for example, LLMs may abruptly end generation before answering the query, rather than outputting newline (e.g., "The items are:" [ends]).
identify newlines
easy-to-
We report the newline rate across all methods and tasks in Table of Appendix B.6, and find that training on reference data helps mitigate the effect of the spurious attributes. Training on the names retrieval task routinely reduces the newline rate, but adding in the reference data recovers much of the drop. We measure the presence of newlines as we identified it as potential spurious attribute and could measure it easily, but reference data likely similarly combats unknown spurious attributes.
4 RELATED WORK
Hallucination. We aim to reduce hallucination in text generation systems when all salient informa- tion is included context. Text generation systems frequently hallucinate; see a general survey (Ji al. 2023a), and surveys restricted to abstractive summarization (Maynez al. 2020 Huang
et al. 2021) for examples. Hallucination is one of many documented potential risks of deploying
LLMs (Bender al. 2021 Bommasani et al. 2021 Weidinger et al. (2021).
An alternative metric would be to test what fraction of outputs have no ungrounded entities, but this pro- duces false-positives due noise in the entity model. For example, if "prescribe" appears in output, the entity model extracts "rib". in which the output is considered hallucinated whenever "rib" not in the input.
8

---

Published as a conference paper at ICLR 2024
Hallucination is hard to detect automatically. Some work measures hallucination by comparing out- puts to reference summaries using BLEU score (Papineni et al.
2002), ROUGE score (Lin & Rey, 2004). entity overlap (Nan et al. 202I). More recent work measures hallucination by decom- 2023), or uses LLMS (Yue et al.) 2023 Gao al., (023a). Another line of work suggests that L.MMs may encode whether they
posing outputs into atomic facts and evaluating them (Min et al.
hallucinating within their activations (Kadavath et al. [0222] Burns et al. 2023; Azaria & Mitchell,
2023). but this has not been scaled to abstractive summarization settings.
There are a few classes of methods to reduce hallucination. Some methods adjust the generation process, either by changing the decoding strategy (Tian et al.] [019 Shi al. (023), teaching the LLM to cite (Gao et al. (2023b), edit (Gao et al. (223a)), or abstain (Cao et al. (023) as it generates,
or incorporating knowledge graphs (Ji et al., 2023b) and external documents (Mallen et al. 2023)
One line of work aims to reduce hallucination with prompting strategies (Jung et al. [022] Zhou
et al. 2023), while another edits model internals to make LLMS more honest (Li et al., 2023), or
faithful to the context (Hernandez et al. (023). The closest line of work to ours trains models to
hallucinate less with contrastive learning (Cao & Wang 202T; Tang et al. 2022) or reinforcement
learning (Roit et al. 2023). Our work optimizes to reduce hallucination directly synthetic data.
Other work optimizes for other LLM behaviors such as helpfulness or harmlessness. They do so by learning reward function capturing the behavior with human feedback (Sadigh et al., 2017 Christiano et al. 2017), then optimizing LLMSs using this reward function (Stiennon et al. 2020
Bai et al.] [022] Ouyang al., 2022). Such work aims to learn an approximate a general-purpose
objective to optimize, while SYNTRA optimizes an exact objective on a narrower domain.
Synthetic data. SYNTRA leverages synthetic data to reduce hallucination. Synthetic data generated
by LLMs has been used to train higher-quality small models (Eldan & Li
2023: Gunasekar et al.
2023). train models follow instructions (Dubois al. 2023 Chiang et al.
2023 Mukherjee al.,
2023). and make models better at clinical text mining (Tang et al. 2023). A closer line of work to ours trains LLMs with synthetic data that comes from non- LLM sources, Sanh et al. (202T) convert existing NI. benchmarks to tasks for instruction tuning. Wei et al. (2023a) add random synthetic labels, and Wei et al. (2023b) adapt benchmarks to reduce sycophancy. The closest work to ours is Zhang et al. (2023), which aims to characterize hallucination by evaluating it on synthetic tasks.
Prompt optimization. SYNTRA reduces hallucination by optimizing the LLM system message; to do so we append continuous postfix, then optimize using prefix-tuning (Li & Liang 2021). Both Li & Liang| (202I) and [Su et al. (2022) demonstrate that prefixes can transfer well between work provides further evidence of this for generative tasks. Some work aims to optimize a discrete prompt directly to improve performance on classification tasks (Shin)
classification tasks;
et al. 2020 Wen et al. 2023): such methods could in principle be plugged into SYNTRA directly.
DISCUSSION
We introduce SYNTRA a method to reduce hallucination by defining and exploiting synthetic tasks. SYNTRA reduces the hallucination rate across suite of realistic evaluation tasks.
There are many natural ways to improve SYNTRA. We could improve the optimization method by searching for better hyperparameters for fine-tuning and prefix-tuning, testing other fine-tuning methods like LoRA (Hu et al. 2022), or jointly optimizing the system prompt and model weights. We could optimize over discrete system prompts, which are harder to optimize over but might gener- alize better. We could also choose better synthetic tasks; this could include searching for synthetic task that reduces hallucination more over all tasks, separate synthetic tasks for each general task, or mixtures of synthetic tasks. We expect many of these would further reduce the hallucination rate.
Finally, we discuss the tradeoffs between synthetic tasks and demonstrations. Synthetic tasks allow us to scalably generate lots of data and facilitates optimization, but SYNTRA induces a lossy transfer step. On the other hand, optimizing on demonstrations or preferences eliminates the transfer step and may generalize more broadly, but demonstrations are expensive to obtain, and the optimization objective can be misspecified (Casper et al. 2023). These methods may be complementary - SYN- TRA already complements existing methods reduce hallucination, Orea and Vicuna are both instruction tuned already (likely reducing hallucination). Understanding the comparative strengths of synthetic tasks and real demonstrations could help us train more reliable and safe L.LMs.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Besmira Nushi, Alex Pan, Mert Yuksekgonul, Olivia Watkins, Jacob Steinhardt, and Ruiqi Zhong for feedback on this work.
REFERENCES
Amos Azaria and Tom Mitchell. The internal state an LLM knows when its lying. arXiv preprint MIXiv:2304.13744 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly. S. El-Showk. Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, S. Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, C. Olah, Benjamin Mann, and J. Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
Emily Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchel. On the dan- gers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2021.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Pe- ter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In International Conference on Learning Representations (ICLR), 2023.
Meng Cao, Yue Dong, Jingyi He, and Jackie Chi Kit Cheung. Learning with rejection for abstractive text summarization. arXiv preprint arXiv:2302.08551 2023.
Shuyang Cao and Lu Wang. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaél Segerie, Micah Carroll, Andi Peng. Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menlll. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.155 152 2023.
10

---

Published as a conference paper at ICLR 2024
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-souree chatbot impressing GPT-4 with (0htps://mass..r//bbgg/223333333 -033v-vuaa---- 2023.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang. Ishaan Guirajani, Jimmy Ba, Carlos
Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXhv:2305.14387 2023.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.007553 2023.
Luyu Gao, Zhuyun Dai, Pamupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. Rarr: Researching and revising what language models say, using language models. In Association for Computational Linguistics (ACI), 2023a.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models generate text with citations. arXiv preprint arXiv:2305. /4627, 2023b.
Nuno M. Guerreiro, Elena Voita, and André Martins. Looking for needle in haystack: A com- prehensive study of hallucinations in neural machine translation. In European Association for Computational Linguistics (EACL), 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann. Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2306. 11644, 2023.
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representa- tions in language models. arXiv preprint arXiv:2304.00744 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Con- ference on Learning Representations (ICLR), 2022.
Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. The factual inconsistency problem in abstractive text summarization: A survey. arXiv preprint arXiv:2104. /4839, 2021.
Ziwei Ji, Naycon Lec, Rita Fricske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wen- liang Dai, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language gener- ation. ACM Computing Surveys (CSUR), 55, 2023a.
Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu, Bryan Wilie, Min Zeng, and Pascale Fung. RHO: Reducing hallucination in open-domain dialogues with knowledge grounding. In Findings of the Association for Computational Linguistics (Findings of ACL), 2023b.
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse. Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann. Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522), 2022.
11

---

Published as a conference paper at ICLR 2024
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. Can pretrained language models generate persua- sive, faithful, and informative ad text for product descriptions? In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems (NeurlPS), 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021.
Chin-yew Lin and Marina Rey. Looking for few good metries: ROUGE and its evaluation. In NTCIR Workshop, 2004.
Nelson Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. arXiv, 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not trust language models: Investigating effectiveness of parametric and non-parametric memories. In Association for Computational Linguistics (ACL), 2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Association for Computational Linguistics (ACL.), 2020.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of fac- tual precision in long form text generation. arXiv preprint arXiv:2305.1425/1 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv preprint arXiv:2306.0277 02707, 2023.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kath- leen Mckeown, and Bing Xiang. Entity-level factual consistency of abstractive text summariza- tion. In European Association for Computational Linguistics (EACL), 2021.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: human generated machine reading comprehension dataset. Workshop Cognitive Computing NIPS, 2016.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. J. Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Welinder, P. Christiano, J. Leike, and Ryan J. Lowe. Training language models follow instructions with human feedback. arXiv, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics (ACL), 2002.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Pupers, 2018
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ ques- tions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016.
Philippe Remy. Names dataset. http://gtthiacco////iiinnmmmmmmm 2021.
12

---

Published as a conference paper at ICLR 2024
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk. Nino Vieillard. Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, and Idan Szpektor. Factually consistent summarization via reinforcement learning with textual entail- ment feedback. In Association for Computational Linguisties (ACL.), 2023.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS), 2017.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. arXiv, 2021.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.14739, 2023.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP). 2020.
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language processing. In North American Association for Computational Linguistics (NAACL), 2022.
Ruixiang Tang, Xiaotian Han, Xianqian Jiang, and Xia Hu. Does synthetic data generation of Ilms help clinical text mining? arXiv preprint 23X1v:2303/04660, 2023.
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. Confit: Toward faithful dialogue summa- rization with aequisttialll-informee contrastive fine-tuning. North American Association for Computational Linguistics (NAACL), 2022.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. Sticking the facts: Confident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08664, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel. Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
13

---

Published as a conference paper at ICLR 2024
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc
Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint 10Xiv:2201111900 2022.
Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning in language models. arXiv preprint arXiv:2305.08220 0822888, 2023a.
Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03995. 2023b.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese. Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 2021.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint wxxix23322.33666 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' R'm Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910003771, 2019.
Wenwai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. ACI- BENCH: a novel ambient clinical intelligence dataset for benchmarking automatic visit note gen- eration. arXiv preprint arXiv:2306 02022, 2023.
Xiang Yue, Boshi Wang, Kai Zhang, Zinu Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language model. arXiv preprint arXiv:2305.063 2022.
Muru Zhang. Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534. 2023.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadal- lah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. Qmsum: A new benchmark for query- query-based multi-domain meeting summarization. In North American Association for Com- putational Linguistics (NAACL), 2021.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. arXiv preprint arXiv:2303. 1131,, 2023.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL DETAILS FOR THE SYNTRA PIPELINE
In this section, we provide additional details on our adaption of Li & Liang 2021 's prefix-tuning
method.
LLM preprocessing. We first define some prelimaries for LLM preprocessing Given a prompt the language model splits the prompt into tokens with the tokenizer function t, i..., t(p) Pi,.... .m. where p; the jth token of the tokenized prompt. Language models then map each token to continuous embedding using simple lookup table, and concatenate them. This produces an embedded prompt e(p) that is a X m matrix where d is the dimension of the language model embeddings.
Appending the postfix. To add the postfix to the system message, we simply initialize a random d x n matrix, then append it to the embedded system message form d x m matrix.
In practice, the implementations of Orca and Vicuna embed the system message and prompt jointly. To identify where the system prompt ends, we exploit the formatting they use and insert the system prompt right before the first "newline" embedding, ensuring it is placed correctly.
Optimizing the postfix. To optimize the postfix, Li & Liang (202)) introduce lower-dimensional vector that they optimize, then transform back to embedding space with a matrix. Instead, we optimize the postfix in the full embedding dimension, which is 5120 for both variants of Llama that we consider. Li & Liang (22T)) do this to stabilize optimization for smaller models; our work suggests that this might not be necessary for larger models.
B ADDITIONAL DETAILS FOR EVALUATING SYNTRA
B.1 OPTIMIZATION, INFERENCE, AND HYPERPARAMETER DETAILS.
In this section, we describe the specific experimental details used.
Compute details. We perform all of our experiments on single NVIDIA A100-PCIE-80GE GPU, except for fine-tuning, for which we use four A100s. We run inference and optimization for all models in 6 precision. We use 13B parameter models, as these were the largest that could fit in memory when optimizing the postfix.
Inference details. For all experiments (including the original LLMs with no optimization), we use the system message "You are a helpful, honest, and conservative Al system designed to answer queries using only the provided context." Intuitively, we aim to make the baseline LLMs hallucinate less via prompting, so that the gains made by optimizing do not rediscover easy prompting gains. We sample with temperature 0.7 when generating, and have a max sequence length of 1024 tokens. Since the context length of the models we study is 2048 tokens, prompts that have more than 1024 tokens can only have up 2048 combined tokens in the prompt and output.
Prefix tuning details. We append a n = 10 token postfix to the embedded system message, which has 51 51200 parameters (10 tokens times model embedding dimension 5120). We initialize the postfix to be 10 copies of Llama's 's "space" token embedding. We optimize the postfix with Adam (Kingmal Ba 2015), using learning rate le-4, no weight decay, epsilon 1e-7 and otherwise the default HuggingFace parameters (Wolf et al. 2019). We chose the learning rate out of two options (1e-3 and le-4), and the number of tokens out of three options (1, 10, and 20) based on the hallucination rate using Orca on a separate MS-Maree Marco validation set; we then recycle the same hyperparameters for all other realistic tasks and models. This means that SYNTRA is not hyperparameter tuned on Vicuna at all. We use episilon 1e-7 to avoid numerical stability issues that arise when converting models to 16.
Fine-tuning details In order to fine-tune the baseline models Orca and Vicuna 1.1, both of which have 13 billion parameters, we use copy the hyperparameters from (Chiang et al. 2023). Specifi- cally, we a learning rate of 5e-5, warm up ratio of 0.03, weight decay is O. and we fine-tuning for one epoch with batch size of 12 per device on four NVIDIA A 1100-PPP P00-PCIE-S00 8OGB GPUs in 6 precision.
15

---

Published as a conference paper at ICLR 2024
For every method, optimize for epoch which is exactly 100000 examples. When optimizing on synthetic data mixed with the reference data, we use 50000 examples on the names task and 50000 examples the reference task.
B.2 DATASET DETAILS
We next describe the dataset details for each of the realistic tasks we study.
MS MARCO. We first describe how we get our MS MARCO subset, using the data from (Nguyen et al. 2016). We start with the validation set, then filter for examples where response type is "description" (i.e., requires long-form response), and where at least one document useful for producing the answer (to enforce that the model only needs to use the context to answer). These labels are included the data. Of the remaining examples, we randomly choose 1000.
For MS Marco, we use the following prompt
Prompt:
### System: You are a helpful, honest, and conservative Al system designed to answer queries using only the provided context.
### Human: The following is a list of passages:
[[Passage 1]
[Paassage 10]
Using the passages, respond the following query: [query]
### Assistant:
Where here the passages and the query are from the dataset.
QMSum. We next curate subset of QMSum (Zhong et al. 2021p. To do so, we start with the training set, then take all queries from the "specific query list" associated with each meeting. For each query in the list, we filter the meeting transcript based transcript, then only include those that have under 1800 tokens according to the Llama tokenizer models can generate least 248 tokens response). This produces 852 examples. We use the training set rather than the validation because the validation set has far fewer examples, and we don't 't optimize against this dataset directly.
the "relevant text spans" our
For QMSum we use the following prompt
Prompt:
### System: You are a helpful, honest, and conservative Al system designed to answer queries using only the provided context.
### Human: The following is a meeting transcript:
[relevant lines of meeting transcript]
Using the transcript, respond to the following query:
Query: [query]
#### Assistant:
Where the relevant lines of the meeting transcript and the query are taken directly from the dataset.
Finally, deseribe adaptation of the ACI-Bench dataset (Yim al. 2023). We combine the train, validation, and three test splits for a total of 207 examples; this is all the possible examples in the dataset. Each examples contains dialog between patient and doctor.
ACI-Bench.
We use the prompt that Yim et al. 2023 use with our system prompt directly. This gives us the following prompt:
16

---

Published as a conference paper at ICLR 2024
Prompt:
### System: You are a helpful, honest, and conservative Al system designed to answer queries using only the provided context.
### Human: Summarize the conversation to generate clinical note with four sections:
HISTORY OF PRESENT ILLNESS, PHYSICAL EXAM, RESULTS, ASSESSMENT AND PLAN. The conversation is: [dialog].
#*# Assistant:
Here, the dialogues are taken directly from the dataset.
B.3 NAMES DATA
We next describe our names synthetic data. We generate names by taking the top 50000 first and last names in the U.S. from Remy 2021), then from these select 100 random first and last names, then combine them.
Prompt:
#### System: You helpful, honest, and conservative Al system designed to answer queries using only the provided context.
### Human: The following is list of names
[Name 1]
[Name 100]
List the first names where the first name starts with [first letter] in the order that they appear. Include both the first and last name in the response. If there are not 5 names that start with [first letter), return all of the names in the list that start with [first letter] the order that they appear. ### Assistant:
Here, the first letter is randomly chosen among the all letters for which there is at least one name, and the names are randomly generated according the above procedure.
In Figure we evaluate on 1000 randomly generated names prompts that are separate from the 100000 training examples.
B.4 REFERENCE DATA
source reference data, we use SQuAD (Rajpurkar al. 2016). Specifically, we take random SQuAD passages and their associated queries to create the following prompts.
Prompt:
### System: You are a helpful, honest, and conservative system designed answer queries using only the provided context.
### Human: The following a passage:
[SQuAD passage]
Using the passage, respond to the following query
[query]
[response type]
#### Assistant:
Here, SQuAD passage and query come directly from the SQuAD dataset. To get a more diverse set of prompts and outputs, try different options for [response type]. For 25000 examples we leave it
empty, and for 25000 examples we use chain-of- -f-thuggttt style prompts (Wei et al., 2022);; for 10000 examples set response type to "While performing the task explain your reasoning" and for 15000 examples we set the response type to a ch thouhht style prompt: "While performing the task think step-by-step and justify your steps".
17

---

Published as a conference paper at ICLR 2024
Model
Parameters
Data
MS MARCO
QMSum
ACI-bench
Average
Original Orca
12.4
4.7
88.4
35.2
Synthetic
6.7
2.0
86.5
31.7
Orca
Sys. message
Synth. Ref.
6.9
3.8
88.4
33.0
Synthetic
0.0
0.2
16.9
5.7
Full model
Synth. Ref.
4.7
3.2
77.8
28.5
Original Vicuna
6.9
2.2
83.1
30.7
Synthetic
2.8
0.7
28.5
10.7
Vicuna
Sys. message
Synth. Ref.
3.8
0.7
57.0
20.5
Synthetic
0.0
0.4
22.7
7.7
Full model
Synth. Ref.
4.3
2.7
82.6
29.9
Table 4: Newline rate (%) when optimizing either the system message or full model on just synthetic or synthetic + reference data, for both Orca and Vicuna. Optimizing on just the synthetic data tends to reduce the newline rate (a spurious attribute), adding the reference data restores it.
B.5 ADDITIONAL EVALUATION DETAILS
GPT-4 evaluation. this section, we describe how we used GPT-4 as a judge, following Yue al. 2223). The prompt in Yue et al. (2023) asks GPT-4 if the context supports/ / contradicts the response, or if there's 's not enough information. However, we found that using this prompt reported halluci- nation rates of under 3%, which we empirically found was far lower than the true hallucination rate.
Instead, we prompt GPT-4 to score whether each piece of information in the model response is supported by the context, and return ungrounded spans. We encourage to check each piece of information from the reply, especially focusing numbers. dates. names, etc., without worrying about paraphrases. We describe the task of GPT-4 as validating the outputs another language model. The full prompt is proprietary, and mimics prompts used test for grounding in production.
Similarity evaluation. We next provide additional details for computing the BLUE, ROUGE-1, -2, annd -L scores from figure Table [ When computing BLEU score, in (Post 2018). When there are multiple possible reference summaries available, we choose the last reference summary.
the method introduced
Entity evaluation. For retrieval of healthcare entities on the ACI-Bench dataset, we used a Named- Entity Recognition service from Microsoft Azure called Text Analytics for Health In order to reduce noise, we select subset of medical and regular entity types to be extracted from the doc- uments, and minimum confidence score of 0.75. The healtheare entities that select include Allergen, BodyStructure, ConditionQuullifer ConditionScale, Course, Diagnosis, Direction, Exam- inationName, Expression, GeneOrProtein, MedicationClass, MedicationForm.. MedicationNane, Medicationnoute, MutationType, SubstanceUse, SymptomOrSign, Trealmen/Name and Variant. We include yet one more non-healthcare entity, Quantity, as numerical reasoning known to chal- lenging task that can lead to hallucination more easily (Ji et al. 2023a).
B.6 ADDITIONAL REFERENCE DATA RESULTS
In this section, present the newline experiment results. We include the results in Table 4 and find that the newline rate tends to decrease on the synthetic data, but increases again when training the synthetic and reference data jointly. The newline rate is proxy for other spurious attributes that are harder to anticipate and measure.
Azure Text Analytics health one pre-built features offered by Azure Al Language Service. is a cloud-based API service that applies machine-learning intelligence to extract and label relevant med- ical information from a variety of unstructured texts such as doctor's notes, discharge summaries, clinical documents, and electronic health records. More information available on: http:://eaam.mmccoooff.oooooo hwwp:/aiiieiiesaagueeeeeeeeerere
18

---

Published as a conference paper at ICLR 2024
Model
Paramelers
Data
MS MARCO
QMSum
BLEU
R-1
R-2
R-L.
BLEU
R-1
R-2
R-T.
Original Llama Chat 7B
5.6
0.182
0.114
0.157
4.7
0.300
0.095
0.195
Llama 2 Chat 7B
Sys, message
Synth. Ref..
6.0
0.202
0.122
0.174
4.8
0.304
0.098
0.199
Original Llama 2 Chat 13B
3.9
0.154
0.090
0.130
0.307
0.097
0.199
Ilama 2 Chat 13B
Sys. message
Synth.
Ref.
7.9
0.236
0.144
0.205
4.2
.267
0.088
0.180
Comparison between SYNTRA and human-generated reference datasets on Llama 2 7B Chat and Llama 2 13B Chat. We abbreviate ROUGE-1, ROUGE-2, and ROUGE-L with R-1, R-2, and R-L. We find that SYNTRA is consistently closer to the reference summaries on MS MARCO than the original model, and comparable on QMSum. For all metrics, higher better.
Table 5:
B.7 ADDITIONAL ENTITY ANALYSIS
In this section, we provide additional analysis of the entity results from Table In particular, we measure the change in grounded and ungrounded entities on two axes: whether the whole model is fine-tuned, or only the system messages is optimized, and whether the parameters are trained on only synthetic data. Overall, we find that fine-tuning tends to eliminate more grounded and ungrounded entities than optimizing the system message (potentially leading to responses with less content), while adding in reference data tends to recover many of the grounded entities, while adding back few of the ungrounded entities. We provide additional details below.
Full model versus system message. We first compare the entity loss between optimizing the full model or only the system message. We find that optimizing the system message consistently pre- serves more grounded entities: optimizing the system message decreases the average number of grounded entities by 10.6 less (61.6%, which is a substantial loss) than the full model when op- timizing just synthetic data, and 1.2 less than the full model when optimizing with synthetic and reference data jointly (7%, which is far less substantial). In contrast, the number of ungrounded entities decreases by an average of 4.5 more (45%) when training on just synthetic data, and 1.4 (14%) more when training on synthetic and reference data. Overall, this analysis demonstrates that training on the full model tends to reduce the number of both grounded and ungrounded entities by more than just the system message (sometimes at unacceptable levels), indicating that training the full model tends to shorten responses more (and potentially leave out important details) than training the system message.
Synthetic data versus synthetic + reference data. We next compare training on synthetic data, compared to synthetic and reference data. For the system message, adding in the reference data adds 2.4 (14%) grounded entities compared to just training on the synthetic data, while only adding 0.9 ungrounded entities back (9%). Similarly, fine-tuning with reference entities adds 11.8 grounded entities (68.8%), while adding 4.1 ungrounded entities (41.1%). These numbers have caveats; they measure the gain in entities over just training on synthetic data, which is more degenerate for fine tuning, yet nevertheless shows that reference data recovers any of the lost grounded entities, at the cost of adding (substantially fewer) ungrounded entities. Changing a to mix in less reference data is way for practitioners to adaptively change this tradeoff; in our work, we only study a 1 (no reference data) and a 0.5 (equally mixing).
Warnings about applying variants of SYNTRA. While variants of SYNTRA offer different trade- offs between the number of ungrounded and grounded entities removed, our entity evaluation reveals that some should be entirely avoided. In particular, fine-tuning the whole model only on synthetic data (without reference data) tends to lead to massive reductions in grounded entities; it reduces the number of grounded entities by 66% for Orca, and removes nearly all of them (99%) for Vicuna. We think this is because Vicuna overfits to the synthetic task without reference data. Overall, these results highlight the importance of fine- ine-grained hallucination for detecting degenerate solutions, and show that S SYNTRA be tweaked to adaptively trace a pareto frontier between reducing grounded and ungrounded entities.
19

---

Published as a conference paper at ICLR 2024
Model
Parameters
Data
Ungrounded (0)
Grounded (11)
% Ungrounded (i)
% l Grounded (0)
Original Llama 2 Chat
16.2
21.8
Llama 7B Chat
Sys. message
Synth, Ref
20.4
21.2
6.2
Original Llama 13B Chat
14.1
22.6
Llama Chat
Sys. message
Synth, Ref
7.1
14.0
49.1
38.1
Table 6: Entity evaluation between the prompts and outputs in ACI-Bench. Using NER model, we measure the number of entities that are the output that not appear the prompt (Ungrounded, lower is better), and that do appear in the prompt (Grounded, higher is better), along with the per- centage decrease in ungrounded and grounded entities relative the original output.
B.8 LLAMA 2 EXPERIMENTS
In this section, we present results on Llama 2-chat 7B and Llama 2-chat 13B (Touvron et al. 2023b). We use all of the same hyperparameters that were selected on Orca, without completing hyperpa- rameter sweep.
and Table 6 we report the overlap with reference summaries and entity analysis described in Section 3.3 We find comparable results to Orca and Vicuna. We do not run the GPT-4 evaluation from Section 3.2 due resource constraints.
In Table and
20


Reviews:
[{"review_id": "txYlUdJqwZ", "reviewer": "Reviewer_MYvf", "summary": "The paper proposes a simple method for reducing hallucinations in LLMs: finetuning on a synthetic task where hallucinations are easy to elicit and measure (copying certain entries from a provided list of last names). The authors' best method boils down to optimizing system message via prefix tuning and then using this soft system message generalises well to realistic abstractive summarisation tasks, reducing the hallucination rate (measured in terms of BLEU, number of grounded named entities and GPT-4 judge scores).", "questions": "1. It was a mildly surprising finding for me that soft system message optimisation is so much better than supervised finetuning. Have you tried manipulating the $\\alpha$ hyperparameter to increase regularisation for supervised finetuning? I could imagine this being helpful.\n2. Is the KL in eq. 2 forward (KL(base, new)) or reverse (KL(new, base))?\n3. It’s confusing that “hallucination rate” in Fig. 2 is actually non-hallucination rate (higher is better) while for “hallucination rate” in Table 1 lower is better.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a big open problem (hallucinations in LLMs) and stays up-to-date with state-of-the-art advancements in the field (using benchmarks such as ACI-Bench and models such as Vicuna).\n2. I like the general “[just ask for generalization](https://www.notion.so/Teaching-language-models-to-hallucinate-less-with-synthetic-tasks-128f91205a1d45208d3add1298c95f18?pvs=21)” approach to reducing hallucinations that does not use human feedback or human demonstrations directly. One could hope it would scale well as LLM capabilities grow (though the authors don’t show this) and won’t limited by errors and bias of human-provided, on-domain gold demonstrations.\n3. The paper is written clearly and was easy to follow.\n4. The experiments conducted by the authors are well-designed and comprehensive. For instance, I like how they control for limiting hallucinations by exploiting spurious features in their synthetic task and exploiting the errors of GPT-4 judge and report additional metrics.", "weaknesses": "1. The authors only experiments with one model size (13B) and it’s hard to say how well their method scales. Will be benefits of SynTra increase or decrease for finetuned LLaMA 30B or 65B? What about LLaMA 6.5B?\n2. The absolute improvements in hallucination rate (Table 1) don’t strike me as very high. I’m not sure they justify the software complexity of soft system message optimization.\n3. Relatedly, it would be good to compare SynTra to some simpler baselines, e.g. finetuning on gold labels for some other summarization dataset (e.g. OpenAI’s cleaned Reddit data). I’d also be curious to see how RLHF’d models (e.g. LLaMA-2-*-chat series) do on those benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ixRFbhWNpD", "reviewer": "Reviewer_C5mJ", "summary": "The paper proposes a method called SynTra to reduce hallucination in LLMs. SynTra designs a synthetic task where hallucination is easy to evaluate -- retrieving names from a given list. The synthetic data was used for soft-prompt tuning or full fine-tuning from Orca or Vicuna. The evaluation of real-world tasks such as MS MARCO, QMSum, and ACI-Bench shows that soft-prompt tuning with both synthetic data + reference data performs the best in general.", "questions": "- In figure 2, the y-axis says hallucination rate (the lower the better), but according to the passages, the y-axis should be the accuracy (the higher is better). Please clarify this.\n- Citation error: (yew Lin & Rey, 2004) -> (Lin & Rey, 2004)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The optimized model or system message can be transferred to real-world downstream tasks. Experiments show this reduces hallucination compared to the 13B parameter models Vicuna and Orca on tasks like search-and-retrieve, meeting summarization, and clinical report generation.\n- Novel idea of using a synthetic task to easily isolate and optimize against hallucinations.", "weaknesses": "- Lack of baselines: The proposed method was applied to existing fine-tuned LLMs like Vicuna and Orca and the experiments only show that they become better than the original LLMs. \nHowever, we do not know how the quality of the SynTra task compared to other datasets.\nFor example, a more fair comparison would be finetuning two LLMs starting from LLaMA, one on the VIcuna sharegpt data, the other on the SynTra data. I wonder if the effects of the SynTra data are not as much as the Vicuna data, in terms of reducing hallucinations.\n- Comparison with the code datasets: [previous work](https://arxiv.org/abs/2210.07128) [1] has found that training the model on code datasets is beneficial for reasoning. I wonder how the quality of the SynTra task compared to directly fine-tuning the model on the code dataset?\n- In general, training on the rule-based synthetic data would help is expected. However, the question that matters is how useful this data could be compared to other existing datasets such as instruction tuning datasets or code datasets. I suspect that the effects of synthetic data would be marginal when compared to others.\n- The transfer from synthetic to downstream tasks could be brittle: Sometimes full fine-tuning is better than soft-prompt tuning in system messages, but sometimes it's not. Sometimes reference data regularization is helping, sometimes it's not. \n\n[1] \"Language Models of Code are Few-Shot Commonsense Learners,\" Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig. EMNLP 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dv7Mxy1xue", "reviewer": "Reviewer_tStg", "summary": "The paper talks about how to optimize the postfix of system message to make LLM hallucinate less. By introducing a synthetic task – the names retrieval task – the authors create an environment where hallucination occurrences are both frequent and easily traceable. This approach allows for a straightforward and clear identification of hallucinations - when the LLM generates a name not present on the provided list.", "questions": "1. How is the convergence of the optimization? Is there any constraint on the postfix such as any constraint on the $d\\times n$ matrix? \n\n2. Can SYNTRA be added on top of other classes of methods mentioned in section 2 to further reduce hallucination? \n\n3. The authors mentioned at the end of the Related Work that the direct optimization of a discrete prompt might be viable in SYNTRA as well, can they give more discussions about the pros and cons compared to SYNTRA? Will it be more difficult in terms of the optimization?\n\n4. From Figure 2 and Table 1, it looks like when LLM weights are fine-tuned, they are overly optimized for the synthetic task and as such it might ends up hallucinate more on realistic tasks. Is there any possibility that an early-stopping of the fine-tuning process might actually help in finding better parameters for both synthetic and realistic tasks?\n\nMinor:\n1. Appendix A, Appending the postfix, end of sentence, should it be $d\\times (n+m)$?\n2. Do authors believe reducing hallucination of LLMs solely by optimizing postfix has a limit and is there a way to find that limit?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality: Though previous works have characterized hallucination using synthetic tasks, this paper goes a step further by utilizing synthetic data to actively reduce hallucination. \n\nQuality and Significance: The paper's relevance is evident, considering the persistent challenge of hallucination in LLMs. By offering a tangible method to evaluate and optimize against such hallucinations, the paper contributes a valuable tool to the domain.\n\nClarity: The paper is generally well-written.", "weaknesses": "The evaluation of SYNTRA is somewhat limited, as it focuses on only 2 models and 3 realistic tasks. This raises concerns about the method's ability to generalize across diverse models and tasks. Also, the effect on Vicuna seems marginal. This weak effect could limit the practical utility of SYNTRA, especially if similar results persist across other models.\n\n\nI believe the way the results are presented can be improved. There are some key takeaway messages that are hidden in the table/figure. Important insights, such as the significant reduction in grounded entities generated by Orca and Vicuna (by 66% and 99% respectively) when using SYNTRA without reference data, are buried within tables and figures. For example, this observation implies that the LLMs, in their attempt to avoid hallucination, compromise by producing limited details. Such significant findings should be more prominently highlighted, especially to stress that this setup may not be a viable solution for hallucination reduction.\n\nI find the axis label \"Hallucination rate\" misleading in Figure 2 - I think the number presented therein should instead be 1-hallucination rate? or did I miss anything?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)