Detecting Hallucinated Content in Conditional Neural Sequence Generation
Authors: Chunting Zhou
Jiatao Gu
Mona T. Diab
Paco Guzmán
Luke Zettlemoyer
Marjan Ghazvininejad
Venue:  (2021)
Keywords: conditional text generation
hallucination detection
sequence generation evaluation
neural machine translation
abstractive text summarization

Abstract: Neural sequence models can generate highly fluent sentences but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input, which can cause a lack of trust in the model.
To better assess the faithfulness of the machine outputs, we propose a new task to predict whether each token in the output sequence is hallucinated conditioned on the source input, and collect new manually annotated evaluation sets for this task.
We also introduce a novel method for learning to model hallucination detection, based on pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations.  
Experiments on machine translation and abstract text summarization demonstrate the effectiveness of our proposed approach -- we obtain an average F1 of around 0.6 across all the benchmark datasets.
Furthermore, we demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in the low-resource machine translation and achieve significant improvements over strong baseline methods.
We will also release our annotated data and code for future research.

Under review as a conference paper at ICLR 2021
DETECTING HALLUCINATED CONTENT IN CONDI- TIONAL NEURAL SEQUENCE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural sequence models can generate highly fluent sentences but recent studies have also shown that they are also prone to hallucinate additional content not sup- ported by the input, which can cause a lack of trust in the model. To better assess the faithfulness of the machine outputs, propose new task to predict whether each token in the output sequence is hallucinated conditioned on the source in- put, and collect new manually annotated evaluation sets for this task. We also introduce a novel method for learning to model hallucination detection, based on pretrained language models fine tuned synthetic data that includes automat- ically inserted hallucinations. Experiments on machine translation and abstract text summarization demonstrate the effectiveness of our proposed approach we obtain an average FI of around 0.6 across all the benchmark datasets. Further- more, we demonstrate how use the token-level hallucination labels define a fine-graines loss over the target sequence in the low-resource machine translation and achieve significant improvements over strong baseline methods. We will also release our annotated data and code for future research.
INTRODUCTION
Neural sequence models have achieved impressive breakthroughs in wide range of applications, in- cluding data-to-text generation (Puduppully et al. 2019). machine translation /Vaswani et al. 2017]]
Wu et al. 2016) and text summarization (Rothe al.] 2020). Although these models can generate
fluent sentences that are even sometimes preferred to human-written content (Läubli et al. 2018
Brown et al. 2020), recent work has also shown that they lack global logical consistency (Marcus
& Davis 2020), sometimes degenerate to dull and repetitive outputs (Welleck et al.) 2019) and can often hallucinate content that is not entailed by the input (Maynez et al. 2020). In this paper, we focus on the faithfulness of machine outputs in conditional sequence generation tasks, aiming to automatically identify and quantify content in the output that is not faithful to the input text.
This risk of generating unfaithful content impedes the safe deployment of neural sequence genera- tion models. The first step to building models that do not suffer from these failures the assessment and identification of such hallucinated outputs. Prior work has shown that standard metrics used for sequence evaluation, such as BLEU scores (Papineni et al. 2002 Post 2018). ROUGE (Lin) & Hovy 2004) and BERTScores (Zhang al., (2019), do correlate well with the faithfulness of model outputs (Maynez et al. 2020 Wang & Sennrich] 2020: Tian et al. 2019). and they also require reference text, limiting their applicability detecting halluciations a deployed system at run-time. Very recent efforts (Maynez et al.) 2020 Durmus et al., 22200) Wang et al. 2020a) have started to develop automatic metrics measure the faithfulness of output sequences. These methods use external semantic models, e.g. the
systems (Wang) jet al. 2020a Durmus al. 2020) or textual entailment inference models, to score faithfulness tai- lored for abstract text summarization. However, these scores do not directly measure the number of hallucinated tokens and only correlate weakly with human judgements due to compounded errors.
and question-answerin
We propose a new task for faithfulness assessment hallucination detection at the token level, which aims to predict if each token in the machine output a hallucinated or faithful to the source input. This task does not the reference output to assess faithfulness, which offers the ability to apply it in the online generation scenario where references are not available. Similar to the spirit of our proposed task, word-level quality estimation (Fonseca al. 2019) in the machine translation

---

Under review as a conference paper at ICLR 2021
(Source meaning: Mike
0
0
0
bookstore Thursday.)
Jerry
goes
to
the
bookstore
with
friend
happily,
Source Input
Machine Translation
Figure 1: A toy example of token-level hallucination detection from machine translation. The words in grey blocks is an example of machine translation output and the labels above them indicate if each word is faithful (0) to the source input or hallucinated one (I),
community predicts if tokens are correctly translated based on human post-editing. However, they do not distinguish errors terms of fluency and adequacy (Specia et al.] (0T1). In contrast to estimating the amount of human post-editing work required to fix errors, we specifically focus only on hallucination (not fluency) errors.
We measure hallucination for two conditional sequence generation tasks abstractive summariza- tion and machine translation (MT). For the former, we produce a benchmark dataset from the recent released annotations in (Maynez et al. 2020). For MT, we carefully design the human assessment guideline and create high-quality annotations. We will also release our human annotated data for future research. To learn teken-level hallucination prediction for general conditional sequence gen- erations tasks, we propose a novel method that creates synthetic "hallucinated" dataset with pseudo labels and finetunes a pretrained language model (Liu et al. 2019 Conneau et al. 2020) on it. With- out any human annotated supervised training data, we achieve an average Fl of around 0.6 across all the benchmark datasets, setting initial performance levels for this new task. We also show that pretraining on MT can actually produce more faithful translations, confirming recent findings in abstractive summarization (Maynez et al. 2020).
Predicting hallucination labels at token-level provides a tool for diagnosing and interpreting model outputs, which allows us to flag potential risks inference time for previously unseen inputs. On the other hand, the token-level labels also offer possibility of fine ine-grained controls over the target sequence to improve the generation. We show how to use these token-level hallucination labels to improve self-training in low-resource MT, where the teacher can produce hallucinated outputs that are harmful to the student model. However, many outputs are only partially hallucinated (see exam- ples in Appendix D.6) and the rest of the output is still useful for training, different token-level loss truncation schemes. Our best method outperforms strong baselines by a large margin both in translation quality and hallucination reduction.
show by introducing
TASK: HALLUCINATION PREDICTION AT TOKEN -LEVEL
For source sequence S and its model generation G from neural conditional generation model, following Maynez et al. (2020) we define any span Wi, .... , WW+, >= 0) in G as being halluci- nated if it cannot be entailed by the source input S. More specifically, we consider two not mutually exclusive types of hallucination:
Content Insertion: a span Wi, Wi+j in G consists of additional content that is not supported by S, i.e. its paraphrase or other equivalent form cannot be inferred from S. In Fig. [ the word "happily" in the machine translation belongs to this case. This is also referred as "extrinsic halluci- nations" in Maynez al. 2020).
Incorrect Substitution: a span of word(s) is misrepresented information based on the S. In Fig. [
"Jerry" in the machine translation is a hallucinated word and should be replaced by "Mike " This
type of hallucination is similar to the concept of "intrinsic hallucination" in Maynez et al. (2020)
Note that there are cases where certain words (e.g. "This is not a great book." becomes "This is
great book.") are dropped in G and hence the meaning of S is changed, and we consider any spans in G that misrepresent S as hallucinated contents (e.g. the entire sentence of "This is a great book.").
We aim to identify all the span(s) satisfying the above conditions in the machine generation GP Note that the above definition is only used in the guidelines for human annotators, who do not need distinguish between these types rigorously.
We do consider the under-generations e.g. the source input is only partially translated or summarized.
2

---

Under review as a conference paper at ICLR 2021
Hallucination label assignment with edit distance
0
0
0
Jerry
goes
to
bookstore
friend
happily:
BART
Generate synthetic hallucinated sentence
<MASK> goes to the bookstore <MASK>
T Mike goes to bookstore on Thursday.
Figure 2: Generation of synthetic hallucinated data set with hallucination labels. The bottom block generates a hallucinated version T by feeding the noised sentence to the pretrained encoder- decoder model BART. The top block assigns hallucination labels to each token by computing the edit-distance between 7" and T. Labels of refer to hallucinated words and vice versa.
LEARNING TOKEN-LEVEL HALLUCINATION DETECTION
We propose a general-purpose method for token-level hallucination detection for conditional se- quence generation tasks. Given the source input S, we first formulate the task token-level hallu- cination detection sequence labeling problem where binary label is predicted at each position G, of the machine generation G. One straightforward way learning such task is to train a model with supervised data in the form of G), LG) where LG are the labels every positions of G that indicate if each word is a hallucinated one or not. However, because such labeled training data readily available, we also propose approach to automatically create synthetic training data.
3.1 SYNTHETIC DATA CREATION
We use bi-text from the training data to create synthetic examples by automatically inserting new, hallucinated tokens. More speficially, take target sequence T and create hallucinated version of it denoted 7" with associated hallucination labels for each token in T". Then we can train a supervised model this synthetic labeled data set of ((S, T'), L. The key challenge is that T should be a fluent sentence that does not differ too much from T.
Generation of hallucinated sentences To control this synthetic hallucination process, build on recent work on large scale denoising autoencoders. These models learn to map a corrupted sentence back the original text it was derived from, including for example learning to reconstruct missing words that have been arbitrarily masked out. We use BART (Lewis et al.) 2020) by first inserting noises and then reconstructing parts of the target sentence, without providing it any access to the source sentence, thereby encouraging it to insert new content needed to ensure fluency, As shown Fig. we first apply noising functions to the target sentence T' in the bi-text and then use pretrained BART to generate T' conditioned on the noised T with beam search.
Label assignments After obtaining the hal- lucinated sentence 7" with BART, we need to assign appropriate labels to each token in T' to mark which words are hallucinated. We com- pute the edit distance between T" and T, and back-trace the deletion and substitution opera- tions with dynamic programming. All the po- sitions in T' involving these two operations are labeled as hallucinations and everything else is considered faithful to T. Fig. 3 shows example of label assignment with edit distance, where words in red are replaced and the word in blue is deleted to convert T' to T. Assigning labels with edit-distance not always guarantee the correct labels, however, we find that this simple approach provides sufficiently high quality training data for effective hallucination detection in practice.
Mike
goes
to
the bookstore on Thursday.
Janny
goes
the
bookstore with
Frlen
happlly-
Figure 3: An example label assignment.
3.2 FINETUNING PRETRAINED LM ON SYNTHETIC DATA
Hallucination prediction loss We follow the common practice in natural language understanding (NLU) tasks and finetune a pretrained language model (LM) on our synthetic data. We finetune cross-lingual LM (Conneau et al.) 2020) for machine translation and monolingual model (Liu jet al., 2019) for summarization. In both cases, we concatenate the input source, true target and hallucinated target denoted (S, T, T') as single input sequence to the pretrained LM. Then
3

---

Under review as a conference paper at ICLR 2021
1
0
0
0
0
XLM-Roberta Roberta
Mike goes to the bookstore on Thursday.
<SEP>
<SEP>
Jerry goes to the bookstore; with friend
happily.
Source S
True target T
Hallucinated version of target T'
Figure 4: Finetuning XIM-RRoerta (for cross-lingual generation task, e.g. machine translation) or Roberta (for monolingual generation task, c.g. text summarization) on the synthetic training data.
minimize the standard classification loss Ppred over the pseudo hallucination labels LT on top of the final hidden vectors of each token in T' as shown in Fig.
Although using only the source text and hallucinated target (S, T') as the input should be sufficient to learn the hallucination prediction task, the true target 7" in the input could help the model. At test time, when evaluating the faithfulness of the machine outputs G, do not use the true target I and perhaps surprisingly find our model can generalize well without references, even when it was present during training.
also easily measure the extent which including
To prevent the model from overly relying on the true target T and learning spurious correlations (e.g. the edit distance), we explored two techniques: (1) dropout randomly drop out tokens in T to force the dependence on the source input; (2) paraphrase recall that at synthetic data generation time, we generate T' from BART conditioned on the noised T. Instead, we can apply noise functions to the paraphrased sentence of T. Let denote the paraphrased sentence of T and D' denote the generation from BART conditioned on the noised D. Then we create pseudo labels of D' denoted LD by computing the edit-distance between the D' and and use ((S, T, D'), LD)) as the training data for finetuning. Since the pseudo labels are created based on D, it can prevent the model from learning the edit-distance between and D' easily. Ablation studies will be provided in $ $5.4
Masked LM loss We also add the masked language model loss (MLM) Lmtm following (Devlin et al. 2019). To learn this loss, we create different batch from the above by concatenating only the source S and target T as the input, since the hallucinated target T" could provide erroncous informa- tion for predicting masked words in T. We find that such multi-task learning objective helps learn better representations of the inputs and further improves performance on predicting hallucination labels. The final loss is L Lpred + a Lulm where is hyperparameter.
EVALUATION TASKS AND DATA
We evaluate hallucination in conditional neural sequence models on abstractive text summarization and machine translation (MT) tasks, using the models and datasets described below.
4.1 ABSTRACTIVE TEXT SUMMARIZATION
Maynez et al. (020) studied the hallucination problems in extreme summarization on the XSUM dataset which comprises 226,711 British Broadcasting Corporation (BBC) articles paired with their single-sentence summaries. They randomly sampled 500 articles from the test set of XSUM and evaluated summaries from four abstractive summarization systems: (1) PtGen (See et al. 2017). an RNN-based attentional sequence-to-sequenee (Seq2Seq) model. (2) TConvS2S (Narayan et al.
2018) is a topic-aware convolutional (Gehring et al. BERTS2S are standard Transformer Vaswani et al.
Seq2Seq model. (3) TranS2S and (4) encoder-decoder models. The weights in TranS2S are randomly initialized while the encoder and decoder of BERTS2S are initialized with the pretrained BERT-base (Devlin al., (2019) checkpoint. Maynez et al. (2000) asked the human annotators to label the spans in the machine generated summaries if they were unfaithful to the article (for more details see Maynez al. (2020)). We post-processed their human annotations by majority voting and create our test datasets for each of the abstractive summarization systems.
2017
2017
4.2 MACHINE TRANSLATION
Previous work (Wang & Sennrich 2020: Müller et al. 2019; Koehn & Knowles 2017) has shown that translation models are particularly prone hallucination when tested out of domain. We simi-
4

---

Under review as a conference paper at ICLR 2021
larly focus this evaluation regime and additionally consider the low resource case where modest amount of out of domain data is available at training time.
Data We use a molti-domain Chinese- Chinese-English EZh-En) translation dataset (Wang et al., 2020b) which consists of four balanced do-
Fleiss' Token
Kappa Sent
Models
mains: law, news, patent and subtitles. We create a new training data Dirain with law (1.46M sentences), news (1.54M), subtitles (1.77M) train data and randomly sample 0.03% parallel sentences from the patent (870) training data. We train (described below) on this dataset and test on 150 examples sam- pled from patent test data. In addition, we also evaluate the trained models on the COVID-19 domain. We sample 100 examples from the recently released translation benchmark dataset (Anastasopou- los et al. 2020p. Together with the 150 examples from patent test data, we create our benchmark dataset with 250 examples in total and we denote this dataset Devat- We ask human annotators to eval- uate the level of hallucinations on this dataset D Decal-
MT
TranS2S MBART
0.58
0.72
0.54
0.62
NMT models
XSum
PtGen
0.81
TConvS2S
0.83
Tran22S
0.79
BERTS2S
0.79
Table 1: Fleiss's Kappa scores (t): agreements on to- ken-level hallucination labels or sentence-level (sent) rat- ings among different annota- tors. The token-level agree- ments for XSUM are com- puted on the released annota- tions by |Mynnzz et al. (2020).
Models Our evaluation data is generated from two reference mod- els on which we will measure ballucination (see Appendix for more details): (1) TranS2S (Vaswani al. 2017) is the standard Trans- former Seq2Seq model with 6 encoder layers and 6 decoder layers. (2) MBART (Liu et al., 2020) is a Seq2Seq denoising entoden pretrained on large-scale monolingual corpora in many languages. We finetune MBART checkpoint on Drrain which a standard Transformer architecture with 12 layers of encoder and 12 layers of decoder pretrained on multilingual corpora of 100 languages.
4.3 HUMAN ASSESSMENT OF HALLUCINATIONS
In the human evaluation, three bilingual annotators were presented the Chinese source sentence, the English reference sentence and the machine translation. We conducted the pilot study and practice sessions with annotators before annotating the blind test set Deval- Annotators were asked to only focus on the hallucinations during assessment and label each sentence with one of the three types of labels: incomprehensible, faithful, and hallucinated. If the translation is a hallucinated one, we asked the annotators to tag all the tokens that were not faithful to the source. We dropped all the translations that were labeled as incomprehensible (15 for TranS2S and 3 for MBART). The final benchmark datasets were created by taking majority labels among three annotators. (See Appendix A for more annotation details.)
In Tab. D we show the Fleiss's Kappa scores of our annotations for MT and the processed an- notations from (Maynez et al.] (020)) on abstractive summarization. A higher Fleiss's Kappa score indicates higher agreements among anmotators | We achieved moderate agreement on the token-level hallucination annotations and substantial agreement on the sentence-level annotations, while Maynez et al. (2020) achieved substantial or almost perfect agreement on the XSUM dataset. For MT, it's relatively harder to achieve consistent agreements among annotators for the following reasons: first, although we have made strict annotation guidelines following the definition of hallu- cination in Section 2 could still be difficult for annotators distinguish between bad translations and hallucination; second. it was sometimes difficult for annotations to understand the specialized text in the patent domain, which can contain complex scientific deseriptions.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Synthetic Data Generation We use pretrained BART (1 (Lewis et al.) 2020) model in the fairseq toolkit (Ott et al. 2019) with 12 layers of encoder and 12 layers of decoder for synthetic labeled data generation. We uniformly sample the percentage of tokens pm. to mask from [0, hm] for each sentence. We also uniformly sample the probability of replacing a token with a random token from [0, h,] denoted pr- Pm and p. are two important factors that affect the noise level when generating
https: httpas::/eew..iiieeeee.e www.iiiiaa..gg///ooo. /
5

---

Under review as a conference paper at ICLR 2021
MT
Summarization TranS2S
Methods
TranS2S
MBART
PtGen
TConvS2S
BERTS2S
Alignment
29.47
9.93
38.92
37.94
34.47
35.81
Overlap-based
9.14
3.24
57.22
54.25
53.79
55.13
Synonym-base based
59.54
63.73
58.66
53.07
Ours (w/o reference)
65.75
41.92
63.66
65,94
61.70
55.45
Ours (w/ reference + synonym) Ours (w/ reference)
64.72
69.37
63.88
56,49
66.08
46.81
63.89
66.28
62.24
55,88
Table 2: Fl (x of hallucination labels on the outputs of different systems from a machine transla- tion task (.ee)4.2) and the abstract summarization task (XSUM dataset). The first block are baseline methods and the second block are our results. We highlight the best results without using reference.
the synthetic data. For MT, we set hm and hy to 0.6 and 0.3 respectively. For abstractive summa- rization, we use 0.4 and 0.2. We use beam search for decoding from BART with beam size of 4 and length penalty of 3. For MT, we first create paraphrased target sentences D' through knowledge distillation (Kim & Rush 2016 by using the outputs from the same trained TranS2S model on the source inputs.
Finetuning Pretrained LM For MT, we finetune XLM-R (Conneau et al. 2020) on the synthetic dataset with batch size of 128, and we annotated 50 examples (different from those in Deval) from the patent test data as the validation dataset. For summarization, we finetune RoBERTa (Liu et al. 2019) with batch size of 96 and early stop training with 10K update steps. In addition, we dropout tokens from the reference T in the input with a rate of 0.5 and 0.3 respectively for summarization and MT to learn Lpred. We set a to be 0.6 for MT and 0.5 for summarization based on the scales of Lpred and Lmim- For both tasks, we set the mask probability used for Lmim to be 0.5, and the initial learning rate to 2e with polynomial decay. We describe other hyperparameters, including training of MT models, in the Appendix B
5.2 EVALUATION ON TOKEN-LEVEL HALLUCINATION PREDICTION
In Tab. we present the Fl of token-level hallucination labels across six benchmark datasets for MT and astractive summarization (the full results of precision, recall and Fl are presented in Tab. D and Tab. 10 in appendix). We compare with three baseline methods that we proposed for this new task: (1) The alignment-based method uses a word alignment model for hallucination assessment. We employ SimAlign (Sabet et al. (2020), an unsupervised word aligner, that extracts word alignments from similarity matrices induced from pretrained word embeddings. SimAlign is essentially used for crosslingual tasks, and we adapt it to summarization by using embeddings from the pretrained BERT-large (Devlin et al., 2019). We predict a target token as being hallucinated if it is not aligned to the source tokens. (2) The overlap- based method is heuristic one that predicts target token as being hallucinated does not appear the source input. Since it's not feasible to perform string matching between two languages for MT, we use the bilingual lexicon induction method (Zhou et al. 2019) to first translate each English word into the Chinese word and then check its existence in the source text. (3) We go further by exploiting synonyms to assess hallucination in the summarization task where we use WordNet (Miller 1998) to find synonyms of nouns, verbs, adjectives and adverbs of the target summary and the source article; we predict a target as being hallucinated if its synonym can not be found in the set of the source synonyms.
From Tab. we have the following observations: (1) We achieve descent performance on this task and rank the best among all the baseline methods, however it is still far from being solved. Token- level hallucination prediction requires deep semantic understanding of the inputs and worthy of study in the future. (2) We can see that even though our model learns hallucination prediction with reference during training (Sec. 3.2), by applying token dropout to T, our model generalizes well without feeding the reference at test time. As a contrast, we report the results of predicting with reference at test time and observe that the model can achieve significantly higher recall but worse precision (Tab. 10). (3) The two non-neural baselines we proposed work surprisingly well on the summarization datasets, especially the synonym- based system. We guess this is because the infor- mation of the summaries should come from the source article and a majority of hallucinated words are nouns 5.3 which can be easily detected by string matching or synonym matching. Our neu- ral system performs better than these baseline methods but not significantly, and we hypothesize

---

Under review as a conference paper at ICLR 2021
MT
Summarization
Methods
TranS2S
MBART
PtGen
TConyS2S
TranS2S
BERTS2S
True hal. tokens (%) Prod hal. tokens (%)
18.12
11.10
46.09 57.22
$2.89 57.68
46.74 55.78
37.51
18.56
7.99
48.84
Table 3: Comparisons of annotated (True) and predicted (Pred) percentage of hallucinated tokens on the benchmark test sets.
MT
XSum
0.6
Gold
Hallcchttn
0.5
Gold Our predection
Our predictions
0.6
0.4
0.3
0.2
0.1
otherss
Vis
CD
5TM
others
SYM
RBI
POS tag
POS tag
Figure 5: Relationship of Part-of- Pert-of-Speech tags and percentage of hallucinations for machine transla- tion (left) and summarization (right).
that this is because the Roherta model we finetune on only allows a maximum input length 512, which results in average cutoff of 158 subwords from the source article and losses of the source information. By taking the union of the predictions from the synonym-based and our models, we can further obtain improvements on the summarization datasets. We believe the advances in long sequence modeling (Beltagy et al. 2020 Kitaev et al. (020) are advantageous to our models. (4) At the same time, the baseline methods can not obtain reasonable performance for MT since crosslin- gual semantic matching is more challenging and our model shows significant improvements over these brittle baselines.
In Tab. 3 we show the percentage of annotated and our model predicted percentage of hallucinated tokens across the six benchmark test sets. We can see that model predictions correlate well with human assessment and has Pearson correlation coefficient of 0.986.
5.3 ANALYSIS
Analysis on Pretrained Models for Conditional Sequence Generation Recent work (Maynez cct al. (020) has shown that pretrained models are better at generating faithful summaries as evalu- ated by humans. In Tab. summaries generated from BERTS2S contain significantly less hallucina- tions than other model outputs. We also confirmed this trend in machine translation that translations from MBART contain much less hallucinated contents than that from TranS2S.
Analysis on Hallucination Words and their Part-of-Specch Tags In Fig. we present the per- centage of hallucinated tokens categorized by their Part-of-Speech tags predicted by POS tag- ger Tootanova et al. 2003). First, we see that for both MT and summarization datasets, nouns are the mostly hallucinated words. In abstractive summarization, verbs also account for certain amounts of hallucinations. Second, our model predicted hallucinated words match well with gold annotations on the distributions of POS tags. We also compare the percentage of hallucinations within each POS tag in Appendix (D.4
5.4 ABLATION STUDIES
60.
Effects of including reference at training time Recall that we concatenate the source, reference and machine generation together as the input when learning hallucination predic- tions (Sec. 3.2). In 6i we vary the dropout rate of tokens in the reference at training time and evaluate the models on the outputs from the TranS2S model for both tasks, where dropout rate of 1.0 indicates that do not include the reference at all. First, different dropout rates do not signficinatly affect performance for
MT
50.
40
Dropout Rate of Reference
Figure 6: Performance on the TranS2S bench- mark from MT and summarization by varying the dropout rate of tokens in the reference at learning hallucination predictions.
7

---

Under review as a conference paper at ICLR 2021
MT, this is likely because use the paraphrased target when creating the synthetic data in- stead of the reference sentences. Thus, the "hallucinated" sentences D' from BART do not re- semble the reference T as closely as T", and the model will not learn spurious correlations be- tween the T and D'. Second, for summarization we see that applying word dropout is cru- cial since we have used the reference more directly for generating synthetic data. On the other hand, if reference is removed at learning time (dropout = 1.0), the resulted model per- forms poorly, which shows that including reference training time also has positive effects.
Effects of paraphrased data We investigate the effects of using paraphrased data in Tab. 4 where we apply the noise functions to different forms of targets when gen- erating synthetic data. For MT, we create paraphrased targets via knowledge distillation (Kim & Rush 2016) where we use the output from TranS2S conditioned on the source sentence in the bi-text corpus as the paraphrased target. We can see that with distillation data for syn- thetic data generation, the model achieves better results compared to using the references. However, note that we need to choose proper word dropout rate when using the reference- eeference-based synthetic data as discussed above.
Input to M(()
Precision Recall F1
MT
58.35
70.12
63.70
TranS2S distill
64.27
67.30
65.75
Summarization raw
57.02
67.23
61.70
Extractive distill
54.10
36.45
43.55
Ahstractive distill
57.33
28.59
38.16
Table 4: Performance the TranS2S benchmark from MT and summariza- tion by using different data as the in- put to the noised function WV.).. "raw" refers the original targets in the train- ing data.
For abstractive summarization, data out of an abstractive and an extractive summarization systems respectively. We finetune BART on the bi-text of XSUM and create distillation data from this finetuned ab- stractive model. For the extractive system, recent proposed MatchSum (Zhong et al. 2020) as the distillation model. We see a significant drop in the performance for both of the variants. This likely due to the fact that: (1) it has been shown that abstractive summarization systems are prone to hallucinate contents themselves (Maynez et al., 2020), thus we are not able to create reliable pseudo labels based on the generated summaries, and (2) the extractive system generates summaries out of the input article which diverge from the actual abstractive summaries we evaluate on, and the model cannot generalize well under such data shift.
create paraphrased
use the
A CASE STUDY: IMPROVING SELF-SUPERVISED MACHINE TRANSIATION
Predicting hallucination labels at token-level not only allows us to flag potential risks in genera- tion models. but also offers potential improve text generation by defending against hallucinated outputs. Token-level hallucination labels provide fine-grained signals which can be used to define new learning objectives. In this section, show such fine-grained signals allow for improved semi-supervised method for machine translation.
6.1 RECTIFIED TRAINIGG IN NOW-RESOURCE NEURAL MACHINE TRANSLATION
is an important semi-supervised approach that utilizes unlabeled source data to improve system performance. In a conditional sequence generation task, teacher model first trained with bitext Dj {s,, 1 and used to make predictions on each sequence in unlabeled dataset D,,- = د to create pseudo parallel data D, = {s). t' ) N The model then trained on Dj p- He et al. (2020) finds that with self-training the student model can benefit from such pseudo parallel data that acts as regularizer. However, such results require a relatively high quality teacher, and performance suffers in low-resource setting where there isn't t enough parallel data. In these cases, the teacher may hallucinate content when it makes predictions, thus producing noisy parallel data that actually hurts the student model.
Self-training (Seudder 1965)
We propose to use our token-level hallucination predictions in machine translation, by penalizing errors less on tokens that more likely to be hallucinated. This is in strong contrast to previous data filtering in MT, which showed it was possible improve per- formance by removing entire sentence pairs Junczys-Dowman
fine-grained control during training
2018 Kang & Hashimoto 2020).
First, we predict the token-level hallucination labels the target side of the pseudo parallel data D,. Then we propose two simple methods of using these labels in self telf-training: (1) We discard the losses of tokens that are predicted as hallucinations and compute the loss on the remaining tokens for
8

---

Under review as a conference paper at ICLR 2021
each target sequence (token loss truncation). (2) Instead of adjusting losses, mask the decoder hidden states of those hallucinated positions after the target-to-sourg cross attention in each decoder layer (decoder HS masking)
Methods
BLEU (t)
BLERUT (1)
Hal words (%. 4)
bascline
16.14
-0.166
13.69
ST
19.31
-0.059
10.00
ST paraphrase noise
19.05
-0.051
13.48
ST random noise
19.97
-0.041
12.55
ST seq loss truncation
19.91
-0.048
8.26
ST random noise + seq loss truncation
19.37
-0.057
10.06
ST token loss truncation
20.32
0.00244
6.37
ST decoder HS masking
20.57
-0,0001
6.38
ST + random noise + token loss truncation
21.02
0.043
7.34
ST random noise + decoder HS masking
20.64
0.0308
8.70
Table 5: BLEU, BLEURT and percentage of hallucinated tokens on the CWMT2017 patent test
set. We compare with the noised self-training method (He et al. (2020) in the second block and sequence-lever loss truncation method (Kang Hashimoto] 2020) in the third block.
6.2 EXPERIMENT SETUPS AND RESULTS
Experimental Setup To train a teacher model (baseline in Tab. 5). we use the same training data described in $4.2 using patent (870) as the low-resoure domain. We evaluate on the full patent test set (1,500) from CWMT2017 (Wang et al., 2020b). For the unlabeled data, we use the withheld Chinese patent training data (2.9M).
Baselines First, we compare with the state-of-the-art self-training (ST) method (He et al. 2020) that injects noise the inputs. They use two types of noise: (1) Paraphrase noise created by round- trip translations, and (2) Random noise from droping, masking and shuffling input tokens. We also compare with the recently proposed loss truncation method (Kang Hashimoto/ 2020) that adaptively removes entire examples with high log loss, which can reduce hallucinations as shown in the paper.
Results and Analysis We present the tokenized BLEU score (Papineni al. 2002). BLEURT score (Sellam et al.) (020) and the percentage of hallucinated tokens predicted by our system in Tab. We can see that ST improves over the baseline by around 3 BLEU scores and our best result further improves ST by 1.7 BLEU scores. Compared with strong baseline methods, our method not only achieves the best translation quality measured by BLEU and BLEURT but also the largest hallucination reduction. We also observe that: (1) Our method with ST alone can outperform other baseline methods, when combined with perturbed (noise), using fine-grained control over the tar- get tokens can further improve the results. (2) ST with paraphrase noise (by round-trip translations) does not perform as well as the random noise, which further confirms that the noisy outputs from teacher model may hurt the student model. (3) The sequence-level loss truncation approach can improve over the vanilla ST and reduce level of hallucinations as measured by our system. However, the performance drops when combined with the noised ST.
CONCLUSIONS
In this work, proposed generation and created human-annotatee benchmark datasets. We also proposed novel and general- purpose method to learn this task. In the future, hope create large-scale pretrained evaluation model for any datasets or models to be evaluated. We also would like to extend our method to any data-to-text generation scenarios. We also interested in investigating how to leverage detection methods to mitigate hallucination problems in conditional sequence generation.
evaluation task for hallucination detection in conditional sequence
We also tried removing the hallucinated target words before training. This does not perform as well as the above methods, likely because produces many ungrammatical target sentences.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, et al. Tico-19; 19: the translation initiative for covid-19. arXiv preprint (xXXi:2007.01788 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 2xXv:200440055000 2020.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. URL https: //www. wwwweb.......................... org/ anthology/20200 wwt.:20..ccllme aww.aalllaaa.m.m.aaa 747
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. URL https:
ac aclweb. org www.tnhwwwgggoooooooeeoo acl 1aa-mmain... 454
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging application for natural lan- guage inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2214-2220, 2019.
Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pp. 1-10, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 118653/1//1995530 URL httpt::///// / www. a c web. www..o/atggtgoo///eeooe.o. 0400
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Voike Volume 70, pp. 1243-1252, 2017.
Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. Revisiting self-training for neural sequence generation. aInternational Conference Learning Representations (ICLR), 2020.
Marcin Junezys-Dowmun Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 888-895, 2018.
Daniel Kang and Tatsunori Hashimoto, Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020. URL/ttts:: htts::///www..lc..o.... www.:a.wwwwbbbb...c............. org wwtpo/////ggtto.//o/oooooo/e///e
-main. 66
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the
2016 Conference on Empirical Methods Natural Language Processing, pp. 1317-1327, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412 42,26980, 2014.
10

---

Under review as a conference paper at ICLR 2021
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04453 2020.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pp. 28-39, 2017.
Wojciech Kryšcińíski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization, arXiv preprint wwXiv:1910.12844 2019.
Samuel Läubli, Rico Sennrich, and Martin Volk. Has machine translation achieved human parity?
case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4791-4796, Brussels, Belgium, October- 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/D18- 1512. URL wttttts:////t
ac solweb.or wwtp:/ggggaaaaaa oowoogyy//// /
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, Online, July 2020. URL https: / www. ac web.
703.
Chin-Yes Yew Lin and Eduard Hovy. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens (ed.), Text Sunmarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74-8 81, Barcelona, Spain, July 2004, Association for Computational Linguistics.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1192,, 2019.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre- pre-training for neural machine translation. 2020.
Ruixuan Luo, Jingjing Xu, Yi Zhang, Xuancheng Ren, and Xu Sun. Pkuseg: A toolkit for multi- domain chinese word segmentation. CoRR, abs/1906. 11455, 2019. URL https://arxiv:
org/ brgg/bs//100 1906. 11455
Gary Mareus and Ernest Davis. Gpt-3, bloviator: Openai's language generator has no idea what it's talking about. MIT Technology Review, 2020.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan Thomas Medonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020.
George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
Mathias Müller, Annette Rios, and Rico Sennrich. Domain robustness in neural machine translation. arXiv preprint arXiv:1911.03100 2019.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL 2002, 2002.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https::/// /www aclweb. httpeeb.ogggaaahoogggg..
W18-6319
11

---

Under review as a conference paper at ICLR 2021
Ratish Puduppully, Li Dong, and Mirella Lapata. Data-to-text generation with content selection and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9008-6915,
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Association for Computational Linguistics, 8:264- 280, 2020.
Masoud Jalili Sabet, Philipp Dufter, and Hinrich Schüitze Simalign: High quality word align- ments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08725 2020.
H Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363-371, 1965.
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 1073-1083, 2017.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lingnistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 10.18653/v1/P16-1 1162. URL https: / www. acl web.
org/antho wtttchoooyy/P1-11-
Lucia Specia, Najeh Hajlaoui, Catalina Hallett, and Wilker Aziz. Predicting machine translation adequacy. In Machine Translation Summit, volume 13, pp. 19-23, 2011.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv preprint 19X1v:1910.08684 2019.
Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of- speech tagging with a cyclic dependency network. Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics human language technology-volume 1, pp. 173-180. Association for Computational Linguistics, 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, Online, July 2020a. htthtt:::: //wp:///wwwwwww// aallwed web. org/
anthology/20233 acl masimmminn.cc 450
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural
machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, Online, 2020. URL https::///// http::///www... wwwww..lwweboom wwaeee.rr///.... wwt..ee....r..aatoooo////0000/0. anthology/2000
acl main. 326
Yong Wang. Longyue Wang, Shuming Shi, Victor OK Li, and Zhaopeng Tu. Go from the general to the particular: Multi-domain translation with domain transformation networks. In Thirty-Fourth AAA/ Conference on Artificial Intelligence (AAAI), New York, USA, February 2020b.
Sean Welleck. Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2019.
12

---

Under review as a conference paper at ICLR 2021
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), New Orleans, Louisiana, June 2018. ht..sts:s: org/antho 1 0
www. ac web.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint
1609v00009001444 2016.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations, 2019.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang, Extractive summarization as text matching. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020.
Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. Density matching for bilingual word embedding. In Meeting of the North American Chapter of the Association for Computational Lin- guistics (NAACL), Minneapolis, USA, June 2019, URL https////awww..g//////////a 02343
13

---

Under review as a conference paper at ICLR 2021
A HUMAN EVALUATIONS
We asked three bilingual speakers to annotate the Chinese-to-Eine ton-t----ngihhh evaluation set Deval. which is composed of 150 sentences from the test set of Zh-En multi-domain dataset (Wang al., 2020b),
and 100 sentences from the COVID-19 translation benchmark dataset (Anastasopoulos et al. 2020) TICO. TICO contains 6 finegrained domains including Wikisource, Wikivoyage, Wikinews, CMU, PubMed and Wikipedia. we randomly sample 25 examples from each of the four domains - Wik- isource, Wikinews, CMU and PubMed, these 100 samples for evaluation. We train standard base Transformer Seq2Seq and finetune the MBART (Liu et al. 2020) model on the training data from Dirain.
Annotation Guidelines We ask each annotator to read the tokens in the sentence carefully and check if they can be supported by the source sentence:
If there are tokens (or the entire sentence) that cannot be supported by the source, label all the span(s) with color and mark the sentence hallucinated one;
If the evaluator can not understand the entire translation at all, mark the sentence as incom- prehensible:
If all the tokens in the translation can be entailed from the source, mark the sentence faithful one.
We shuffled the orders of sentences so that annotators did not know which translation model was used (TranS2S or MBART). Before the annotation work on the final blind test set, we first perform pilot study on different evaluation set and analyze the annotation results from each annotators. Then we conduct an education session to make sure that all the evaluators can fully follow the guidelines.
Note that we ask annotators to work on the raw sentences, ie. punctuation marks can also be labeled as hallucinations along with the span and we didn't 't apply special treatments to them. And all the evaluations are compared against the raw form of sentences, and we always convert the model predictions subwords to labels the raw sentences. Besides, based annotators labeled the entire span of hallucinated words that may also contain prepositions and other stop words.
guidelines,
For each token in the evaluation set, we label it by majority voting, i.e. the label that two or more annotators agree on. We also aggregate the evaluation data from Maynez et al. (2020) in the same way to produce our own test set for abstract text summarization.
B TRAINING OF NMT MODELS: TRANS2S AND MBART
Models
TranS2S
MBART
dmodel
512
1024
dhiddon
1024
4096
Thaaers
6
12
Aheads
8
16
Paropout
0.1
0.3
Table 6: Basic hyper-parameterss of architecture for NMT models.
Tokenization For TranS2S, we first segment the Chinese corpus with an opensource Chinese word segmentation tool (Luo et al. 2019), then we Iearn separate BPE vocabularies with 32k merge oper- ations (Sennrich et al. 2016 over the source (Zh) and the tokenized target (En) corpus respectively. For MBART, we directly apply the contained sentence-piece dictionary in the finetuned model to the data of Chinese and English corpus.
Model We use the implementation of Transformer model from fairseq eOt al. 2019). Following the notations in Vaswani et al. (2017), show hyperparameters of TranS2S and MBART in Tab. 6
14

---

Under review as a conference paper at ICLR 2021
Training For TranS2S, apply the standard hyperparameters reported in the example of fairseq- The
We use the Adam optimizer (Kingma & Ba 2014) using 81 = 0.9, B3 = 0.98, c = 1e learning rate is scheduled using inverse sqrt with maximum learning rate 0.0005 and 4000 warmup steps. We set the label smoothing as 0.1. I. We apply dropout 0.1 and select the hest model with validation BLEU scores. We the model 8 GPUs for 300, 000 updates with an effective batch size of around 64, 000 tokens.
When finetuning MBART, we use learning rate of 3e-5, and use po poomi a al_decay for learning rate scheduling with warmup updates of 3,000. The effective batch size is 16,384. Dropout is set to be 0.3 and the attention dropout rate is 0.1. The label smoothing to be 0.2. We finetune MBart for 60,000 updates.
Decoding After training. we use beam-searde with a fixed beam size 5 for all the models and set the maximum generation length be twice the source length.
C EXPERIMENTAL DETAILS FOR TOKEN-LEVEL. HALLUCINATION PREDICTION
Subword Tokenization Depending on the pretrained model (Roberta / (.MM-Robbeta)) we finetune on, we apply corresponding subword segmentation the synthetic data set (S, T, T') and calculate the edit-distance between the 'I' and T' at the subword level. At evaluation time, the model predicts the hallucination labels for each subword the sentence, thus we predict a word to be hallucination word if any subword of it is predicted as a hallucinated one.
Synthetic data generation There are a couple of hyperparameters of noised functions in the BART implementation (Ott et al. 2019). The main noised functions include (1) random mask- ing (randomly replace word with MASK), (2) random replacement (randomly replace a word with another word in the vocabulary), (3) random insertion of masks. We found that random masking and random replacement are the two key factors affecting the generated sentences and we have provided their settings in the main paper. We apply a random insertion masks rate of 0.2 for all settings. In addition, the noise functions are applied to words instead of spans in setting.
Finetuning For MT, we finetune a large M---Robbrta (Conneau et al. 2020) released in fairseq (Ott al., 2019) which is trained on 2.5T of filtered CommonCrawl data in 100 languages. For summarization, we finetune a large Roberta (Ott et al. 2019) on the synthetic data where we For both models, we use the Adam optimizer (Kingma & Ba 2014) with 81 = 0.9, B2 0.98, = le 6 and weight decay of 0.1. We set the masking probability be 0.35 for Lmim loss. The dropout and attention dropout rates are set to be 0. We adopt polyyell collymmiii a 1_decay for learning rate scheduling with learning rate of 2e-5.
truncate articles that exceed 512 tokens (allowed by the Roberta) to 512.
D RESULTS AND ANALYSIS
D.1 HALLUCINATION STATISTICS ON PATENT AND COVID-19
Patent
COVID-19
Models
True hal. toks (%)
Pred hal. toks (%)
True hal. toks (%)
Pred hal. toks (%)
TranS2S
22.78
22.83
9.44
11.79
MBART
8,84
6.94
6.39
5.55
Table 7: Comparisons of TranS2S and MBART Patent and COVID-19 domains. True hal. toks and Pred hal. toks are the ground-truth and our model predicted percentage of hallucinated tokens in the benchmark test set for each domain.
We present the percentage of hallucinations from the annotations and our model predictions for the Patent and COVID-10 evaluation data respectively in Tah. We that both the TranS2S and
15

---

Under review as a conference paper at ICLR 2021
MBART models produce less hallucinations for the COVID-19 19 domain although is complete out- of-domain test set that has not seen in the training data. We suppose that this is because sentences in COVID-19 are more alike the training domains in terms of writing styles which bring positive transfer. We also note that the predictions from our model match pretty well with human annotations for both of the domains.
D.2 COMPARISONS ON SENTENCE-LEVEL HALLUCINATION SCORES
Entail
Align
Ours (P)
Ours (R)
Gold (%)
MT
TranS2S
-0.32
0.23
0.78
0.79
64.1 I
MBART
-0.19
0.05
0.36
0.48
74.3
Summarization
PtGen
0.25
0.32
0,43
0.42
89.4
TConvS2S
0.26
0.31
0.43
0.42
92.8
TranS2S
0.13
0.19
0.35
0.33
92.4
BERTS2S 0.23
0.27
0.37
0.36
88.2
Spearman's correlation coefficients of the sentence level hallucination by different systems with the human annotations (percentage of hallucinated tokens in a sentence). Gold %) is the percentage of sentences that contain hallucinations in the test
Table 8:
computed
We aggregate the token-level predictions to score the level hallucination of each sentence and compute the Spearman's 's correlation coefficients between scores and the human annotations of percentage of hallucinated tokens in a sentence. In Tab, g we present two scores from our model: (1) Probability score (P) computes the score by averaging the halluciation probabilities across all the tokes in G. (2) Ratio-based score (R) first predicts the hard labels by taking arg max at each position and then the percentage of hallucinated labels the score.
We compare with two baseline metrics that do not require reference as well: (1) Textual entail- ment prediction model (Entail) has been proposed to evaluate the faithfulness of abstractive sum- maries (Maynez et al., 2020 Kryšciñski et al., 2019 Falke et al.) 2019). We adapt it to MT by finetuning XLM-Roberta on the machine translated Zh-En Multi-NLI dataset (Conneau al. 2018). For summarization, we use the finetuned Roberta- large (Ott et al. dataset (Williams et al. 2018). We calculate the entailment probability Pa between the source input and the machine output G and use 1 Pe to score its hallucination level (which is better than use of contradiction probability in our experiments).
2019) on the Multi-LL
(2) For the alignment-based method (:5.2). we compute the percentage of tokens in the machine output G that get aligned aligned to the source input to measure the faithfulness G.
Overall, in Tab. 8 we see that: (1) For MT, achieve very significant improvements over the base- line metrics. And both entailment and alignment scores correlate very poorly with the hallucination assessments. (2) For abstractive summarization,
metrics also significantly outperform baseline metrics. The alignment scores stand out as a better hallucination assessment compared to the com- monly adopted entailment scores. We also note that for MT our ratio-based scores correlate better with human judgements while for abstractive summarization the probability- -based scores have a better correlation. This shows that our model can produce more confident predictions for MT.
D.3 FULL RESULTS OF TOKEN-LEVEL HALLUCINATION PREDICTIONS
We found the synonym and string-matching based methods are strong and effective baselines the monolingual (summarization) token-level hallucination prediction task as an alternative to neu- ral methods. However, previous work (Maynez et al. 2020; Wang et al. 2020a; Durmus et al., 2020) on hallucination assess did not study synonym based non-neural baselines when measuring the faithfulness of the summarization model outputs.
16

---

Under review as a conference paper at ICLR 2021
Methods
TranS2S
MBART
Aligument
(18.90, 66.82, 29.47)
(5.63, 42.09, 9.93) (1.98, 8.97, 3.24)
Overlap-based
(7.02, 13.10, 9.14)
Synonym-based
Ours (w/o reference)
(64.27, 67.30, 65.75)
(49.56, 36.32, 41.92) (43.13, 53.63, 46.81)
Ours (w/ refenrence)
(59.92, 74.27, 66.08)
Table 9: Triplets represent (Precision, Recall, F1 (x (x100)) of hallucination labels on the outputs of different systems from a machine translation task (.2e) The first block are baseline methods and the second block are our results. We highlight the best results without using reference.
Methods
PtGen
TConvS28
TranS2S
BERTS2S
Alignment
(60.66, 28.65, 38.92)
(66.14, 26.60, 37.94)
(56.24, 24.85, 34.47)
(50.68, 27.69, 35.81)
Overlap-based
(67.72, 49.54, 57.22)
(60.39, 49.24, 54.25)
(53.22, 54.37, 33.79)
(62.57, 49.26, 55.13)
synonym-based
(50.52, 72.50, 59.55)
(57.06, 72.16, 63.73)
(50.29, 70.37, 58.66)
(41.80, 72.67, 53.07)
Ours (w/o ref)
(57.47, 71.35, 63.66)
(63.21, 68.93, 65.94)
(57.02, 67.23, 61.70)
(49.83, 62.50, 55.45)
Ours (w/o ref syn)
(50.33, 90,27, 64.72)
(56.86, 88.93, 69.37
(50.21, 87.78, 63.88)
(41.70, 87.52, 56.49)
Ours (w/ ref)
(56.51, 73.48, 63.89)
(61.68, 71.63, 66.28)
(55.88, 70.19, 62.24)
(48.39, 66. 11, 55.88)
Table 10: Triplets represent (Precision, Recall, Fl (x100) of hallucination labels on the abstract summarization task (XSUM dataset). The first block are baseline methods and the second block are our results. We highlight the best results without using reference.
D.4 ANALYSIS ON PART-OF-SPEECH TAGS AND WITH-IN GROUP HALLUCINATION PERCENTAGE
Inghonp Helluciaatio Ratt
MT
Halucraate crttato Ratte
XSum
Gurd
2.0
Gold pred ctans
0.33
predid Ora
0.8
Ovedap Baseline
0.2
0.6
0.1
0.2
NN
VB
RB
A
RB
NN
IN
V9
POS teg
tag
Figure 7: Analysis of part-of-speech tags and with-in-group percentage of hallucinations for ma- chine translation (left) and summarization (right).
We have shown that the macro Part-of-Speech tag distribution of hallucinated tokens in $5.3 In this section, we analyze the micro-percentage of hallucination labels within each POS tags. We show the gold annotations well our model predictions of hallucination words within each POS tags. For summarization, we also show the results from the string-matching baseline. From Fig. 7 we can see that for MT nouns are most likely hallucinated words while for summarization cardinal numbers (e.g. one, two) are most likely hallucinated words. And we can see that our model predictions align well with the gold annotations on the percentage of hallucinated words within each POS tags.
D.5 ANALYSIS ON SYNONYM PREDICTION ERRORS
In Tab. II we analyzed the error rates of hallucination predictions of the summarization data sets w.r.t. the synonyms of source words. Specifically, we first obtain the synonyms of nouns, adjectives, adverbs and verbs through WordNet (Miller 1998), then for each word that belongs one of the mentioned POS tags in the model output we also obtain their synonym set. If the synonym set of a target word has overlap with the synonym set in the source article, we check if model predicts this word hallucinated word (if so, it's error). In this way, model performs on synonyms, and we hope that our model can correctly identify synonyms as non- hallucinated words. However, found that our model performs worse synonym predictions
analyze how well our
17

---

Under review as a conference paper at ICLR 2021
Methods
Noun Synonyms
All Synonyms
String-mate match baseline
15.37
17.40
Ours (w/o reference)
26.36
23.22
Ours (w/o reference, filter overlapped NN with source)
10.05
13.62
Table 11:
Error rates of hallucination predictions target words that synonyms (predicted by WordNet) of words in the input on XSUM test sets.
than the maximum input length of 512 subwords and many subwords of the source article prediction time, which might be one reason that our model fails to identify the synonyms due to losses of source information. We integrate the string matching based method at prediction time in a conservative manner where we always assign non-hallucination labels to noun target tokens that appear in the source. As shown in Tab. [T (3rd row), can reduce the rates of synonym predictions by a large margin, Although we observe an increased precision, the integration of string- matching also leads to decreased recall and the FI is not improved.
baseline. As we pointed out in 55. the model we finetune on only permits
off
D.6 EXAMPLES OF PARTIALLY HALLUCINATED OUTPUTS FROM TEACHER MT MODEL
Source
Reference
the set of information is called page data. the foreign[1] mix[1] is called the page data.
Generation
Source
Reference
the metal lines correspond to first resistors.
Generation
the wire corresponds with the first capital[1].
Source
T T.MAAR 1100m.
Reference
the average droplet sizes are below 100 nm.
Generation
the average exhaust[1] dimensions of the droplets were less than 100 cubic[]] meters[l].
Source
Reference Generation
driving samples to flow through flow channel; driving samples pass the flow of people[1];
Table 12: Examples of partially hallucinated outputs from the teacher MT model used in self-training and the hallucinated labels predicted by our system. We only highlight words with hallucination labels with [1].
In Tab. 12 we randomly select some examples for which we present the source sentences from the patent monolingual Chinese dataset, the corresponding reference English sentences and the gen- erations from teacher model trained on the training data described in 44.2 where patent is a low- resource domain. We and the rest of the outputs are good translations that are faithful to the source. Through our approach in {
that in these examples, only parts of the model outputs are hallucinated
we can still make use of these good parts of translation during training.
D.7 EXAMPLES HALLUCINATION PREDICTIONS ON THE MT SET
In Tab. 13 we present some examples of annotations and our model predictions. We can see that our model performs well in general but can be inaccurate in case spelling emrors the translations. Besides, we also find some annotation errors while our model predicts correctly.
18

---

Under review as a conference paper at ICLR 2021
Reference
the arrangement pattern of the projections 2 will now be explained with reference to figs. 5-7.
Annotation
next,[O] we[O] use[0] fig-[0) 5[0] -[0] 7[0] to[0] explain[0] the[0] dispo-
sition[O] pattern[O] with[0] pm-2.[1]
Prediction
next,[0] we[O] use[0] fig-[0] 5[0] -[0] 7[0] to[0] explain[O] the[0] dispo- sition[O] pattern[O] w1th[1] pm-2.[1]
Reference
a swivel joint 557 is provided in radially outer region, on an end sur- face of the drive plate 556.
Annotation
a[O] rotation[O] hinged[1] 557[0] is[O] provided[O] to[0] the[0] exter- nal[O] area[0] on[O] a[O] trail[1] that[O] has[O] a[O] preface[1] state.[1] a[0] rotation[O] hinged[0] 557[0] is[0] provided[1] to[0] the[0) exter-
Prediction
nal[O] area[O] on[O] a trail[1] that[1] has[1] a[O] preface[1] state.[1]
Reference
is small, the kinetic energy of the droplet is small as compared to the air resistance.
Annotation
on[0] the[0] other[0] hand,[0] radius[O] r.[0] which[O] is[0] shorter[1] [] than[1] the[0] hour[1] in[0] which[ 1] it[1] can[1] be[1] used.[1] is[0] smaller[O] than[0] the[0] air(0) resistance.
Prediction
on[O] the[O] other[O] hand.[O] radius[O] r,[0] which[O] is[0] shorter[1] I]
than[0] the[0] hour[0] in[1] which[1] it[1] can[1] be[1] used.[1] is[0] smaller[O] than (0 the[0] air(0) resistance.[0]
Reference Annotation Prediction
if you have a fever of a hundred and two or higher. if[0] your[0] heat[O] reaches[O] 102.d[0] egree.[O] f.[0] or[O] above.[]] if[0] your[O] heat[0) reaches[O] 102.d[1] egree. f.[1] or[0] above,[0)
Table 13:
Examples of annotations and model predictions. [0] indicates faithful word while [1)
indicates hallucinated word.
19


Reviews:
[{"review_id": "whrv7HTlmer", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new task: Word-level hallucination detection in text generation, where the authors identified two scenarios of word-level hallucination (content insertion and incorrect substitution). \n\nThen the author synthesized hallucinating text by de-noisng auto encoder and use dynamic programming to infer hallucination labels. They are used to fine-tune pertained LMs for detecting hallucination. \n\nExperiments show that the fine-tuned LMs can achieve certain performance compared with human annotations, and that the token-level hallucination is correlated to sentence-level hallucination.\n\nConcerns:\n\n1) Hallucination is recognized as an important issue for text generation, and intuitively, detecting hallucination is also an important task. But the real benefit of detecting hallucination is unclear from this paper. I would be more convinced if the authors could show how their word-level hallucination can indeed help text generation. \n\nCurrently, the paper appears to be some fine-tuned LM for a new task (whose real benefit is unclear) with little technical depth. At this stage, I am unsure if this paper has enough contribution as an ICLR paper.\n\n2) The categorization of \"content insertion\" and \"incorrect substitution\" is unclear and confusing. \n\nIs there any breakdown analysis on \"content insertion\" and \"incorrect substitution\"? For the below example, what type is \"Monday\" given Input1 or Input2, respectively?\n\nInput1: We had a meeting in that room.\n\nInput2: We had a meeting.\n\nGen: We had a meeting Monday.\n\n3) There's very little technical development and empirical analysis on the role of synonyms and stop words in hallucination. \n\nA synonym is supposed to have the same meaning as the original word, and is not hallucination. Synonyms are actually desired in many applications, like paraphrase generation and summarization. \n\nStop words mainly serve for grammatical functions, and may not be directly related to content hallucination.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "k-oLq98Oxaq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary: \nThis paper proposes hallucination detection at the token level, which predicts if each token in the generation output is hallucinated or faithful to the source input. In contrast, previous studies usually work on the sentence level. To create synthetic training data, a denoising pre-trained LM is first used to generate (potentially) unfaithful counterparts T’ of the references T. Then, token-level labels are obtained by comparing T and T’ via edit distance. Finally, a standard classification model is trained on the token-level labels by concatenating the source S, true and unfaithful targets T (T’).\n\n---\nPros:\n+ Overall, the paper is clearly presented with detailed experiments and ablation studies. It shows that the proposed method works better than existing entailment-based and alignment-based methods on the new task that it proposes.\n+ The new annotated data on token-level hallucination could possibly be useful for future research in similar directions.\n\n---\nCons:\n- My major concern is about the effectiveness of defining hallucination at the token level itself. First of all, it looks un-intuitive to me whether it makes sense to define hallucination for every type of token. As the task aims to label every token in the text sequence, I wonder how different types of tokens (e.g., tokens with different POS tags) contribute to the model hallucination. The proposed metrics treat all the tokens equally, while in reality, tokens such as noun phrases or verb, for example, may have a larger impact on the hallucination issue than prepositions or articles. Similarly, for the creation of synthetic training data, I wonder whether it makes sense to replace tokens with different POS tags uniformly. It will be good to present more analysis in terms of both quantitative results and case studies on this aspect. \n- Is the evaluation on all the tokens or only the hallucinated tokens? Could the decent performance (F1) of the proposed method come from the fact that most tokens are not hallucinated (labeled with 0)?\n- For comparisons on sentence-level hallucination (of abstractive summarization), why don’t you compare with the baselines presented in the related papers you listed directly? \n\nTypo: \"if each token in the machine output is **a** hallucinated or faithful to the source input\"\n\n---\nComments after reading author response and revised paper \n\nThanks for showing more results of how token-level hallucination detection can be useful/effective: e.g., (i) labels per POS tagging in Fig. 5 and (ii) application to low-resource MT in Sec. 6.\n\nFor (i), I don't think it resolved my question directly: \"The proposed metrics treat all the tokens equally, while in reality, tokens such as noun phrases or verb, for example, may have a larger impact on the hallucination issue than prepositions or articles.\" That is, a hallucinated NN, for example, might be worse than a hallucinated II, rather than asking how the labels are distributed by POS categories. So I still wonder if it makes sense (as a reliable metric) to measure hallucination at the token level (e.g., Table 5 Hal words %) but it remains unanswered.\n\nFor (ii), it is indeed an interesting plus to the paper, showing that the token-level labels appear to be useful for downstream applications (even if it's not quite meaningful to measure the % of hallucinated words). I would suggest doing more studies on downstream tasks as mentioned in your response to enhance the paper if you are \"not proposing a reference-free evaluation metric for quality estimation\" (which seems a bit contradictory to \"we hope to create a large-scale pretrained evaluation\nmodel for any datasets or models to be evaluated\" in the conclusion section btw). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gMcBPcwKWmq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper addresses the problem of \"hallucinated\" content in conditional neural generation for two specific tasks: machine translation and summarization. It proposes a new task for faithfulness assessment, which classifies each token as either hallucinated or not. The classifier uses a pre-trained LM (either XLM-R or ROBERTa) and is fine-tuned on synthetic classification data created using both 'noisified' real data and a pretrained LM (BART). Experiments on either summarization and MT system outputs labeled for hallucinations show relatively encouraging classification results (e.g., F1 of 0.46 to 0.66 for MT, and 0.56 to 0.66 for summarization).\n \nPros:\n \nAddressing the hallucination problem is essential, as inserting non-factual content is probably the most harmful error a machine translation or summarization system can make.\n \nThe paper contributes a new dataset for training classifiers for hallucination detection and a methodology for creating new ones relatively easily. That said, the dataset is partially synthetic and may not reflect the kind of hallucinations actual neural MT and summarization systems make.\n \nThe paper contributes an evaluation task for assessing the faithfulness of MT and summarization output. It could be used, e.g., to flag or prevent output whose content is deemed inaccurate. The paper shows an improved correlation (Spearman) relative to an entailment metric and a word alignment metric.\n \nCons:\n \n1. Lack of experimental comparison: I find it hard to conclude much from the paper's main results, as it doesn't provide a comparison across different hallucination detection systems or models. In general, it is difficult to tell if a given absolute score (e.g., F1 = 0.60 mentioned in the introduction) is good or bad without a point of comparison, especially if the task is new and there is relatively little information about the type of content errors the underlying systems make. I understand that the authors are trying to establish a new task and that there is little work to compare against, but the paper could have provided more in the way of ablation experiments.\n\n2. Results lack interpretability: Two of the main tables (Table 2 and 4) do not seem to contribute much, as the different rows correspond to different datasets (underlying data to classify is different) and are therefore strictly not comparable. For example, Tab. 2 suggests that the classifier does a much better job on TransS2S (F1 = 0.66) than on MBART (F1 = 0.47). However, Tab. 5 indicates that MBART is much less prone to producing hallucinated words than TransS2S, which probably explains why classification results on MBART data are worse. So it seems Tab. 2 and 5 together suggest classification results are lower when the classification task is harder, which wouldn't be a surprising finding.   \n\n3. Contribution as a reference-free evaluation metric is suspect and possibly subject to gaming: The authors point out that nearly all existing evaluation metrics (BLEU, METEOR, BLEURT, BERTScore) require reference text, which is characterized in the paper as a disadvantage. This is a bit of an odd argument to make for an *evaluation* metric, as this implies both the metric and the model are only given a source string to achieve their purposes. If the evaluation metric isn't given additional information (in the form of, e.g., a gold standard or reference), then on what basis is the metric supposed to do a better job at assessing the goodness of a target string, given that both the metric and the model are machine-learned using the same amount of input signal? If an evaluation metric is reference-free and proves indeed useful, then one can and probably should incorporate it directly inside the model or system (e.g., as a feature function or in a fine-tuning setup).  But incorporating the hallucination detection model as part of the generation model would render the former model rather useless as an *evaluation* metric. Note: I understand there is also a reference-based setup (which makes little difference), but the reference-free aspect is emphasized in the paper.\n\n4.  No evaluation on any downstream task: But this could easily have been done as the model for hallucination detection does not require references: see point 3.\n \nOverall, I think this paper's general direction with fine-tuning on hallucination classification data is probably worth pursuing. However, I think the paper's actual impact in its current form could be fairly limited, given the cons listed above. It is hard to draw conclusions from the main results of the paper (cons 1-2); the use as an evaluation metric is rather questionable (con 3); there is no application of the work on any end-to-end task (con 4).\n \nOther comments:\n \nTables 2 and 4 are somewhat misleading in their references to different \"models.\" Each of these tables evaluates hallucination classification with a *single* model, but the Models column refers to different models used to generate the underlying classification *data*. I think this could easily be misunderstood by the reader, as the established practice is that \"models\" refer to models for the task at hand (i.e., classification).\n \nExperiments on hallucination detection are on MT and summarization, but the paper (and introduction in particular) discuss more generally \"conditional neural sequence models\" and mention other tasks such as free-form QA (Fan et al., 2019) and indirectly LM-based generation (Wang and Sennrich, 2020). By being reference-free, the work would, however, only apply to semantics-preserving tasks (output conveys and same meaning as the input or a subset) as summarization as MT. Otherwise, I don't see how to evaluate whether something is hallucinated without any reference.\n \nMissed related work:\n\nImproved Natural Language Generation via Loss Truncation. Daniel Kang, Tatsunori Hashimoto. ACL 2020.\n\nSticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation. Ran Tian, Shashi Narayan, Thibault Sellam, Ankur P. Parikh.  arXiv 2019.\n\n\n===========\nUpdate: Thank you for your clarifications and updated paper, which addressed several of my concerns. I therefore increased my score by 1.\n\n> First, we want to stress that we are not proposing a reference-free evaluation metric for quality estimation. \n\nI understand it is not meant as a general quality estimation metric, and my point was more about the *reference-free* aspect. I tried to make a broader point which I think applies to any kind of reference-free metric (whether it is for quality estimation or specifically about hallucination).", "extra_scores": {}, "extra_text": {}}, {"review_id": "6WccRTbT1OI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThis paper presents a method to detect hallucinated tokens in generations from neural machine translation and summarization. Given a source input S and its output G generated by a sequence generation model, this paper formalizes the task of detecting hallucinated tokens as a labeling problem on the output G. In order to train the labeler, the method synthetically generates supervision data by using a BART model. The BART model receives a text with noises ([MASK] tokens) and tries to predict [MASK] tokens. In this way, the method obtains a pseudo hallucinated text T' from a text T, and assigns hallucination labels by estimating edit operations between T and T'. The labeler is trained by fine-tuning pre-trained cross-lingual (for MT) and mono-lingual (for summarization) language models. In training, the labeler receives a source text S, true target text T, and pseudo hallucinated text T' separated by [SEP] tokens and tries to reproduce the hallucination labels on T'. Receiving a source text S and its output G, the labeler predicts hallucination labels on G during the inference time.\n\nThe experiments use the XSUM dataset for summarization and multi-domain Chinese-English (Zh-En) translation dataset for machine translation. The authors manually prepared test sets for evaluating the hallucination labeler. In Section 6.2, the authors reported that they achieved decent performance on this task of hallucination labeling although it is far from being solved. They also aggregated token-level hallucination predictions to sentence-level scoring and computed the Spearman's rank correlation coefficients between the scores and the percentage of tokens annotated as hallucinations by humans. The comparison between the proposed method and two baselines (entail and align) show that the proposed scores correlate well with human judges than the baselines.\n\nPROS\n\nThis paper is clearly written and easy to read.\n\nThe methods used in this study, e.g., BART for generating pseudo hallucinated text and (XLM-)RoBERTa for labeling hallucinated tokens, are appropriate.\n\nCONS\n\nFindings from the experiments are unclear. In addition, this paper does not explicitly explain how the research outcomes contribute to an advance in MT or summarization. In particular, I'm not sure whether the performance of Table 2 was successful enough to advance the research in MT and summarization. The goal of the correlation analysis in Table 3 is unclear. I guess that the initial motivation for this experiment was to develop an automatic method for evaluating hallucinations, but the current experimental results do not support this so strongly.\n\nQUESTIONS\n\nPlease report the number of hallucinated and faithful sentences in the test set. This paper explains that the test set for machine translation included 250 instances initially, but removed instances judged as incomprehensible. I have no idea how many translations were hallucinated and faithful. This may be important because the Entail baseline works not at token level but at sentence level (it should not be used on a dataset only with non-entailment instances).\n\nI would like to see results of a simple baseline in Table 2. For example, we can consider a baseline that indicates hallucinations for tokens that do not appear in the source text. Currently, I'm not sure how good the performance values of Table 1 are. Hence, I have no idea whether the method for generating pseudo supervision data and for detecting hallucinations at token level was a worthy contribution.\n\nI am wondering of the usefulness of the evaluation with Spearman's rank correlation coefficients. Here is a summary of the methods.\n\nReference values: the percentage of tokens annotated as hallucinations by humans.\n\nProposed model (1): the average of hallucination probabilities across all the tokens.\n\nProposed model (2): the percentage of tokens labeled as hallucinations by the model.\n\nEntail: the probability estimate of entailment relation between the source and output.\n\nAlign: the percentage of tokens aligned to the source input estimated by SimAlign.\n\nThe Entail baseline was not designed to work at token level but at sentence level. There is no guarantee that a probability estimate from the baseline represents the 'degree' of entailment relation that corresponds to the percentage of tokens. The comparison would be fairer if the proposed model could yield a binary decision about hallucination/faithful at sentence level.\n\nIn addition, Spearman's rank correlation coefficients of the Entail baseline on the MT dataset were mostly negative (weak inverse correlation). If -0.32 could be obtained by chance, we can conclude that coefficients around 0.3-0.4 do not indicate a correlation. I also suspect that this baseline could not work well because most instances on the MT dataset were non-entailment instances.\n\nP3: Figure 2\n\nDoes the BART model often produce multiple tokens (e.g., \"with friend happily\") from a single [MASK] token? This example is impressive, but I think that a BERT model mostly predicts a single token for a single [MASK] token.\n\nP3: \"we consider any spans in G that misrepresent S as hallucinated contents\"\n\nThis treatment may cause an inconsistency in labeling hallucinated tokens. How does this treatment affect the evaluations in Section 5?\n\nP4: It was difficult to follow the description of \"paraphrase\". How does this study generate paraphrases?\n\nMINOR COMMENTS\n\nP7: \"aligned aligned\" -> \"aligned\"\n\nCOMMENTS AFTER THE REVISION\n\nIt was amazing to see the authors updated paper with the new experiment on the baselines (Table 2) and machine translation (Section 6). Although the baselines (overlap and synonym) were strong, I can now see that the proposed approach is better than these simple baselines both for machine translation and summarization. Section 6 also demonstrates the usefulness of this work for machine translation trained with self-training. For this reason, I increased my rating.\n\nThe impact of this paper would be greater if Section 6 could include more results on different MT datasets (e.g., WMT) and/or self-training for summarization.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper proposes the novel task of detecting hallucinated tokens in sequence
generation, and a strategy to train such models using artificially generated
samples. The methods show reasonable correlation with human judgements.

The expert reviewers are unanimous in their lack of enthusiasm
about this work, with overall borderline assessments. The
reviewers provided some suggestions for improvement, and it is worth remarking
that the authors provided an impressive amount of work in the revised version,
addressing the suggestions. Specifically, they added baselines that validate that
the task is non-trivial, and the case study on improving machine translation.

In the discussion period, the reviewers appreciated the additions, and some
increased their rating, but the overall assessment remains borderline.  The
reviewers find the work lacks the expected amount of depth.  Some concerns
emphasized in the discussion period involve insufficient empirical analysis
(e.g., more NMT datasets and and analysis); understandable as this work was
added after submission, but still important.  A reviewer stresses concerns about
the definition of the task itself, which I agree is vague ("... cannot be
entailed by the sentence") and does not match the synthetic data generation
entirely, leading to unfortunate edge cases involving synonyms or -- worse --
slight narrowing that technically would still be entailment but maybe should be
considered unfaithful.  This casts doubt on the human evaluations and on
considering the task itself a main contribution, therefore leading to the
empirical framing that the reviewers perceive and expect.  It also seems to me
that there is a incremental, cat-and-mouse spirit to predicting automatically
generated hallucinations. In short, it seems like this paper is caught in
between trying to be a significant empirical contribution and a linguistically
well-motivated task and annotation project, and I understand that the reviewers
would prefer committing to one of these directions.

While I encourage the authors to pursue this direction more deeply,
in light of the borderline reviews, I do not recommend acceptance.
Decision: Reject