Detecting Hallucinated Content in Conditional Neural Sequence Generation

1 INTRODUCTION . Neural sequence models have achieved impressive breakthroughs in a wide range of applications , including data-to-text generation ( Puduppully et al. , 2019 ) , machine translation ( Vaswani et al. , 2017 ; Wu et al. , 2016 ) and text summarization ( Rothe et al. , 2020 ) . Although these models can generate fluent sentences that are even sometimes preferred to human-written content ( Läubli et al. , 2018 ; Brown et al. , 2020 ) , recent work has also shown that they lack global logical consistency ( Marcus & Davis , 2020 ) , sometimes degenerate to dull and repetitive outputs ( Welleck et al. , 2019 ) and can often hallucinate content that is not entailed by the input ( Maynez et al. , 2020 ) . In this paper , we focus on the faithfulness of machine outputs in conditional sequence generation tasks , aiming to automatically identify and quantify content in the output that is not faithful to the input text . This risk of generating unfaithful content impedes the safe deployment of neural sequence generation models . The first step to building models that do not suffer from these failures is the assessment and identification of such hallucinated outputs . Prior work has shown that standard metrics used for sequence evaluation , such as BLEU scores ( Papineni et al. , 2002 ; Post , 2018 ) , ROUGE ( Lin & Hovy , 2004 ) and BERTScores ( Zhang et al. , 2019 ) , do not correlate well with the faithfulness of model outputs ( Maynez et al. , 2020 ; Wang & Sennrich , 2020 ; Tian et al. , 2019 ) , and they also require reference text , limiting their applicability to detecting halluciations in a deployed system at run-time . Very recent efforts ( Maynez et al. , 2020 ; Durmus et al. , 2020 ; Wang et al. , 2020a ) have started to develop automatic metrics to measure the faithfulness of output sequences . These methods use external semantic models , e.g . the question-generation and question-answering systems ( Wang et al. , 2020a ; Durmus et al. , 2020 ) or textual entailment inference models , to score faithfulness tailored for abstract text summarization . However , these scores do not directly measure the number of hallucinated tokens and only correlate weakly with human judgements due to compounded errors . We propose a new task for faithfulness assessment - hallucination detection at the token level , which aims to predict if each token in the machine output is a hallucinated or faithful to the source input . This task does not use the reference output to assess faithfulness , which offers us the ability to apply it in the online generation scenario where references are not available . Similar to the spirit of our proposed task , word-level quality estimation ( Fonseca et al. , 2019 ) in the machine translation community predicts if tokens are correctly translated based on human post-editing . However , they do not distinguish errors in terms of fluency and adequacy ( Specia et al. , 2011 ) . In contrast to estimating the amount of human post-editing work required to fix errors , we specifically focus only on hallucination ( not fluency ) errors . We measure hallucination for two conditional sequence generation tasks – abstractive summarization and machine translation ( MT ) . For the former , we produce a benchmark dataset from the recent released annotations in ( Maynez et al. , 2020 ) . For MT , we carefully design the human assessment guideline and create high-quality annotations . We will also release our human annotated data for future research . To learn token-level hallucination prediction for general conditional sequence generations tasks , we propose a novel method that creates synthetic “ hallucinated ” dataset with pseudo labels and finetunes a pretrained language model ( Liu et al. , 2019 ; Conneau et al. , 2020 ) on it . Without any human annotated supervised training data , we achieve an average F1 of around 0.6 across all the benchmark datasets , setting initial performance levels for this new task . We also show that pretraining on MT can actually produce more faithful translations , confirming recent findings in abstractive summarization ( Maynez et al. , 2020 ) . Predicting hallucination labels at token-level provides a tool for diagnosing and interpreting model outputs , which allows us to flag potential risks at inference time for previously unseen inputs . On the other hand , the token-level labels also offer possibility of fine-grained controls over the target sequence to improve the generation . We show how to use these token-level hallucination labels to improve self-training in low-resource MT , where the teacher can produce hallucinated outputs that are harmful to the student model . However , many outputs are only partially hallucinated ( see examples in Appendix D.6 ) and the rest of the output is still useful for training , as we show by introducing different token-level loss truncation schemes . Our best method outperforms strong baselines by a large margin both in translation quality and hallucination reduction . 2 TASK : HALLUCINATION PREDICTION AT TOKEN-LEVEL . For a source sequence S and its model generation G from a neural conditional generation model , following Maynez et al . ( 2020 ) we define any span wi , · · · , wi+j ( j > = 0 ) in G as being hallucinated if it can not be entailed by the source input S. More specifically , we consider two not mutually exclusive types of hallucination : Content Insertion : a span wi , · · · , wi+j in G consists of additional content that is not supported by S , i.e . its paraphrase or other equivalent form can not be inferred from S. In Fig . 1 , the word “ happily ” in the machine translation belongs to this case . This is also referred as “ extrinsic hallucinations ” in Maynez et al . ( 2020 ) . Incorrect Substitution : a span of word ( s ) is misrepresented information based on the S. In Fig . 1 , “ Jerry ” in the machine translation is a hallucinated word and should be replaced by “ Mike ” . This type of hallucination is similar to the concept of “ intrinsic hallucination ” in Maynez et al . ( 2020 ) . Note that there are cases where certain words ( e.g . “ This is not a great book. ” becomes “ This is a great book. ” ) are dropped inG and hence the meaning of S is changed , and we consider any spans in G that misrepresent S as hallucinated contents ( e.g . the entire sentence of “ This is a great book. ” ) . We aim to identify all the span ( s ) satisfying the above conditions in the machine generation G.1 Note that the above definition is only used in the guidelines for human annotators , who do not need to distinguish between these types rigorously . 1We do not consider the under-generations e.g . the source input is only partially translated or summarized . 3 LEARNING TOKEN-LEVEL HALLUCINATION DETECTION . We propose a general-purpose method for token-level hallucination detection for conditional sequence generation tasks . Given the source input S , we first formulate the task of token-level hallucination detection as a sequence labeling problem where a binary label is predicted at each position Gt of the machine generation G. One straightforward way of learning such task is to train a model with supervised data in the form of ( ( S , G ) , LG ) where LG are the labels at every positions of G that indicate if each word is a hallucinated one or not . However , because such labeled training data is not readily available , we also propose a approach to automatically create synthetic training data . 3.1 SYNTHETIC DATA CREATION . We use bi-text from the training data to create synthetic examples by automatically inserting new , hallucinated tokens . More speficially , we take target sequence T and create a hallucinated version of it denoted T ′ with associated hallucination labels for each token in T ′ . Then we can train a supervised model on this synthetic labeled data set of ( ( S , T ′ ) , LT ′ ) . The key challenge is that T ′ should be a fluent sentence that does not differ too much from T . Generation of hallucinated sentences To control this synthetic hallucination process , we build on recent work on large scale denoising autoencoders . These models learn to map a corrupted sentence back to the original text it was derived from , including for example learning to reconstruct missing words that have been arbitrarily masked out . We use BART ( Lewis et al. , 2020 ) by first inserting noises and then reconstructing parts of the target sentence , without providing it any access to the source sentence , thereby encouraging it to insert new content as needed to ensure fluency . As shown in Fig . 2 , we first apply noising functions to the target sentence T in the bi-text and then use a pretrained BART to generate T ′ conditioned on the noised T with beam search . Label assignments After obtaining the hallucinated sentence T ′ with BART , we need to assign appropriate labels to each token in T ′ to mark which words are hallucinated . We compute the edit distance between T ′ and T , and back-trace the deletion and substitution operations with dynamic programming . All the po- sitions in T ′ involving these two operations are labeled as hallucinations and everything else is considered faithful to T . Fig . 3 shows an example of label assignment with edit distance , where words in red are replaced and the word in blue is deleted to convert T ′ to T . Assigning labels with edit-distance can not always guarantee the correct labels , however , we find that this simple approach provides sufficiently high quality training data for effective hallucination detection in practice . 3.2 FINETUNING PRETRAINED LM ON SYNTHETIC DATA . Hallucination prediction loss We follow the common practice in natural language understanding ( NLU ) tasks and finetune a pretrained language model ( LM ) on our synthetic data . We finetune a cross-lingual LM ( Conneau et al. , 2020 ) for machine translation and a monolingual model ( Liu et al. , 2019 ) for summarization . In both cases , we concatenate the input source , true target and hallucinated target denoted ( S , T , T ′ ) as a single input sequence to the pretrained LM . Then we minimize the standard classification loss Lpred over the pseudo hallucination labels LT ′ on top of the final hidden vectors of each token in T ′ as shown in Fig . 4 . Although using only the source text and hallucinated target ( S , T ′ ) as the input should be sufficient to learn the hallucination prediction task , we can also easily measure the extent to which including the true target T in the input could help the model . At test time , when evaluating the faithfulness of the machine outputs G , we do not use the true target T and perhaps surprisingly find our model can generalize well without references , even when it was present during training . To prevent the model from overly relying on the true target T and learning spurious correlations ( e.g . the edit distance ) , we explored two techniques : ( 1 ) dropout – randomly drop out tokens in T to force the dependence on the source input ; ( 2 ) paraphrase – recall that at synthetic data generation time , we generate T ′ from BART conditioned on the noised T . Instead , we can apply noise functions to the paraphrased sentence of T . Let D denote the paraphrased sentence of T and D′ denote the generation from BART conditioned on the noised D. Then we create pseudo labels of D′ denoted LD′ by computing the edit-distance between the D′ and D and use ( ( S , T , D′ ) , LD′ ) as the training data for finetuning . Since the pseudo labels are created based on D , it can prevent the model from learning the edit-distance between T and D′ easily . Ablation studies will be provided in § 5.4 . Masked LM loss We also add the masked language model loss ( MLM ) Lmlm following ( Devlin et al. , 2019 ) . To learn this loss , we create a different batch from the above by concatenating only the source S and target T as the input , since the hallucinated target T ′ could provide erroneous information for predicting masked words in T . We find that such multi-task learning objective helps learn better representations of the inputs and further improves performance on predicting hallucination labels . The final loss is L = Lpred + α · Lmlm where α is a hyperparameter .

Summary: This paper proposes hallucination detection at the token level, which predicts if each token in the generation output is hallucinated or faithful to the source input. In contrast, previous studies usually work on the sentence level. To create synthetic training data, a denoising pre-trained LM is first used to generate (potentially) unfaithful counterparts T’ of the references T. Then, token-level labels are obtained by comparing T and T’ via edit distance. Finally, a standard classification model is trained on the token-level labels by concatenating the source S, true and unfaithful targets T (T’).