Detecting Hallucinated Content in Conditional Neural Sequence Generation

1 INTRODUCTION . Neural sequence models have achieved impressive breakthroughs in a wide range of applications , including data-to-text generation ( Puduppully et al. , 2019 ) , machine translation ( Vaswani et al. , 2017 ; Wu et al. , 2016 ) and text summarization ( Rothe et al. , 2020 ) . Although these models can generate fluent sentences that are even sometimes preferred to human-written content ( Läubli et al. , 2018 ; Brown et al. , 2020 ) , recent work has also shown that they lack global logical consistency ( Marcus & Davis , 2020 ) , sometimes degenerate to dull and repetitive outputs ( Welleck et al. , 2019 ) and can often hallucinate content that is not entailed by the input ( Maynez et al. , 2020 ) . In this paper , we focus on the faithfulness of machine outputs in conditional sequence generation tasks , aiming to automatically identify and quantify content in the output that is not faithful to the input text . This risk of generating unfaithful content impedes the safe deployment of neural sequence generation models . The first step to building models that do not suffer from these failures is the assessment and identification of such hallucinated outputs . Prior work has shown that standard metrics used for sequence evaluation , such as BLEU scores ( Papineni et al. , 2002 ; Post , 2018 ) , ROUGE ( Lin & Hovy , 2004 ) and BERTScores ( Zhang et al. , 2019 ) , do not correlate well with the faithfulness of model outputs ( Maynez et al. , 2020 ; Wang & Sennrich , 2020 ; Tian et al. , 2019 ) , and they also require reference text , limiting their applicability to detecting halluciations in a deployed system at run-time . Very recent efforts ( Maynez et al. , 2020 ; Durmus et al. , 2020 ; Wang et al. , 2020a ) have started to develop automatic metrics to measure the faithfulness of output sequences . These methods use external semantic models , e.g . the question-generation and question-answering systems ( Wang et al. , 2020a ; Durmus et al. , 2020 ) or textual entailment inference models , to score faithfulness tailored for abstract text summarization . However , these scores do not directly measure the number of hallucinated tokens and only correlate weakly with human judgements due to compounded errors . We propose a new task for faithfulness assessment - hallucination detection at the token level , which aims to predict if each token in the machine output is a hallucinated or faithful to the source input . This task does not use the reference output to assess faithfulness , which offers us the ability to apply it in the online generation scenario where references are not available . Similar to the spirit of our proposed task , word-level quality estimation ( Fonseca et al. , 2019 ) in the machine translation community predicts if tokens are correctly translated based on human post-editing . However , they do not distinguish errors in terms of fluency and adequacy ( Specia et al. , 2011 ) . In contrast to estimating the amount of human post-editing work required to fix errors , we specifically focus only on hallucination ( not fluency ) errors . We measure hallucination for two conditional sequence generation tasks – abstractive summarization and machine translation ( MT ) . For the former , we produce a benchmark dataset from the recent released annotations in ( Maynez et al. , 2020 ) . For MT , we carefully design the human assessment guideline and create high-quality annotations . We will also release our human annotated data for future research . To learn token-level hallucination prediction for general conditional sequence generations tasks , we propose a novel method that creates synthetic “ hallucinated ” dataset with pseudo labels and finetunes a pretrained language model ( Liu et al. , 2019 ; Conneau et al. , 2020 ) on it . Without any human annotated supervised training data , we achieve an average F1 of around 0.6 across all the benchmark datasets , setting initial performance levels for this new task . We also show that pretraining on MT can actually produce more faithful translations , confirming recent findings in abstractive summarization ( Maynez et al. , 2020 ) . Predicting hallucination labels at token-level provides a tool for diagnosing and interpreting model outputs , which allows us to flag potential risks at inference time for previously unseen inputs . On the other hand , the token-level labels also offer possibility of fine-grained controls over the target sequence to improve the generation . We show how to use these token-level hallucination labels to improve self-training in low-resource MT , where the teacher can produce hallucinated outputs that are harmful to the student model . However , many outputs are only partially hallucinated ( see examples in Appendix D.6 ) and the rest of the output is still useful for training , as we show by introducing different token-level loss truncation schemes . Our best method outperforms strong baselines by a large margin both in translation quality and hallucination reduction . 2 TASK : HALLUCINATION PREDICTION AT TOKEN-LEVEL . For a source sequence S and its model generation G from a neural conditional generation model , following Maynez et al . ( 2020 ) we define any span wi , · · · , wi+j ( j > = 0 ) in G as being hallucinated if it can not be entailed by the source input S. More specifically , we consider two not mutually exclusive types of hallucination : Content Insertion : a span wi , · · · , wi+j in G consists of additional content that is not supported by S , i.e . its paraphrase or other equivalent form can not be inferred from S. In Fig . 1 , the word “ happily ” in the machine translation belongs to this case . This is also referred as “ extrinsic hallucinations ” in Maynez et al . ( 2020 ) . Incorrect Substitution : a span of word ( s ) is misrepresented information based on the S. In Fig . 1 , “ Jerry ” in the machine translation is a hallucinated word and should be replaced by “ Mike ” . This type of hallucination is similar to the concept of “ intrinsic hallucination ” in Maynez et al . ( 2020 ) . Note that there are cases where certain words ( e.g . “ This is not a great book. ” becomes “ This is a great book. ” ) are dropped inG and hence the meaning of S is changed , and we consider any spans in G that misrepresent S as hallucinated contents ( e.g . the entire sentence of “ This is a great book. ” ) . We aim to identify all the span ( s ) satisfying the above conditions in the machine generation G.1 Note that the above definition is only used in the guidelines for human annotators , who do not need to distinguish between these types rigorously . 1We do not consider the under-generations e.g . the source input is only partially translated or summarized . 3 LEARNING TOKEN-LEVEL HALLUCINATION DETECTION . We propose a general-purpose method for token-level hallucination detection for conditional sequence generation tasks . Given the source input S , we first formulate the task of token-level hallucination detection as a sequence labeling problem where a binary label is predicted at each position Gt of the machine generation G. One straightforward way of learning such task is to train a model with supervised data in the form of ( ( S , G ) , LG ) where LG are the labels at every positions of G that indicate if each word is a hallucinated one or not . However , because such labeled training data is not readily available , we also propose a approach to automatically create synthetic training data . 3.1 SYNTHETIC DATA CREATION . We use bi-text from the training data to create synthetic examples by automatically inserting new , hallucinated tokens . More speficially , we take target sequence T and create a hallucinated version of it denoted T ′ with associated hallucination labels for each token in T ′ . Then we can train a supervised model on this synthetic labeled data set of ( ( S , T ′ ) , LT ′ ) . The key challenge is that T ′ should be a fluent sentence that does not differ too much from T . Generation of hallucinated sentences To control this synthetic hallucination process , we build on recent work on large scale denoising autoencoders . These models learn to map a corrupted sentence back to the original text it was derived from , including for example learning to reconstruct missing words that have been arbitrarily masked out . We use BART ( Lewis et al. , 2020 ) by first inserting noises and then reconstructing parts of the target sentence , without providing it any access to the source sentence , thereby encouraging it to insert new content as needed to ensure fluency . As shown in Fig . 2 , we first apply noising functions to the target sentence T in the bi-text and then use a pretrained BART to generate T ′ conditioned on the noised T with beam search . Label assignments After obtaining the hallucinated sentence T ′ with BART , we need to assign appropriate labels to each token in T ′ to mark which words are hallucinated . We compute the edit distance between T ′ and T , and back-trace the deletion and substitution operations with dynamic programming . All the po- sitions in T ′ involving these two operations are labeled as hallucinations and everything else is considered faithful to T . Fig . 3 shows an example of label assignment with edit distance , where words in red are replaced and the word in blue is deleted to convert T ′ to T . Assigning labels with edit-distance can not always guarantee the correct labels , however , we find that this simple approach provides sufficiently high quality training data for effective hallucination detection in practice . 3.2 FINETUNING PRETRAINED LM ON SYNTHETIC DATA . Hallucination prediction loss We follow the common practice in natural language understanding ( NLU ) tasks and finetune a pretrained language model ( LM ) on our synthetic data . We finetune a cross-lingual LM ( Conneau et al. , 2020 ) for machine translation and a monolingual model ( Liu et al. , 2019 ) for summarization . In both cases , we concatenate the input source , true target and hallucinated target denoted ( S , T , T ′ ) as a single input sequence to the pretrained LM . Then we minimize the standard classification loss Lpred over the pseudo hallucination labels LT ′ on top of the final hidden vectors of each token in T ′ as shown in Fig . 4 . Although using only the source text and hallucinated target ( S , T ′ ) as the input should be sufficient to learn the hallucination prediction task , we can also easily measure the extent to which including the true target T in the input could help the model . At test time , when evaluating the faithfulness of the machine outputs G , we do not use the true target T and perhaps surprisingly find our model can generalize well without references , even when it was present during training . To prevent the model from overly relying on the true target T and learning spurious correlations ( e.g . the edit distance ) , we explored two techniques : ( 1 ) dropout – randomly drop out tokens in T to force the dependence on the source input ; ( 2 ) paraphrase – recall that at synthetic data generation time , we generate T ′ from BART conditioned on the noised T . Instead , we can apply noise functions to the paraphrased sentence of T . Let D denote the paraphrased sentence of T and D′ denote the generation from BART conditioned on the noised D. Then we create pseudo labels of D′ denoted LD′ by computing the edit-distance between the D′ and D and use ( ( S , T , D′ ) , LD′ ) as the training data for finetuning . Since the pseudo labels are created based on D , it can prevent the model from learning the edit-distance between T and D′ easily . Ablation studies will be provided in § 5.4 . Masked LM loss We also add the masked language model loss ( MLM ) Lmlm following ( Devlin et al. , 2019 ) . To learn this loss , we create a different batch from the above by concatenating only the source S and target T as the input , since the hallucinated target T ′ could provide erroneous information for predicting masked words in T . We find that such multi-task learning objective helps learn better representations of the inputs and further improves performance on predicting hallucination labels . The final loss is L = Lpred + α · Lmlm where α is a hyperparameter .

Summary: This paper presents a method to detect hallucinated tokens in generations from neural machine translation and summarization. Given a source input S and its output G generated by a sequence generation model, this paper formalizes the task of detecting hallucinated tokens as a labeling problem on the output G. In order to train the labeler, the method synthetically generates supervision data by using a BART model. The BART model receives a text with noises ([MASK] tokens) and tries to predict [MASK] tokens. In this way, the method obtains a pseudo hallucinated text T' from a text T, and assigns hallucination labels by estimating edit operations between T and T'. The labeler is trained by fine-tuning pre-trained cross-lingual (for MT) and mono-lingual (for summarization) language models. In training, the labeler receives a source text S, true target text T, and pseudo hallucinated text T' separated by [SEP] tokens and tries to reproduce the hallucination labels on T'. Receiving a source text S and its output G, the labeler predicts hallucination labels on G during the inference time.