Do Boat and Ocean Suggest Beach? Dialogue Summarization with External Knowledge

1 . Introduction . Automatically summarizing conversations in our daily lives can benefit users for better organizing and retrieving their historical information . There have been several approaches to conversation summarization , including extractive approaches [ Xie et al. , 2008 , Riedhammer et al. , 2010 ] and abstractive approaches [ Oya et al. , 2014 , Shang et al. , 2018 ] . While extractive approaches focus on using the seen words in a conversation to summarize it , abstractive approaches usually use a text generation model to perform summarization . Different from news articles , in daily dialogues , it is common that a speaker ’ s utterance can suggest something that is not literally expressed but can be interpreted by a cooperative listener . For example , in Figure 1 , if a dialogue mentions boat and ocean , the first impression of an ordinary person would be a boat sailing on the ocean . If , a modifier abandoned is added to boat , combined with ocean , the previous scene will be canceled and we will think of a beach or a shore because that is where the abandoned boat and ocean tend to cooccur . This kind of out-of-context inference is a language phenomenon in the field of pragmatics ground truth summarizations . [ Grice , 1975 ] . To comprehensively understand a dialogue where some context is omitted by the speakers , both a thorough understanding of the context and necessary inference are required . However , to the best of our knowledge , neither evaluation metrics nor suitable methods are available for such kind of out-of-context inference currently , even if it is an easy task for human beings . To study the phenomenon of out-of-context inference in dialogues , we narrow down the definition of out-of-context inference to a certain type so that we can automatically evaluate it . Here , we only focus on summarization ’ s new concepts that are suggested by existing concepts in the dialogue context . This is also related to lexical entrainment [ Brennan , 1996 ] which studies the lexical variability in language use and lexical pragmatics in relevance theory [ Sperber and Wilson , 1986 ] , which studies how to identify and infer concepts from words via broadening and narrowing contexts . To formally define the inference of new concepts in summarization , we distinguish a new concept from existing concepts following three rules according to WordNet [ Miller , 1995 ] : ( 1 ) It should not be a general concept . ( 2 ) It should not be a synonym of an existing concept . ( 3 ) It should not be a hypernym ( or super-concepts ) of an existing concept . In this way , we can distinguish out-of-context inference from logical entailment or synonym as much as we can . Specifically , we name them as the Concept Out of the Dialogue Context ( CODC . ) Current evaluation metrics are usually based on lexical similarities or overlaps , e.g. , BLEU [ Papineni et al. , 2002 ] , ROUGE-L [ Lin , 2004 ] , METEOR [ Lavie and Agarwal , 2007 ] , CIDEr [ Vedantam et al. , 2015 ] , etc . Such lexical metrics usually do not care about the recall of the novel words , and are not precise at evaluating the precision of the out-of-context inference . To tackle these limitations , we propose and study a new evaluation metric that incorporates CODC for text generation . Moreover , to improve the summarization results with the help of CODC inference , we propose an abstractive summarization framework that includes an out-of-context inference module to enhance the model ’ s inference ability . The two-step framework we proposed basically contains an inference module and an extendable knowledge attention model . In the inference part , we use word-relatedness features such as co-occurrence , embedding similarities , and WordNet relatedness features to distinguish whether a candidate word is a plausible out-of-context inference or not . While in the knowledge attention module , we aggregate the retrieved knowledge from the inference module and apply the attention mechanism to extract useful information in the decoding steps . Our contributions are summarized as follows : 1 . We address the problem of out-of-context inference in dialogue summarization for the first time , and provide an elaborative definition of the problem . 2 . We design a related metric based on Concepts Out-of Dialogue Context ( CODC ) to evaluate neural models ’ ability to infer plausible novel concepts . 3 . We proposed Trans-KnowAttn , an abstractive dialogue summarization approach that incorporates an out-of-context inference ( missing-link inference ) module and a knowledge attention module to improve the inference capability in dialogue summarization . 2 . Dialogue Summarization Task . We develop our dialogue summarization ( DialSum ) task based on the VisDial [ Das et al. , 2017 ] dataset . To construct their dataset , VisDial asks two workers on Amazon Mechanical Turk to chat with each other in real-time to discuss an image in the MSCOCO dataset [ Lin et al. , 2014 ] . The MSCOCO dataset contains human-annotated captions of about 120K images . Each image has five captions from five different annotators . A “ questioner ” sees the caption of the image and another person sees both the caption and the image . The questioner is asked to ask questions to “ imagine the scene better ” , and the annotators usually describe the most prominent concepts in an image . Thus , for each image , we have both a dialogue from the VisDial dataset and five captions from the MSCOCO dataset . We align a dialogue with five captions as five alternative summarizations of the dialogue . By nature , there are many out-of-context inference phenomena behind those utterances , as the speakers already have the image in mind as their context so that they don ’ t need to bring it up again in the dialogue . Assuming a gold summarization y is provided by an annotator , we evaluate models ’ inference ability by how many novel concepts ( i.e. , noun phrases that do not appear in the dialogue ) in y can be mentioned by ŷ , the generated summary , without introducing extra noisy concepts . The number of examples in the training , developing , and testing set are 98,256 , 12,282 , and 12,083 , respectively . 3 . CODC-based Evaluation Metric . 3.1 Definition of CODC . In this section , we introduce the new metric based on CODC for evaluating models ’ conceptual inference ability . Here , by concept , we mean a noun word . As most examples in our dataset are general objects rather than specific named entities , we use WordNet [ Miller , 1995 ] to check the relations among all the concepts . For each example in DialSum , which contains a dialog x and a description y , we denote the concept sets extracted from x and y as Cx and Cy respectively . Then we define the set of concepts out of dialogue context as Cy−x = { cy ∈ Cy|∀cx ∈ Cx , I1 ( cx , cy ) = 1 ) } , ( 1 ) where I1 ( cx , cy ) is the function implementing the following three rules : • The least depth of all synsets containing cy is no less than δd , where we set δd = 4 empirically ( 1,790 synsets in total are filtered out . ) This rule is to filter out very general concepts that may be inferred by anything , for example , “ entity ” . • The shortest path between any pair of synsets containing cx and cy is greater than 1 . This rule is to filter out concepts that are synonyms of existing concepts in the dialogue . • Any synset of cy is not a hypernym ( super-concept ) of any synset of cx . This rule is to avoid entailment for the new concepts as essentially abstractive summarization is able to perform conceptual abstraction . To better understand the dataset , we show the distribution of the counts of CODC per dialogue in Figure 2 ( a ) , for one set of the descriptions in the training set . We can find that for more than 80 % of the training dialogues , one or more CODC should be inferred . In Figure 2 ( b ) , we show a real example from the dataset , where we can infer the word sky from the dialogue even if the word is not literally expressed . These observations show that the out-of-context concepts are common in this dialogue summarization task and further prove the importance of understanding how well models can generate summarization with out-of-context concepts . 3.2 CODC Precision , Recall , and F1 . As a model with strong inference ability should be able to infer correct new concepts without introducing wrong ones , inspired by the F1 evaluation metric , we design F1 over CODC to evaluate models ’ inference ability . Let X = { x ( 1 ) , x ( 2 ) , ... , x ( N ) } be all dialogues in the evaluation set , where N is the size of the dataset , and Y = { Y ( 1 ) , Y ( 2 ) , ... , Y ( N ) } be the set of ground-truth descriptions . Note that , for each image , we may have K ground-truth descriptions such that Y ( i ) = { y ( i ) 1 , ... , y ( i ) K } . Denote the generated summary of the i-th dialogue as ŷ ( i ) . We then define the precision , recall , and F1 over CODC as follows : PCODC = ∑N i=1 maxk |h ( x ( i ) , y ( i ) k , ŷ ( i ) ) |∑N i=1 |Cŷ ( i ) −x ( i ) | , RCODC = ∑N i=1 |h ( x ( i ) , y ( i ) k∗i , ŷ ( i ) ) |∑N i=1 |C y ( i ) k∗ i −x ( i ) | , F1CODC = 2PCODCRCODC PCODC +RCODC , ( 2 ) where h ( x , y , ŷ ) and k∗i are defined as follows : h ( x , y , ŷ ) = { cŷ ∈ Cŷ−x|∃cy ∈ Cy−x , I2 ( cy , cŷ ) = 1 } , ( 3 ) k∗i = argmaxk∈ { 1 , ... , K } |h ( x ( i ) , y ( i ) k , ŷ ( i ) ) | |Cy ( i ) k −x ( i ) | , ( 4 ) and I2 ( cy , cŷ ) is the function to determine whether cy and cŷ are identical or entailed following either of two rules : • There exists one pair of synsets of cy and cŷ being identical . • There exists a synset of cŷ being a hypernym of a synset of cy . Note that a summary can be considered as a plausible generation if it covers a reasonable amount of CODCs for one of the K descriptions . Based on this idea , for a certain dialogue , we only focus on the description where the generated sentence performs the best in precision or recall regarding the inference of CODC . In this case , max operation is selected in the calculation of precision and recall , which is different from the average operation that is typically used by previous automatic evaluation metrics in the setting of multi-reference . Also , by using the max operations , we argue that the theoretical upper bound for CODC precision , recall , and F1 are 1.0 , as any ground-truth description will be scored 1.0 under the max operation .

Summary: This paper introduces a new dialogue summarization task that requires summaries to mention new concepts that are not explicitly mentioned in -- but may be implied by -- the dialogue. The task, CODC, repurposes the VisDial dataset (itself based on the MSCOCO captioning dataset), which involved information-seeking question-and-answer dialogues about an image, where the dialogue participants were prompted by the image's caption. The captions frequently mention some concepts that are not mentioned in, but are implied by, the dialogue. In CoDC, the task is to produce the reference caption (which is treated as a dialogue summary) from the dialogue (without using the images). Summarization models are evaluated on standard text overlap with the references, as well as an WordNet-based F1 metric that measures whether the model's output has inferred concepts in the references which were not mentioned in the dialogue.