Generating Multi-Sentence Abstractive Summaries of Interleaved Texts

INTRODUCTION . Interleaved texts are becoming more common with new ways of working and new forms of communication , e.g. , multi-author entries for activity reports , meeting minutes and social media conversations , such as Slack . Quickly getting a sense of or following the content of different threads in interleaved texts , where posts belonging to different threads occur in one sequence , is often difficult . An example of two threads with multiple posts interleaved to form a sequence is : Post1-Thread1 → Post1-Thread2 → Post2-Thread1 → Post3-Thread1 → Post2Thread2→ Post3-Thread2 This intermingling leads to difficulties in not only following discussions but also in retrieving their essence . In conversation disentanglement , interleaved posts are grouped by the thread ; e.g. , the previous example could be rearranged as : Post1-Thread1→ Post2-Thread1→ Post3-Thread1 Post1-Thread2→ Post2-Thread2→ Post3-Thread2 In analyzing interleaved texts , Shang et al . ( 2018 ) went a step further and proposed summarization of the interleaved texts . They designed an unsupervised two-step system and evaluated the system on meeting texts . In the first step , a conversation disentanglement component disentangles the texts thread-wise . Then , in the second step , a multi-sentence compression component compresses the thread-wise posts to single sentence summaries . However , this system has a major disadvantage , in that the disentanglement obtained through either supervised Jiang et al . ( 2018 ) or unsupervised Wang & Oard ( 2009 ) methods propagate its errors to the downstream summarization task , and thus , degrades the overall performance . We aim to tackle this issue of error propagation through an end-to-end trainable encoder-decoder system that takes a variable length input , e.g. , interleaved texts , processes it and generates a variable length output , e.g. , a multi-sentence summary ; see Figure 1 for an illustration . An end-to-end system eliminates the disentanglement component , and thus , the error propagation . The encoder first performs word-to-word encoding to embed each post , followed by post-to-post encoding to embed the overall content of the posts and represent the discourse structure of the interleaved texts . The decoder has a thread-to-thread decoder to generate a representation for each thread , and the thread representation is given to a word-to-word decoder to generate a summary sentence . We also propose to use hierarchical attention similar to Nallapati et al . ( 2016 ) , but instead of computing post-level attention at every word , attentions are only computed when decoding new sentences . Further , the attention networks are trained end-to-end . Despite the availability of a multitude of real-world interleaved texts , a major challenge to train encoder-decoder models is the lack of labels ( summaries ) . As labeling is difficult and expensive Barker et al . ( 2016 ) ; Aker et al . ( 2016 ) ; Verberne et al . ( 2018 ) , we synthesize two separate corpora . Each is derived by mixing texts and associated summaries from a corpus of documents , where the mixed text has a structure reflective of a multi-party conversation with inter-leaved threads and the summary highlights the information in the threads . We find abstracts and titles of randomized controlled trial ( RCT ) articles , a PubMed corpus , and also for Stack Exchange posts and titles . To summarize , our contributions are threefold : • We propose to combine a hierarchical encoder and decoder to obtain multi-sentence summaries of interleaved texts . • We propose a novel hierarchical attention mechanism that is integrated with the hierarchical encoder-decoder architecture and which equips the decoder to disentangle the interleaving further . • We use two synthetic datasets to verify the ideas and show our end-to-end trainable architecture addresses not only the issue of error-propagation observed in competitive methods but also improves the performance . RELATED WORK . Quite often multi-party conversations , e.g . news comments , social media conversation and activity report , have tens of posts discussing several different matters pertaining to a subject . Ma et al . ( 2012 ) ; Aker et al . ( 2016 ) ; Shang et al . ( 2018 ) designed methodologies to summarize posts in order to provide an overview on the discussed matters . They broadly follow the same approach : cluster the posts and then extract a summary from each cluster . There are two kinds of summarization : abstractive and extractive . In abstractive summarization , the model utilizes a corpus level vocabulary and generates novel sentences as the summary , while extractive models extract or rearrange the source words as the summary . Abstractive models based on neural sequence-to-sequence ( seq2seq ) ( Rush et al . ( 2015 ) ) proved to generate summaries with higher ROUGE scores than the feature-based abstractive models . Integration of attention into seq2seq ( Bahdanau et al . ( 2014 ) ) led to further advancement of abstractive summarization ( Nallapati et al . ( 2016 ) ; Chopra et al . ( 2016 ) ) . There are many possible patterns of organization of the information in texts , e.g. , chronological pattern . News articles have an inverted pyramid pattern , i.e. , core information in the lead sentences and the extra information in later sentences . A seq2seq model is appropriate for summarization of a news article as it encodes and decodes sequentially . However , in interleaved texts , related information maybe separated ; thus a seq2seq model may be competent but not sufficient . Li et al . ( 2015 ) proposed an encoder-decoder ( auto-encoder ) model that utilizes a hierarchy of networks : word-to-word followed by sentence-to-sentence . Their model is better at capturing the underlying structure than a vanilla sequential encoder-decoder model ( seq2seq ) . Krause et al . ( 2017 ) ; Jing et al . ( 2018 ) showed multi-sentence captioning of an image through hierarchical Recurrent Neural Network ( RNN ) , topic-to-topic followed by word-to-word , is better than seq2seq . These works suggest a hierarchical encoder , with word-to-word encoding followed by post-to-post , will better recognize the dispersed information in interleaved texts . Similarly , a hierarchical decoder , thread-to-thread followed by word-to-word , will intrinsically disentangle the posts , and therefore , generate more appropriate summaries . Nallapati et al . ( 2016 ) devised a hierarchical attention mechanism for a seq2seq model , where two levels of attention distributions over the source , i.e. , sentence and word , are computed at every step of the word decoding . Based on the sentence attentions , the word attentions are rescaled . Hsu et al . ( 2018 ) slightly simplified this mechanism and computed the sentence attention only at the first step . Our hierarchical attention is more intuitive and computes new sentence attentions for every new summary sentence , and unlike Hsu et al . ( 2018 ) , is trained end-to-end . MODEL PROBLEM STATEMENT . We aim to design a system that when given a sequence of posts , C = 〈P1 , . . . , P|C |〉 , produces a sequence of summaries , T = 〈S1 , . . . , S|T |〉 . For simplicity and clarity , unless otherwise noted , we will use lowercase italics for variables , uppercase italics for sequences , lowercase bold for vectors and uppercase bold for matrices . Figure 2 illustrates the proposed hierarchical encoder-decoder framework . In the framework , first , a low-level , word-to-word encoder converts a sequence of words in a post , Pj , to a sequence of representations , Hj = 〈hw2wj0 , . . . , hw2wj|Pj |〉 . Subsequently , a top-level , post-to-post encoder converts those representations , 〈H0 , . . . , H|C |〉 , to a sequence of top-level post representations 〈hp2p1 , . . . , h p2p |C |〉 . These encoded representations are then passed to the decoder , which utilizes a top-level , threadto-thread , decoder to disentangle them into a sequence of thread representations 〈ht2t1 , . . . , ht2t|T |〉 . Finally , a low-level , word-to-word , decoder takes a feed-forward mapped thread representation ht2ti and generates a sequence of summary words 〈wi1 , . . . , wi|Si|〉 . The maximum number of posts in a sequence of interleaved texts is denoted by n and threads by m. We limit the number of words in posts and summaries to fixed lengths by either truncating or padding , and denote them by p and q respectively . The hidden states of the encoder and decoder have dimensionality l . ENCODER . The hierarchical encoder used in Figure 2 is based on Nallapati et al . ( 2017 ) , where word-toword and post-to-post encoders are bi-directional LSTMs . The word-to-word Bi-LSTM encoder ( Ew2w ) runs over word embeddings of any post , Pj , and generates a set of hidden representations ( 〈hEw2wj0 , . . . , h Ew2w jp 〉 ) ; see left middle section in Figure 2 . The average pooled value of the word- to-word representations of post Pj ( 1p ∑p k=0 h Ew2w jk ) are input to the post-to-post Bi-LSTM encoder ( Et2t ) , which then generates a set of representations ( 〈h Ep2p 0 , . . . , h Ep2p n 〉 ) corresponding to those posts . Overall , output word-to-word representations , W , has dimension n× p× 2l and post-to-post representations , P , has dimension n× 2l for a given channel , C . DECODER . Our hierarchical decoder ( see Figure 2 ) is based on Li et al . ( 2015 ) , where both thread-to-thread and word-to-word decoders are uni-directional LSTMs . The initial state hDt2t0 of the thread-to-thread LSTM decoder ( f Dt2t ) is set with a feedforwardmapped representation of an average pooled post representations ( c′ = 1n ∑n k=0 h p2p k ) . At each step i of the decoder , a sequence of attention weights , 〈β̂i,0 , . . . , β̂i , n×p〉 , corresponding to the set of encoded word representations , 〈hw2w0 , . . . , hw2wn×p〉 are computed utilizing the previous state , h Dt2t i−1 . We will elaborate the attention computation in the next section . A weighted representation of the words ( crossed blue circle ) is then computed : ∑n j=1 β̂ijWij , and used as one input to LSTM fDt2t . Additionally , we use the last hidden state hDw2wi−1· of the wordto-word decoder LSTM ( Dw2w ) of the previously generated summary sentence as the second input to compute the next state of thread-to-thread decoder , i.e. , hDt2ti . The motivation is to provide information about the previous sentence . The current state hDt2ti is passed through a single layer feedforward network and a distribution over STOP=1 and CONTINUE=0 is computed : pSTOPi = σ ( g ( hDt2ti ) ) ( 1 ) where g is a feedforward network . In Figure 3 , the process is depicted by a red circle . The threadto-thread decoder keeps decoding until pSTOPi is greater than 0.5 . Additionally , the current state hDt2ti is passed through another single layer feedforward network k followed by tanh activation to compute the thread representation si = tanh ( k ( hDt2ti ) ) . Given a thread representation si , the word-to-word decoder generates a summary for the thread . Our word-to-word decoder is based on Bahdanau et al . ( 2014 ) . It is a unidirectional attentional LSTM ( fDw2w ) ; see the right-hand side of Figure 2 . We refer to Bahdanau et al . ( 2014 ) for further details . HIERARCHICAL ATTENTION . Our novel dynamic hierarchical attention is based on two popular concepts : attention scores should integrate in top-down manner ( Nallapati et al . ( 2016 ) ) and two distributions over same words must be modeled together ( See et al . ( 2017 ) ) , not just in isolation . Our hierarchical attention works at 3 levels , the post level ( corresponding to posts ) , i.e. , γ , and higher level ( corresponding to source tokens ) , i.e. , β , are computed while obtaining a thread representation , s. The lower level attention ( also corresponding to source tokens ) , i.e. , α , is computed while generating a word , y , of a summary , S . ; see yellow rectangles in Figure 4 . At step i of thread decoding , we also compute elements of post-level attention , i.e. , γi· as below . γij = σ ( attnβ ( hDt2ti−1 , Pj ) j ∈ { 1 , . . . , n } ( 2 ) At the same step , we also compute elements of high-level attention , i.e , βi· as below . βik = σ ( attnβ ( hDt2ti−1 , ak ) ) k ∈ { 1 , . . . , n× p } ( 3 ) ak = add ( Wjl , Pj ) where k : = j ∗ l and j ∈ { 1 , . . . , n } , l ∈ { 1 , . . . , p } ( 4 ) where ak is computed as in Eq . 4 . W in Eq . 4 is a word-to-word encoder representations of dimension n × p × 2l , add aligns a post representation to its constituting word representations and does element-wise addition , and attnβ is a feedforward network that aligns the current thread decoder state hDt2ti−1 with all n ∗ p number of ak vectors . Then , we use γ to rescale high-level attention , β as below . β̂ik = βik ∗ γij where k : = j ∗ l and l ∈ { 1 , . . . , p } j ∈ { 1 , . . . , n } ( 5 ) At step t of word-to-word decoding of summary thread i , we compute elements of low level attention , i.e. , αit· as below . αitk = exp ( eitk ) ∑n·p k=1 exp ( eitk ) where eitk = attnα ( hDw2wt−1 , ak ) ( 6 ) , and ak is same as in Eq . 4 and attnα is a feedforward network that aligns the current word decoder state hDw2wt−1 with all n ∗ p number of ak vectors . Finally , we use rescaled high-level word attentions , β̂ , for rescaling low level attention , α as below : α̂itk = β̂k × exp ( eitk ) ∑n·p k=1 β̂k × exp ( eitk ) ( 7 ) Table 1 compares our hierarchical attention with previous work on hierarchical attention in the text summarization domain .

Summary: This paper proposes a technique for generating summaries of interleaved texts. Unlike previous work that first perform unsupervised clustering to extract ordered threads, the authors instead propose a hierarchical model that directly process interleaved threads. In particular, the authors propose using a hierarchical encoder that encodes words -> post -> threads as well as a hierarchical decoder that decodes threads -> words. On synthetic interleaved datasets composed from PubMed and StackExchange, the proposed method outperform seq2seq with attention as well as a pipeline system that applies unsupervised clustering followed by seq2seq. The authors present results for well-chosen ablations as well as baselines.