NUTS: Network for Unsupervised Telegraphic Summarization

1 INTRODUCTION . Humans tend to relay information to others in a concise manner . Typically , we do not pass on information that we received in exactly the same format . More often than not , we tend to abridge and simplify it for the receiver . This captures the essence of summarization . Formally speaking , summarization refers to capturing all the information in a source , without compromising on understanding . There is a pressing need for summarization techniques to deal with the vast amount of textual data available nowadays . In certain genres , like fictional narratives , each sentence might have an important role to play . This makes the domain of fictional narratives unsuitable for standard extractive summarization methods . Typically , extractive summarization works well if the source text revolves around a central theme with the same information reiterated across multiple sentences . For such text , picking a small number of relevant sentences is enough to summarize the text . Therefore , this method of extractive summarization is widely used for newswire articles ( Lee et al. , 2005 ) , encyclopedic and scientific texts ( Teufel & Moens , 2002 ) etc . However , fictional stories and plays , for instance , do not always focus on a single theme . They describe a sequence of events and often contain dialogue . Information is not repeated and each sentence may contribute to developing the plot further . This problem is tackled in an alternate variation of extractive summarization known as telegraphic summarization . In this summarization technique , each sentence is considered an independent text source with each word acting as a segment . The telegraphic summaries for an input , read like a telegram , hence the name . For example , for an input sentence : “ An earthquake in Tokyo left 12 people dead ” the telegraphic summarization would be : “ earthquake Tokyo 12 dead ” . Recently , the advantages of telegraphic summarization over typical extractive techniques have been explored in Malireddy et al . ( 2018 ) . The independent telegraphic summaries computed for each sentence in a document can be combined to form the summary of the whole document . Current algorithms for telegraphic summarization rely on handcrafted rules ( Grefenstette , 1998 ) or statistics based on syntax to infer important words in a sentence ( Jing , 2000 ; Riezler et al. , 2003 ; Knight & Marcu , 2000 ) . These methods , however , do not generalize for all cases . Additionally , the work by Lin ( 2003 ) suggests that pure syntax based methods can not be used for general purpose summarization . We overcome these limitations by exploring learning based approaches for telegraphic summarization . We propose an unsupervised deep network which implicitly learns to retain significant words in a sentence . We try to reconstruct the source text ( which is the input to our network ) , from an abridged version of it generated at an intermediate stage . This abridged version can be used as the telegraphic summary of the source . Since the proposed architecture follows an unsupervised approach , we do not need any labeled text-summary pairs for our method . Another advantage of our algorithm is that it is agnostic to variations in genre , sentence length , vocabulary , language etc . 2 RELATED WORK . Early work on text summarization started around the mid 20th century . In one of the earliest work by Luhn ( 1958 ) , summarization was used to generate abstracts automatically from the excerpts of technical papers and magazine articles . Since then , various algorithms for text summarization have been proposed for multiple domains like newspaper articles ( Wu & Hu , 2018 ) , scientific articles ( Teufel & Moens , 2002 ) and blogs ( Hu et al. , 2007 ) . A comparative study in Ceylan et al . ( 2010 ) revealed that a single text summarization algorithm does not perform equally well for all domains . This is because of differences in the nature and style of the sources . Therefore it was concluded that separate algorithms need to be designed for different domains of text sources . Traditionally , most text summarization algorithms have focused on summarizing newspaper and scientific articles ( Luhn , 1958 ; Teufel & Moens , 2002 ; Lee et al. , 2005 ) . Although , there have been recent attempts to summarize fictional stories ( Kazantseva & Szpakowicz , 2010 ; Lloret & Palomar , 2009 ; Mihalcea & Ceylan , 2007 ) , the objective of these algorithms is to let the reader decide if the text is interesting enough to read . Therefore , these algorithms do not intend to encapsulate the entire information of the source text . An early general purpose text summarization algorithm for telegraphic purposes was proposed in Grefenstette ( 1998 ) . This was a rule-based algorithm , with each rule allowing some specific components of the source sentence in the summary . The first ( and the most drastic ) generated a stream of all the proper nouns in the text . The second generated all nouns present in the subject or object position . The third , in addition , included the head verbs . The least drastic reduction generated all subjects , head verbs , objects , subclauses , prepositions and dependent noun heads . Since this algorithm is based on definitive guidelines , it does not generalize well for shorter sentences . Additionally , it is not clear which rule should be applied to a sentence to generate its appropriate summary . There have also been numerous approaches using statistics based on syntax to generate telegraphic summaries ( Jing , 2000 ; Riezler et al. , 2003 ; Knight & Marcu , 2000 ) . Typically , these approaches drop words from the source which are not related to the neighboring words , whilst trying to maintain the grammaticality of the summary . The grammatical correctness of the summary is checked using statistical models . These approaches do not tend to give accurate results for shorter inputs as the constraints for the removal of words are difficult to satisfy . Recently , deep learning based methods have been enormously successful in the domain of feature learning with little or no manual intervention . These approaches have been applied to core Linguistics problems such as machine translation ( Bahdanau et al. , 2014 ) , language modeling ( Mikolov et al. , 2010 ) and image captioning ( Johnson et al. , 2016 ) . Although these approaches achieve better results than handcrafted algorithms , they are supervised approaches which require a huge amount of data . Attempts have been made to counter this by exploring unsupervised learning-based algorithms , eliminating the need for labeled ground truth data . In this work , we have proposed a new architecture for telegraphic summarization , known as NUTS . The proposed architecture implicitly learns to retain the significant segments of the source through several encoder-decoder networks ( Sutskever et al. , 2014 ) to generate its telegraphic summary . 3 NUTS MODEL DETAILS . In this section , we explain the details of the proposed model . We first describe the building blocks of the architecture . Thereafter , we introduce the various components of the loss function and explain each loss term and its usage . 3.1 ARCHITECTURE DETAILS . Given an input sentence s = w1 , w2 ... , wk containing k words , the network tries to reconstruct the same sentence s , with fewer number of words . The network architecture is such that it drops the words it deems inferable for the final reconstruction . The network generates an indicator vector I = I1 , I2 ... , Ii , ... , Ik at an intermediate stage . The value of this indicator vector corresponds to the presence of word wi in the summary subset T of the sentence set S. The words present in T are used as the telegraphic summary of the sentence in order of their occurrence in s. Formally , the network tries to find an I∗ such that the probability p ( s|s I ) is maximized and∑k t=1 It is minimized , jointly . The probability p ( s|s I ) can be decomposed further as shown in Equation 1 I∗ = argmax I k∏ t=1 p ( wt| ( w1 × I1 ) , ( w2 × I2 ) , ... , ( wk−1 × Ik−1 ) ) ( 1 ) The entire architecture is built using RNN with LSTM cells proposed in Hochreiter & Schmidhuber ( 1997 ) . The equations for LSTM gates are mentioned in Equation A.1 in the Appendix section . The proposed architecture has the following major components ( illustrated in Fig . 1 ) : 1 . Sentence encoder ( Es ) encodes the input sentence s. Every word wi in the sentence is converted into a d-dimensional vector ei . The sequence of these embedded vectors is fed as input to the encoder at each time step . The final hidden state of Es ( he1 ) acts as a sentence embedding for s. 2 . Indicator decoder ( DI ) is initialized by the final hidden state of Es . The output of this decoder at each time step is passed through a network of two fully connected layers to generate a single output value . We intend this output value to be close to either zero or one , denoting the value of indicator vector I . 3 . Summary encoder ( E s ′ ) encodes the masked sentence s′ = s I , where represents element-wise multiplication . Therefore , the words corresponding to Ii≈ 0 are effectively skipped . The final hidden state of E s ′ acts as a sentence embedding for s′ . 4 . Summary decoder ( D s ′ ) is initialized by the final hidden state of E s ′ ( he2 ) . This decoder aims to regenerate the input sentence s from s′ . This motivates DI to generate I in such a way that s can be easily reconstructed from s′ . The output at each time step from the LSTM cell in D s ′ are fed to a dense layer , Ws . This dense layer computes the distribution over the vocabulary words from the decoders ’ hidden states . 3.2 LOSS FUNCTION . In order to understand the function of each of the modules of the architecture , we outline the need for each module ’ s loss term and their effects on the output of the network . 1 . Summary length loss ( L1 ) : ensures that the ratio between the lengths of s′ and s is close to a specific value r. This loss is calculated from the output of DI . We have observed that extreme values of r fail to generate the desired output . In our experiments , low values of r hindered the ability of the summary decoder D s ′ to reconstruct s from s′ . On the other hand , if r is very large , no words are dropped and s′ and s become equal . Therefore , the value of r is set such that the summary is neither too short nor too long . Mathematically , L1 can be represented as shown in Equation 2 . We calculate the value of Len ( s′ ) in this equation as the sum of elements of I . The optimum value of r for our experiments was found to be 0.65 . L1 = ( Len ( s′ ) Len ( s ) − r ) 2 ( 2 ) 2 . Summary Decoder Reconstruction loss ( L2 ) ensures that the words retained in s′ are sufficient for reconstructing s. The L2 loss is calculated from the output of Ds′ . It aims to maximize the probability of occurrence of wi ( the ith word in s ) , given the final hidden state of E s ′ ( he2 ) and all previous words in the masked sequence ( w′ < i ) , encapsulated by the hidden state vector from the previous time step . The mathematical representation of L2 is shown in Equation 3 . L2 = − Len ( s ) ∑ i=1 logP ( wi|w′ < i , he2 ) ( 3 ) 3 . Binarization loss ( L3 ) is calculated from the output of DI . Ideally , we want all values in I to be either zero or one . However , if the target indicator is set to these hard values , non-differentiability is introduced into the network . Therefore , we relax the requirement of hard values and fix the range of DI output as [ 0 , 1 ] using the sigmoid activation function . Additionally , we model L3 such that the outputs tend to be distant from the mid-value of 0.5 . This pushes the outputs close to zero or one , effectively fulfilling the objective of I . Mathematically , this is achieved using Equation 4 . A larger value of b results in a higher penalty for mid-ranged values of Ii . The value of a is such that L3 is always non-negative . In our experiments , the best value of b was found to be 5 . L3 = ∑Len ( s ) i=1 ( a− b ( Ii − 0.5 ) 2 ) Len ( s ) ( 4 ) 4 . Linkage loss ( L4 ) In our experiments we noticed that without an additional loss to govern how the words are masked , the network tends to learn repetitive masking patterns . Such skewed patterns are undesirable - while certain positions of s are well represented in s′ , other positions are entirely missing . A few examples of such summaries are shown in Fig . A.1 of Appendix 1 . This motivates the need to couple ease of decoding with words dropped from the summary . We introduce a novel ‘ linkage ’ loss function to achieve this . The linkage loss facilitates the network to drop words which are deemed inferable . It is applied to the outputs of DI and Ds′ simultaneously . Therefore , it correlates the indicator vector and its effect on reconstruction . It penalizes the network if a ) it decides to mask a word but is unable to reconstruct it later or b ) it decides to include a word which it could decode easily . The mathematical equation describing the linkage loss is shown in Equation 5 . The variable χi denotes the relative ease of decoding word wi , given w′ < i and he2 . The value of χi lies between 0 and 1 ( both included ) . A larger value χi indicates that wi is difficult to decode . The value of χi is calculated using Equation 6 . The addition of linkage loss also results in significantly better results significantly as seen in Section 5 . L4 = Len ( s ) ∑ i=1 ( Iie ( 1−χi ) + ( 1− Ii ) eχi − 1 ) ( 5 ) χi = |logP ( wi|w′ < i , he2 ) | max1≤i≤Len ( s ) |logP ( wi|w < i , he2 ) | ( 6 ) It can be seen from Equation 5 that L5 is minimized when either a ) χi = 0 and Ii = 0 simultaneously ( signifying that wi is easy to decode and should be dropped ) or b ) χi = 1 and Ii = 1 simultaneously ( indicating that hard-to-decode words should be retained ) . The cumulative loss function ( L ) is a linear combination of the losses described above . Mathematically , the equation for L is represented in Equation 7 . L = λ1L1 + λ2L2 + λ3L3 + λ4L4 ( 7 ) The weights λ1 , λ2 , λ3 , and λ4 have been set to 3 , 2 , 50 and 3 respectively for our experiments . Since this is an unsupervised approach currently the weights are experimentally determined and can be fine-tuned on supervision .

Summary: The authors consider the problem of telegraphic sentence compression: they train a system in an unsupervised fashion to predict which words can be dropped from a sentence without drastic loss of information. To that end, they propose a new auto-encoding type architecture which uses the extracted words as latent code, and, most importantly, a linkage loss which relates a word's perplexity given the summary of its left context to its likelihood of being retained. The model itself is sober and well motivated, and the linkage loss is, to the best of my knowledge, original. The authors show that their method outperforms some simple baselines in terms of ROUGE and compression on a small human-annotated test set.