Multi‐layered attentional peephole convolutional LSTM for abstractive text summarization

ETRI JournalVolume 43, Issue 2 p. 288-298 ORIGINAL ARTICLEFree Access Multi-layered attentional peephole convolutional LSTM for abstractive text summarization Md. Motiur Rahman, Corresponding Author Md. Motiur Rahman motiur@cvasu.ac.bd orcid.org/0000-0001-7472-4085 Chattogram Veterinary and Animal Sciences University, Chattogram, Bangladesh Correspondence Md. Motiur Rahman, Chattogram, Chattogram Veterinary and Animal Sciences University, Bangladesh. Email: motiur@cvasu.ac.bdSearch for more papers by this authorFazlul Hasan Siddiqui, Fazlul Hasan Siddiqui orcid.org/0000-0003-4825-6315 Dhaka University of Engineering and Technology, Dhaka, BangladeshSearch for more papers by this author Md. Motiur Rahman, Corresponding Author Md. Motiur Rahman motiur@cvasu.ac.bd orcid.org/0000-0001-7472-4085 Chattogram Veterinary and Animal Sciences University, Chattogram, Bangladesh Correspondence Md. Motiur Rahman, Chattogram, Chattogram Veterinary and Animal Sciences University, Bangladesh. Email: motiur@cvasu.ac.bdSearch for more papers by this authorFazlul Hasan Siddiqui, Fazlul Hasan Siddiqui orcid.org/0000-0003-4825-6315 Dhaka University of Engineering and Technology, Dhaka, BangladeshSearch for more papers by this author First published: 18 December 2020 https://doi.org/10.4218/etrij.2019-0016AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Abstractive text summarization is a process of making a summary of a given text by paraphrasing the facts of the text while keeping the meaning intact. The manmade summary generation process is laborious and time-consuming. We present here a summary generation model that is based on multilayered attentional peephole convolutional long short-term memory (MAPCoL; LSTM) in order to extract abstractive summaries of large text in an automated manner. We added the concept of attention in a peephole convolutional LSTM to improve the overall quality of a summary by giving weights to important parts of the source text during training. We evaluated the performance with regard to semantic coherence of our MAPCoL model over a popular dataset named CNN/Daily Mail, and found that MAPCoL outperformed other traditional LSTM-based models. We found improvements in the performance of MAPCoL in different internal settings when compared to state-of-the-art models of abstractive text summarization. 1 INTRODUCTION The process of shortening a large text while keeping the original meaning intact is known as text summarization. A well-organized and concise summary helps humans to obtain the overall scenario of a long text quickly with little labor. There are two forms of text summarization: extractive and abstractive [1]. Extractive summarization generates a summary by taking a subset of sentences or phrases from the original text. It does not generate new sentences, nor does it paraphrase any existing sentences [2, 3]. Extractive text summarization has become obsolete with the enhancement of technology. By contrast, abstractive text summarization is a process of summary generation that paraphrases the given text and preserves the salient meaning of the source text in the summary text [4]. Researchers have recently been drawn to this alternative approach of extractive text summarization. Despite notable success in automated text summarization, conventional methods are unable to resolve all the difficulties associated with text summarization. The quality of a generated summary is measured on the basis of semantic similarity and syntactic structure. The conventional methods, however, focus on one factor only: either semantic similarity or syntactic structure [5]. Extractive text summarization is well-suited for ensuring the syntactic structure during summary generation, but the generated summary is not semantically coherent. By contrast, abstractive text summarization fails to ensure the syntactic structure of the generated summary, but it is effective in maintaining the semantic coherence [6]. The reason behind the semantic coherence of abstractive text summarization is accomplished by learning the relations between words and choosing the right words to keep the summary semantically relevant. The working procedure of abstractive text summarization is very similar to sequence-to-sequence modeling as both of them take a sequence of text as an input, and a text sequence is generated as a final summary [7]. Here, the modeling of sequence-to-sequence follows the recurrent neural network (RNN) concept, in which a text sequence is provided as an input and looks after each element of that given sequence to determine the next element of output. The long short-term memory (LSTM) is a well-known framework of sequence-to-sequence modeling [8]. It solves many natural-language-based problems such as machine translation and speech recognition. The task of summary generation from a text is not as easy as machine translation because of the variable length of the source text and its summary text [9]. Hence, the mapping between a source text and its summary text is challenging during summary generation. Most of the existing LSTM-based abstractive text summarization models use the traditional architecture of LSTM. One variation of LSTM is known as the peephole convolutional LSTM (PCLSTM), which heavily depends on previous memory-cell content to determine the content of the memory of the current cell [10]. The more the abstractive summary generation procedures are dependent on the previous memory cell content, the more effective the solution that is found from these procedures. This study, introduces a multilayered attentive peephole convolutional LSTM-based (MAPCoL) model for abstractive text summarization. In this study, we develop an abstractive text summarization model MAPCoL based on the PCLSTM with an attention strategy to generate a summary from a long text. The model is developed by incorporating the multilayer of PCLSTM. The multilayer of PCLSTM is used to capture the complete interaction between the input and output of the model. This multilayer architecture of our model helps to generate a syntactically well-formed summary. The proposed model is implemented by using Keras with a Tensonflow backend, which is a library of Python. The model is applied over a popular dataset to test its versatility. We also compare the performance of the MAPCoL model with state-of-the-art models of abstractive text summarization by tuning internal settings. This paper is divided into five sections. The first section provides a brief overview of related works. The second section examines the limitations and benefits of traditional LSTM and PCLSTM in summary generation. A new methodology and its evaluation criteria are described in the third section. The next chapter describes the obtained results and the reasons behind those results. Our conclusions and future scope of this work are discussed in the last section of this paper. 2 RELATED WORK Summary generation, especially abstractive summary generation, has become popular with the advancement of machine learning. Although a large amount of research has been conducted to develop an efficient and effective model for extractive text summarization, the number of studies that examined abstractive summarization is very small. Some works that focused on both summarization approaches are illustrated below. Several neural-network-based methods have been developed for extractive text summarization. A data-driven approach using a feedforward neural network for text summarization is shown in Ref. [11]. Narayan showed a reinforcement learning-based algorithm for sentence ranking that optimizes the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score for extractive summarization [12]. A neural model LeadR was developed to produce a probability distribution over positions of sentences. This helps to locate sentences for summary generation [13]. An end-to-end neural network framework is presented for extractive summarization that selects sentences efficiently based on a calculated score for the summary [14]. Recently, summary generation in a data-driven manner has become very popular, and some works to accomplish this task have been conducted. A neural attention mechanism was introduced in Ref. [15] to build sentences and use them for summarization. An RNN-based sequence model, SummaRuNNer, was shown in Ref. [16]. This model generates a summary by considering content, salience, and novelty. In a recent work, LSTM was used for extractive text summarization. A deletion-based LSTM neural network model was used for sentence compression [17, 18]. A comparative study showed that RNN-based summarization techniques return better results than traditional extractive summarization techniques [16]. The RNN is also widely used for abstractive text summarization. In several works [19, 20], recurrent neural-network-based models with attention were used for abstractive summary generation. An RNN technique with sequence-to-sequence modeling was shown for abstractive text summarization in Ref. [7]. The researchers utilized a bidirectional neural network and incorporated a Gated Recurrent Unit (GRU) to introduce a new summarization model. Recently, the encoder-decoder framework of RNN has drawn interest in abstractive text summarization. An attentional encoder-decoder model based on LSTM was explained for abstractive text summarization in Ref. [21, 22]. Here, the encoder-decoder model could be bidirectional or unidirectional. LSTM with a convolutional neural network (CNN) has become very popular in recent times. ATSDL (Abstractive Text Summarization using Deep Learning) [9] is an abstractive text summarization technique that is modeled by combining LSTM and CNN, and considers both the semantic coherence and syntactic structure of sentences. In this model, input is given as phrases in lieu of simple text, and a challenging task is to identify the phrases from the input text [9]. Convolutional LSTM consists of a memory block and three controlling gates: input, forget, and output gates. These gates control the contents of the memory block and the output of the current state. Symbols and their corresponding meanings are as follows. ct −1 and ct are, respectively, the contents of the previous and current memory cells; ht −1 and ht are, respectively, the outputs of the previous and current states; xt is an input vector; X is bitwise multiplication; + is bitwise summation; tanh is a hyperbolic tangent function; and σ is a sigmoid function. bf , bi , bc , and bo are the biases of the different gates. Convolutional LSTM is a variation of traditional LSTM that has been applied successfully in different types of prediction. A convolutional LSTM model named ConvLSTM was developed for the precipitation nowcasting problem, and its performance was compared with another model based on a fully connected LSTM (FC-LSTM) [23]. Precise correlation among weather data was considered in this model, which assists predictions with better accuracy than the FC-LSTM model. Another work for time-series classification was proposed based on convolutional LSTM [24]. The researchers used fully convolutional networks along with the recurrent neural network unit LSTM for classifying time-series data, and they achieved the highest classification accuracy. The two aforementioned convolutional LSTM-based models make predictions with more accuracy than traditional LSTM-based models. The configuration of a convolutional LSTM is better suited than the traditional LSTM for capturing long data patterns effectively. This influences the classification and prediction accuracy. The above discussion indicates that most works were conducted to generate abstractive summaries using traditional LSTMs. However, none of the studies used parameter optimization to obtaining the highest accuracy. However, traditional LSTM has some limitations. For example, it does not always ensure the semantic coherence of a summary. Because the gate of a traditional LSTM has no connection with the previous memory cell. Traditional LSTM also does not guarantee syntactic coherence. Additionally, the performance of the proposed model could be increased by finding the optimal configuration of parameters. Hence, in our experiment, a model for generating abstractive summaries is developed by utilizing multiple layers of PCLSTM. We added the concept of an attention mechanism that assists in the generation of a concise memory. The process of abstractive summary generation is well fitted with the working mechanism of PCLSTM as it can predict the sequence of text efficiently. 3 ABSTRACTIVE TEXT SUMMARIZATION MODELS Long short-term memorys in various forms with different architectures have become popular for abstractive text summarization. Traditional LSTM, an initial architecture of LSTM [25], is widely used in text summarization. However, it has some limitations, for example, dependencies between cells are not strong. This can be resolved by using a PCLSTM, another form of LSTM [23]. The details of the traditional LSTM and PCLSTM are described here. 3.1 Traditional LSTM unit Long short-term memory, a recurrent neural network unit, is capable of holding and remembering values for a specific period of time. A popular form of LSTM known as traditional LSTM is well-suited for solving sequence-to-sequence related problems where a text sequence is given as input and a text sequence is generated as output. The structural design of the traditional LSTM is displayed in Figure 1. This architecture is composed of a memory cell and three gates: forget, input, and output. Here, the memory cell stores data, and the three other gates control the cell states [26]. Figure 1Open in figure viewerPowerPoint Traditional LSTM consists of a memory-block, and three controlling gates such as input, forget, and output gates. These gates control the contents of memory-block, and the output of the current state. Symbols and their corresponding meanings are as follows. c t −1 and ct are respectively the contents of the previous and the current memory cells, ht− 1 and ht are respectively the outputs of the previous and the current states, xt is an input vector, X is a bitwise multiplication, + is a bitwise summation, tanh is a hyperbolic tangent function, _ is a sigmoid function. bf , bi , bc , and bo are the bias of the different gates According to Figure 1, a line is passed through the top of the architecture, which is known as a memory pipe. The input of the memory pipe is the previous memory content ct −1. The content of the memory pipe is controlled by the three controlling gates. Here, the first operation found in the memory pipe is bitwise multiplication (x) between the previous memory content and the output received from the forget gate. By using a sigmoid function, the output of the forget gate is converted in a range of 0 to 1. If the generated output from the forget gate is near 0, then most of the previous memory content will nearly be forgotten owing to the multiplication between the previous memory content and the output of the forget gate. By contrast, a significant portion of the previous memory content is passed through the memory pipe when the output of the forget gate is close to 1. Bitwise summation (+) is the second operation in the memory pipe and merges the contents of the temporary memory, generated by the input gate, with that of the previous memory, which generates the final memory ct . The forget gate is a single-layer neural network that performs several operations and determines the amount of memory content of the previous cell that will be used at the current timestamp. The forget gate receives the output state ht −1 of previous timestamps, an input vector xt , and a bias bf as input. The forget gate then generates an accumulated output that is passed through a sigmoid activation function to generate the final output ft within the range of 0 to 1. The final output ft is multiplied by the previous memory content ct −1. The value of ft determines the significance of the old memory content over the current state. Here, the lower value of ft restricts the previous memory content to influence the current state, while the higher value of ft allows more previous memory content to contribute over the current state. (1) The input gate controls how much the temporary memory, as described earlier, influences the current memory cell. It takes similar input as the forget gate and produces a temporary memory. The temporary memory is the multiplicative result of a sigmoid function and a hyperbolic tangent function. The sigmoid function produces an output it that will be merged with the previous memory. The tanh activation function, whose output is between 0 and 1, determines the amount of temporary memory that will be merged with the memory content of the previous timestamp. A temporary memory having a smaller value of tanh function makes a lesser contribution to the current memory cell ct . By contrast, a temporary memory having a larger value of the tanh function makes a greater contribution to the current memory cell. (2) (3) The output gate determines the content ht of the current state at time t. The inputs taken by this gate are similar to those of other gates, such as the output of the previous timestamp ht −1 and an input vector xt with a different bias value bo . This produces ot as output. The value of ot is within the range 0-1 and determines how much of ct should be carried out as the output of the current state. When ot = 1, then ht = ct , that is, the entire value of ct is passed to the next state as ht . (4) (5) According to the structural design of the traditional LSTM as shown in Figure 1, the controlling gates are not connected to the memory cell of the previous timestamp. Additionally, the output gate of the traditional LSTM remains mostly closed at the time of training. Hence, as mentioned earlier, owing to these demerits and the closure of the output gate, the memory content of the previous timestamp is inaccessible. This problem has an adverse effect on the performance of text summarization. A variation of the LSTM, called the peephole convolutional LSTM, can overcome this problem. These problems can be resolved by utilizing the merits of the PCLSTM. 3.2 Peephole convolutional LSTM unit Long short-term memory, a recurrent neural network unit, is widely used in sequence-prediction-related problems. There are different architectures of LSTMs for different purposes. The configuration between gates in the traditional LSTM restrict the utilization of the memory content of previous timestamps at the time of closure of the output gate [23]. This problem in the traditional LSTM can be solved by using an additional dedicated connection between each gate and the memory content of the previous timestamp. An LSTM with this extra connection is known as PCLSTM, and this connection is denoted as a peephole connection. The peephole connection of a PCLSTM enables all three controlling gates to access the memory content of previous cells even when the output gate is closed. The working principle of the PCLSTM is similar to that of the traditional LSTM except for calculations for the additional connections. The architecture of a PCLSTM is shown in Figure 2 and is implemented as follows: (6) (7) (8) (9) (10) Figure 2Open in figure viewerPowerPoint Peephole convolutional LSTM (PCLSTM) is an extension of the traditional LSTM. In a PCLSTM, the previous memory cell ct −1 is linked with each controlling gate, known as peephole connections. These peephole connections allow an extra parameter, memory content ct −1, to be fed as an input in PCLSTM. This is absent in a traditional LSTM. This extra input in each gate permits access to the memory content of previous cells in any condition The PCLSTM takes an additional parameter of the memory content of previous cell ct −1 as input along with the inputs of the traditional LSTM, while the traditional LSTM does not take ct −1 as input. The connection with ct −1 in a PCLSTM has a significant impact on the accuracy of sequence-to-sequence prediction tasks. We know that abstractive text summarization is a type of sequence-to-sequence prediction. Hence, this interesting feature of the PCLSTM has motivated the application of this approach to generating abstractive summaries. 3.3 MAPCoL model In this work, we develop a model for abstractive text summarization using multiple layers of an attentional peephole convolutional LSTM (MAPCoL) that is based on sequence-to-sequence modeling. The architecture of our developed MAPCoL model is shown in Figure 3. The model is composed of two parts: encoder and decoder. Both the encoder and decoder consist of a collection of PCLSTMs, a type of recurrent neural network. The first four PCLSTMs in Figure 3 represent the encoder, and the next four PCLSTMs represent the decoder. The encoder takes a source text as input, tries to understand the input sequences, and produces a small dimensional representation of the source text. This representation is provided to the decoder, which generates another representation, the ultimate output of the encoder-decoder network, after all of the information is decoded. Figure 3Open in figure viewerPowerPoint An encoder-decoder network of a sequence-to-sequence model that consists of a PCLSTM block. It takes a sequence of input and produces another sequence as output. The LSTM encoder understands the input sequence and generates a representation that is forwarded to the decoder that eventually generates an output. EOL stands for end of line, which indicates the end of given input Figure 3 shows that an input "How are you" is passed through the encoder-decoder network. Here, all the words are passed sequentially through the encoder one at a time. After passing the last word through the encoder, the generated representation of the encoder is forwarded to the decoder as input. The decoder generates one word at each time step as output, and the generated word is provided along with other information to generate the next word. After all iterations, the generated output is "I am fine.". The working procedure of MAPCoL is shown in Figure 4. We introduced multiple layers of PCLSTMs in MAPCoL to generate an abstractive summary of a given text. Here, multiple layers of PCLSTMs denote the availability of more than one hidden layer in this model. When the layer size is greater than 3, the overall performance of this model often drops because of overfitting and gradient decay over the layers from the saturated use of activation functions. Hence, two hidden layers are used to build the model. In the beginning, the source text is transformed into a distributed representation by an embedding layer. The embedding layer performs a simple matrix multiplication that transforms each word into its corresponding word embedded in a distributed representation. A multilayer neural network with a high number of hidden units is used to concatenate these distributed representations. The distributed representation is an intermediate representation of a text that a neural network can process very easily. The given input sequence is transmitted through two hidden layers of PCLSTMs. Each hidden layer of the PCLSTM contains 100 hidden units. When the encoder finishes the encoding of the entire distributed representation, the decoder starts to predict the first word of the summary using an attention mechanism in the softmax layer. The attention mechanism estimates a weight for each word using a probability function that helps remember some important factors of the input. The attention value of a word determines how much attention should be paid to that word during generation of the output as a summary [22]. After passing through the last hidden layer, the average weight of each word is computed. This is passed through the softmax layer along with the content of the last hidden layer for predicting the next word. Figure 4Open in figure viewerPowerPoint Topology of our MAPCoL model for generating abstractive summaries. Two layers of PCLSTM are used to build the model. The text is embedded using an embedding layer. Each PCLSTM layer has 100 hidden units. The softmax layer removes outliers from the output and maps a vector to a real value The attentional weight of an input word at position t is computed when outputting the t′-th word as follows: (11)where represents the last hidden layer generated after processing the tth input word, and represents the last hidden layer generated from the current step of decoding. The introduction of attention strategy handles the difficulties of mapping a large source text into a fixed-length output. The softmax layer removes outliers from the output and maps a vector to a real value. The performance of our MAPCoL model was evaluated over the CNN/Daily Mail dataset. A sample of the reference summary (manmade) and system summary is shown in Table 1. Table 1 shows that the quality of the system-generated summary is better in terms of semantic coherence than the reference summary. The syntactic structure of the system-generated summary also is more structured than that of the reference summary. Table 1. System summary and reference summary Text System summary Reference summary The Administration of Union Territory Daman and Diu has revoked its order that made it compulsory for women to tie rakhis to their male colleagues on the occasion of Rakshabandhan on August 7. The administration was forced to withdraw the decision within 24 h of issuing the circular after it received flak from employees and was slammed on social media. Daman and Diu revokes mandatory Rakshabandhan in offices order Daman and Diu revokes mandatory Rakshabandhan in offices order Malaika Arora slammed an Instagram user who trolled her for "divorcing a rich man" and "having fun with the alimony." "Her life now is all about wearing short clothes, going to [the] gym or salon, [and] enjoying vacation[s]," the user commented. Malaika responded, "You certainly got to get your damn facts right before spewing shot on me… when you know nothing about me." Malaika hits people who trolled her for divorcing [a] rich man Malaika slams [a] user who trolled her for 'divorcing [a] rich man' Note The 'text' column represents the input text in our model. The 'system summary' column shows the generated summary from our MAPCoL model. The human generated summary is shown in the 'reference summary' column. 4 EXPERIMENT In this study, a model named MAPCoL is developed using a PCLSTM for generating abstractive summaries. Here, we use multilayers of a PCLSTM with 100 hidden units in each layer. We also use an attention mechanism for predicting precise results. 4.1 Experimental setup The goal of this experiment is to develop a robust and reliable model for abstractive text summarization based on PCLSTM. MAPCoL is implemented using TensorFlow backed with Keras, which is a Python library. In our experiment, the batch size and number of epochs have significant roles with regard to performance. The batch size denotes the portion of the dataset that is passed together through the model at a single time. The same data are passed through the model multiple times; this is known as the epoch number. For this experiment, after performing parameter tuning, we select 64 as the batch size and 200 as the epoch number to train and test the model. By using this configuration, we hypothesize that our MAPCoL model produces better accuracy with regard to the syntactic structure and semantic coherence with this configurations. 4.2 Experimental dataset We evaluated the performance of our MAPCoL model over the CNN/Daily Mail dataset, which utilizes the news from CNN and the Daily Mail newspaper. The dataset contains 286 817 training pairs, 13 368 validation pairs, and 11 487 test pairs of data. 4.3 Evaluation method Traditionally, the quality of a summarized text is evaluated manually by humans by judging parameters such as conciseness, relevancy, coherence, grammar, and readability [27]. Manual evaluation to determine the quality is time-consuming, difficult, and repetitive. Therefore, researchers developed automated methods to evaluate summaries. Most of these methods are based on the similarity measure between a summary and its original text, but they do not relate the judgment with human judgment [28]. Hence, a recall-oriented method named ROUGE was developed to evaluate the quality of the summary [28]. Here, the system summary and the reference summary (human-generated) are compared to evaluate the summary quality. The ROUGE score is measured as follows: (12) where n is the number of consecutive words, Ref is the set of reference summaries, Count (gram n ) counts the number of n consecutive words of the reference summary, and Countmatch (gram n ) estimates n number of consecutive words matched between the reference summary and system summary. Here, the human-generated summary is denoted as the reference summary, and our model-generated summary is denoted as the system summary. The division of the number of matches between the two referred summaries by the length of the reference summar