Text Summarization Using Deep Neural Networks | by Shivam Duseja | Towards Data Science

The amount of textual data being produced every day is increasing rapidly both in terms of complexity as well as volume. Social Media, News articles, emails, text messages (the list goes on..), generate massive information and it becomes cumbersome to go through lengthy text materials (and boring too!). Thankfully with the advancements in Deep Learning, we can build models to shorten long pieces of text and produce a crisp and coherent summary to save time and understand the key points effectively.
We can broadly classify text summarization into two types:
1. Extractive Summarization: This technique involves the extraction of important words/phrases from the input sentence. The underlying idea is to create a summary by selecting the most important words from the input sentence
2. Abstractive Summarization: This technique involves the generation of entirely new phrases that capture the meaning of the input sentence. The underlying idea is to put a strong emphasis on the form — aiming to generate a grammatical summary thereby requiring advanced language modeling techniques.
In this article, we will use PyTorch to build a sequence 2 sequence (encoder-decoder) model with simple dot product attention using GRU and evaluate their attention scores. We will further look into metrics like — BLEU, ROUGE for evaluating our model.
Dataset used :We will work on the wikihow dataset that contains around 200,000 long sequence pairs of articles and their headlines. This dataset is one of the large-scale datasets available for summarization with the length of articles varying considerably. These articles are quite diverse in their writing style which makes the summarization problem more challenging and interesting.
For more information on the dataset: https://arxiv.org/abs/1810.09305
Pre-processing and cleaning is an important step because building a model on unclean and messy data will in turn produce messy results. We will apply the below cleaning techniques before feeding our data to the model:
Converting all text to lower case for further processingParsing HTML tagsRemoving text between () and []Contraction Mapping — Replacing shortened version of words (for e.g. can’t is replaced with cannot and so on)Removing apostropheRemoving punctuations and special charactersRemoving stop words using nltk libraryRetaining only long words, i.e. words with length > 3
Converting all text to lower case for further processing
Parsing HTML tags
Removing text between () and []
Contraction Mapping — Replacing shortened version of words (for e.g. can’t is replaced with cannot and so on)
Removing apostrophe
Removing punctuations and special characters
Removing stop words using nltk library
Retaining only long words, i.e. words with length > 3
We will first define the contractions in the form of a dictionary:
## Find the complete list of contractions on my Github Repocontraction_mapping = {"ain't": "is not", "aren't": "are not"}stop_words = set(stopwords.words('english'))def text_cleaner(text,num):  str = text.lower()  str = BeautifulSoup(str, "lxml").text  str = re.sub("[\(\[].*?[\)\]]", "", str)  str = ' '.join([contraction_mapping[t] if t in contraction_mapping      else t for t in str.split(" ")])  str = re.sub(r"'s\b","",str)  str = re.sub("[^a-zA-Z]", " ", str)    if(num==0):      tokens = [w for w in str.split() if not w in stop_words]  else:      tokens=str.split()  long_words=[]    for i in tokens:      if len(i)>3:          #removing short words          long_words.append(i)    return (" ".join(long_words)).strip()#calling the above functionclean_text = []clean_summary = []for t in df['text']:    clean_text.append(text_cleaner(t,0))for t in df['headline']:    clean_summary.append(text_cleaner(t,0))
Before feeding the training data to our deep model, we will represent each word as a one-hot vector. We will then need a unique index per word to use it as the input and output of the network.
In order to do so, we will create a helper class Lang which has word-> index and index-> word mappings along with the count of each word. To read data in our model, we have created pairs of our input and output in the form of a list (Pairs-> [Input, Output])
We will be using a seq2seq(encoder-decoder architecture) model with a simple dot product attention. The underlying idea of choosing this architecture is that we have a many-to-many problem at hand(n number of words as input and m number of words as output). The figure below shows the detailed architecture diagram for this model.
There are four major components in this architecture:
Encoder: The encoder layer of the seq2seq model extracts information from the input text and encodes it into a single vector, that is a context vector. Basically, for each input word, the encoder generates a hidden state and a vector , using this hidden state for the next input word.
We have used GRU(Gated Recurrent Unit) for the encoder layer in order to capture long term dependencies - mitigating the vanishing/exploding gradient problem encountered while working with vanilla RNNs.The GRU cell reads one word at a time and using the update and reset gate, computes the hidden state content and cell state.
I found the below two links useful for getting more information on the working of GRU:Empirical Evaluation of Gated Recurrent Neural Networks on Sequence ModelingGated Recurrent Units (GRU)
Decoder: The decoder layer of a seq2seq model uses the last hidden state of the encoder i.e. the context vector and generates the output words. The decoding process starts once the sentence has been encoded and the decoder is given a hidden state and an input token at each step/time. At the initial time stamp/state the first hidden state is the context vector and the input vector is SOS(start-of-string). The decoding process ends when EOS(end-of-sentence) is reached. The SOS and EOS tokens are explicitly added at the start and end of each sentence respectively.
Attention Mechanism: Using the encoder-decoder architecture, the encoded context vector is passed on to the decoder to generate an output sentence. Now what if the input sentence is long and a single context vector cannot capture all the important information. This is where attention comes into picture!!!
The main intuition of using attention is to allow the model to focus/pay attention on the most important part of the input text. As a blessing in disguise, it also helps to overcome the vanishing gradient problem.There are different types of attention — additive, multiplicative, however, we will use the basic dot product attention for our model.
1. Attention scores are first calculated by computing the dot product of the encoder(h) and decoder(s) hidden state2. These attention scores are converted to a distribution(α) by passing them through the Softmax layer.3. Then the weighted sum of the hidden states (z) is computed.4. This z is then concatenated with s and fed through the softmax layer to generate the words using ‘Greedy Algorithm’ (by computing argmax)
In this architecture, instead of directly using the output of last encoder’s hidden state, we are also feeding the weighted combination of all the encoder hidden states. This helps the model to pay attention to important words across long sequences.
Supporting Equations
Teacher Forcing: In general, for recurrent neural networks, the output from a state is fed as an input to the next state. This process causes slow convergence thereby increasing the training time.
What is Teacher ForcingTeacher forcing addresses this slow convergence problem by feeding the actual value/ground truth to the model. The basic intuition behind this technique is that instead of feeding the decoders predicted output as an input for the next state, the ground truth or the actual value is fed to the model. If the model predicts a wrong word it might lead to a condition wherein all the further words that are predicted are incorrect. Hence, teacher forcing feeds the actual value thereby correcting the model if it predicts/generates a wrong word.
Teacher forcing is a fast and effective way to train RNNs, however, this approach may result in more fragile/unstable models when the generated sequences vary from what was seen during the training process.To deal with such an isssue, we will follow an approach that involves randomly choosing to use the ground truth output or the generated output from the previous time step as an input for current time step.
Below code snippet shows the dynamic implementation of Teacher Forcing
teacher_forcing_ratio = 0.5use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False    if use_teacher_forcing:        # Teacher forcing: Feed the target as the next input        for di in range(target_length):            decoder_output, decoder_hidden, decoder_attention = decoder(                decoder_input, decoder_hidden, encoder_outputs)            loss += criterion(decoder_output, target_tensor[di])            decoder_input = target_tensor[di]  # Teacher forcing    else:        # W/O teacher forcing: use own predictions as the next input        for di in range(target_length):            decoder_output, decoder_hidden, decoder_attention = decoder(                decoder_input, decoder_hidden, encoder_outputs)            topv, topi = decoder_output.topk(1)            decoder_input = topi.squeeze().detach()  # detach from history as input            loss += criterion(decoder_output, target_tensor[di])            if decoder_input.item() == EOS_token:                break
For our text summarization problem, there can be multiple correct answers and as we do not have a single correct output, we can evaluate our model using different parameters like — Recall, Precision, F-score. Below are some of the metrics that we will use:
BLEU(Bilingual Evaluation Understudy): The cornerstone of this metric is precision having values between [0,1], wherein 1 represents a perfect match and 0 represents a complete mismatch. This metric is basically calculated by comparing the number of machine-generated words that are a part of the reference sentence with respect to the total number of words in the machine-generated output. Let’s understand this with the help of an example:
Reference Sentence: The door is lockedMachine Output: the the theBLEU Score=1
The above machine output is an extreme condition, however, in order to overcome this problem, the BLEU score is calculated in a way that each word in the generated sentence will be clipped to the number of times, that word occurs in the reference sentence. This basically makes sure that if one word is generated more number of times than it is present in the reference sentence– it is not considered while evaluating the similarity.After applying the above rule, we get the modified BLEU Score as 1/3
Let’s take a look at another extreme example:
Reference Sentence: The door is lockedMachine Output: theBLEU Score: 1 (even after applying the above rule)
In such a scenario, where the length of the generated sentence is less than the reference sentence, a Brevity penalty(BP) is introduced, i.e. there is a penalty to the BLEU score, if the generated sentence is smaller than the reference sentence, however, when the generated sentence is longer than the reference sentence no penalty is introduced.
Brevity Penalty is defined as-
where, r is the effective reference corpus lengthc is the length of generated/candidate sentence
This metric was first proposed by Kishore Papineni in 2002. Refer the below link for more details around this metric: BLEU: a Method for Automatic Evaluation of Machine Translation
ROUGE (Recall-Oriented Understanding for Gisting Evaluation):
ROUGE is basically a recall-oriented measure that works by comparing the number of machine-generated words that are a part of the reference sentence with respect to the total number of words in the reference sentence.
This metric is more intuitive in the sense that every time we add a reference to the pool, we expand our space of alternating summaries. Hence, this metric should be preferred when we have multiple references.
ROUGE implementation is pretty similar to BELU, however, there are other underlying implementations like LCS (Longest common subsequence) and skip-gram, etc. We can directly use the ROUGE-N implementation using the python library ROUGE.
For more details, you can refer to the below paper: ROUGE: A Package for Automatic Evaluation of Summaries
We have seen a precision-based metric(BLEU) and a recall oriented metric(ROUGE), however, if we want our evaluation on the basis of both recall and precision, we can use F-Score as an evaluation measure.
Result: The implementation code can be found on my Github.
The model ran on Google Colab Pro(T4 & P100 GPU - 25GB with high memory VMs) for ~6–7 hours and it seemed to work fine on shorter summaries (~50 words). However, the model can be optimized by further tuning the hyperparameters (learning rate, optimizer, loss function, hidden layers, momentum, iterations, etc.)
Bi-directional/stacked GRU cells can be used for improving performanceImplementing different kinds of attention-mechanism — multiplicative attention, multi-head attention, etc.Output words can be selected using beam search instead of greedy-algorithm
Bi-directional/stacked GRU cells can be used for improving performance
Implementing different kinds of attention-mechanism — multiplicative attention, multi-head attention, etc.
Output words can be selected using beam search instead of greedy-algorithm
Attention is all you needhttps://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.htmlhttps://www.aclweb.org/anthology/W04-1013.pdf
Attention is all you need