Text Summarization with GloVe Embeddings.. | by Sayak Misra | Towards Data Science

This story is a continuation to our previous blog post, where we have discussed the basics of text summarization, the various approaches and how we implemented an encoder-decoder model(with attention) to solve the problem in hand.
Just to recap, text summarization is a process of generating a concise and meaningful summary of text from multiple text resources such as books, news articles, blog posts, research papers, emails, and tweets.
We have seen an encoder-decoder(seqtoseq) model is a perfect choice for summarization tasks, so we will continue with that architecture. Here in addition to that we will be using GloVe pre-trained word embeddings to give our model a head-start, and check if it really performs better in understanding the language semantics and in turn summarizing.
“Word embeddings” are a family of natural language processing techniques aiming at mapping semantic meaning into a geometric space. This is done by associating a numeric vector to every word in a dictionary, such that the distance (e.g. L2 distance or more commonly cosine distance) between any two vectors would capture part of the semantic relationship between the two associated words. The geometric space formed by these vectors is called an embedding space.
For instance, “coconut” and “polar bear” are words that are semantically quite different, so a reasonable embedding space would represent them as vectors that would be very far apart. But “kitchen” and “dinner” are related words, so they should be embedded close to each other.
Ideally, in a good embeddings space, the “path” (a vector) to go from “kitchen” and “dinner” would capture precisely the semantic relationship between these two concepts. In this case the relationship is “where x occurs”, so you would expect the vector kitchen - dinner (difference of the two embedding vectors, i.e. path to go from dinner to kitchen) to capture this "where x occurs" relationship. Basically, we should have the vectorial identity: dinner + (where x occurs) = kitchen (at least approximately). If that's indeed the case, then we can use such a relationship vector to answer questions. For instance, starting from a new vector, e.g. "work", and applying this relationship vector, we should get sometime meaningful, e.g. work + (where x occurs) = office, answering "where does work occur?".
Word embeddings are computed by applying dimensionality reduction techniques to datasets of co-occurence statistics between words in a corpus of text. This can be done via neural networks (the “word2vec” technique), or via matrix factorization.
We can play with this beautiful Tensorflow projector, to get a better understanding of word embeddings.
The two of the most common word embeddings are: Word2Vec and GloVe, and both of them are equally popular. But GloVe(“Global Vectors for Word Representation”) as the name suggests is better for preserving the global contexts as it creates a global co-occurrence matrix by estimating the probability a given word will co-occur with other words. Here for summarization the global context is a necessity, so we are moving ahead with GloVe but in most of the use-cases there is very little between the two to choose from.
Specifically, we will use the 100-dimensional GloVe embeddings of 400k words computed on a 2014 dump of English Wikipedia. You can download them here (warning: following this link will start a 822MB download).
Here is the complete code of the LSTM Encoder-Decoder model with Attention and GloVe embeddings added to it.
We are not going through the details of the model architecture, as we have discussed it in our previous blog-post and in turn focusing on adding the GloVe embeddings to it and assessing the performance.
First let’s download and unzip the GloVe embeddings.
!wget 'http://nlp.stanford.edu/data/glove.6B.zip'!unzip '/content/glove.6B.zip'
Next, we compute an index mapping words to known embeddings, by parsing the data dump of pre-trained embeddings:
embeddings_index = {}f = open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt'))for line in f:    values = line.split()    word = values[0]    coefs = np.asarray(values[1:], dtype='float32')    embeddings_index[word] = coefsf.close()print('Found %s word vectors.' % len(embeddings_index))
At this point we can leverage our embedding_index dictionary and our word_index to compute our embedding matrix:
EMBEDDING_DIM = 100embedding_matrix = np.zeros((len(word_index) + 1, EMBEDDING_DIM))for word, i in word_index.items():    embedding_vector = embeddings_index.get(word)    if embedding_vector is not None:        # words not found in embedding index will be all-zeros.        embedding_matrix[i] = embedding_vector
We load this embedding matrix into an Embedding layer. Note that we set trainable=False to prevent the weights from being updated during training. Previously we used the default Embedding layer, withtrainable=True , which learned the Embeddings through the training process.
embedding_layer = Embedding(len(word_index) + 1,                            EMBEDDING_DIM,                            weights=[embedding_matrix],                            input_length=MAX_SEQUENCE_LENGTH,                            trainable=False)
Here, we are using the 100 dimension GloVe embeddings and the embeddings are saved in glove.6B.100d.txt.
An Embedding layer should be fed sequences of integers, i.e. a 2D input of shape (samples, indices). These input sequences should be padded so that they all have the same length in a batch of input data (although an Embedding layer is capable of processing sequence of heterogenous length, if you don't pass an explicit input_length argument to the layer).
All that the Embedding layer does is to map the integer inputs to the vectors found at the corresponding index in the embedding matrix, i.e. the sequence [1, 2] would be converted to [embeddings[1], embeddings[2]]. This means that the output of the Embedding layer will be a 3D tensor of shape (samples, sequence_length, embedding_dim).
We get a validation loss of around 2.25 which is similar to what we got without the GloVe embeddings.
Review: tried several tassimo discs gevalia best coffee ever tasted coffee snob love coffee  Original summary: great coffee  Predicted summary:  best coffee everReview: almonds unbelievable amount salt love salt top bought brand stores never salty maybe machine went wacko processing cannot eat unless take time try wipe salt talks forever  Original summary: too salty to eat  Predicted summary:  not the bestReview: sure much nourishment hair getting shampoo dandruff nice job cleaning hair leaving tangle free somewhat conditioned using pantene pro shampoo compares favorably pantene like way bottle made easy open dispensing far like way hair coming shampooing seem little better started used bottle seen little improvement really much improvement would say shampoo would worth trying nice job cleaning smells nice bottle easy use relatively inexpensive keep mind make many shampoos one person likes another hate  Original summary: seems like an excellent shampoo  Predicted summary:  great for hair
As we can see from the above examples, it performs quite well on most of the use-cases.
Here is the colab-notebook of the complete implementation.
To keep it short, NO.
Though this model might produce better summaries for some cases but it can not be affirmed that it has better results across all the examples. The main reason behind this is we have enough records here(100000 to be exact) for our Embedding layer in Encoder-Decoder model to learn the semantics of the language, so it performed quite well even without the pre-trained embeddings.
Pre-trained models (like GloVe here) can be used with great improvements in results if we have a short dataset. If the number of records in the dataset are less, the Embedding layer won’t be able to generate proper embeddings of it’s own and in those cases using pre-trained embeddings will enhance the performance accuracy.
Here we have seen how we can add pre-trained embeddings to our existing LSTM Encoder-Decoder architecture, though the results didn’t spike up much, but it can perform brilliantly for a smaller dataset.
We can further improve the summarising process by a huge margin if we can leverage the true essence of Transfer Learning, by using state-of-the pre-trained language models like: BERTSum, BART, T5. In our next story we shall look into the details of those model and how wonderfully they can summarize texts.