Abstractive Summarization Using Pytorch | by Raymond Cheng | Towards Data Science

Abstractive Summarization is a task in Natural Language Processing (NLP) that aims to generate a concise summary of a source text. Unlike extractive summarization, abstractive summarization does not simply copy important phrases from the source text but also potentially come up with new phrases that are relevant, which can be seen as paraphrasing. Abstractive summarization yields a number of applications in different domains, from books and literature, to science and R&D, to financial research and legal documents analysis.
To date, the most recent and effective approach toward abstractive summarization is using transformer models fine-tuned specifically on a summarization dataset. In this article, we demonstrate how you can easily summarize a text using a powerful model within a few simple steps. The model that we are going to be using is already pretrained, so no additional training is required :)
So without further ado, let’s get started!
The library we are going to be using is Transformers by Huggingface. If you are not familiar with Transformers, you can go ahead and read my previous article on what it can do:
towardsdatascience.com
To install transformers, you can simply run:
pip install transformers
Note: Transformers requires Pytorch to be installed beforehand. If you don’t have Pytorch installed yet, head to the Pytorch official website and follow its instructions to install it.
After successfully installing transformers, you can now start importing it to your Python script. We can also import os in order to set the environment variable for the GPU to use in the next step. Note that this is completely optional but a good practice to have to prevent overflowing to other GPUs if you have more than one GPUs (if you are using a Jupyter notebook).
from transformers import pipelineimport os
If you decided to set GPU (e.g. 0), then you can set it as shown below:
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
Now, we are ready to select the summarization model to use. Huggingface provides two powerful summarization models to use: BART (bart-large-cnn) and t5 (t5-small, t5-base, t5-large, t5–3b, t5–11b). You can read more about them in their official papers (BART paper, t5 paper).
To use the BART model, which is trained on the CNN/Daily Mail News Dataset, you can directly use the default parameters via Huggingface’s inbuilt pipeline module:
summarizer = pipeline("summarization")
If you want to use the t5 model (e.g. t5-base), which is trained on the c4 Common Crawl web corpus, then you can do this:
summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf")
You can refer to the Huggingface documentation for more information.
Now, after we have our model ready, we can start inputting the text we want to summarize. Imagine, we want to summarize the following text about COVID-19 vaccine taken from an article from MedicineNet:
One month after the United States began what has become a troubled rollout of a national COVID vaccination campaign, the effort is finally gathering real steam.
Close to a million doses — over 951,000, to be more exact — made their way into the arms of Americans in the past 24 hours, the U.S. Centers for Disease Control and Prevention reported Wednesday. That’s the largest number of shots given in one day since the rollout began and a big jump from the previous day, when just under 340,000 doses were given, CBS News reported.
That number is likely to jump quickly after the federal government on Tuesday gave states the OK to vaccinate anyone over 65 and said it would release all the doses of vaccine it has available for distribution. Meanwhile, a number of states have now opened mass vaccination sites in an effort to get larger numbers of people inoculated, CBS News reported.
We define our variable:
text = """One month after the United States began what has become a troubled rollout of a national COVID vaccination campaign, the effort is finally gathering real steam.Close to a million doses -- over 951,000, to be more exact -- made their way into the arms of Americans in the past 24 hours, the U.S. Centers for Disease Control and Prevention reported Wednesday. That's the largest number of shots given in one day since the rollout began and a big jump from the previous day, when just under 340,000 doses were given, CBS News reported.That number is likely to jump quickly after the federal government on Tuesday gave states the OK to vaccinate anyone over 65 and said it would release all the doses of vaccine it has available for distribution. Meanwhile, a number of states have now opened mass vaccination sites in an effort to get larger numbers of people inoculated, CBS News reported."""
Finally, we can start summarizing the inputted text. Here, we declare the min_length and the max_length we want the summarization output to be, and also turn off sampling to generate a fixed summary. We can do so by running the following:
summary_text = summarizer(text, max_length=100, min_length=5, do_sample=False)[0]['summary_text']print(summary_text)
Voila! We get our summary text:
Over 951,000 doses of vaccine given in one day in the past 24 hours, CDC says . That’s the largest number of shots given in a month since the rollout began . The federal government gave states the OK to vaccinate anyone over 65 on Tuesday . A number of states have now opened mass vaccination sites in an effort to get more people inoculated, CBS News reports .
As shown from the summarized text, we can see that the model knows that 24 hours is equivalent to a day and is smart enough to abbreviate the U.S. Centers for Disease Control and Prevention to CDC. In addition, the model successfully links information from the first and second paragraphs by indicating that it is the largest number of shots given since the rollout that began last month. We can see that the performance of this summarization model is quite decent.
Congratulations, you now know how to perform abstractive summarization on any piece of text! To put it all together, here’s the entire code in the form of a Jupyter notebook:
from transformers import pipeline
import os

## Setting to use the 0th GPU
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

## Setting to use the bart-large-cnn model for summarization
summarizer = pipeline("summarization")

## To use the t5-base model for summarization:
## summarizer = pipeline("summarization", model="t5-base", tokenizer="t5-base", framework="tf")

text = """One month after the United States began what has become a troubled rollout of a national COVID vaccination campaign, the effort is finally gathering real steam.
Close to a million doses -- over 951,000, to be more exact -- made their way into the arms of Americans in the past 24 hours, the U.S. Centers for Disease Control and Prevention reported Wednesday. That's the largest number of shots given in one day since the rollout began and a big jump from the previous day, when just under 340,000 doses were given, CBS News reported.
That number is likely to jump quickly after the federal government on Tuesday gave states the OK to vaccinate anyone over 65 and said it would release all the doses of vaccine it has available for distribution. Meanwhile, a number of states have now opened mass vaccination sites in an effort to get larger numbers of people inoculated, CBS News reported."""

summary_text = summarizer(text, max_length=100, min_length=5, do_sample=False)[0]['summary_text']
print(summary_text)

 Over 951,000 doses of vaccine given in one day in the past 24 hours, CDC says . That's the largest number of shots given in a month since the rollout began . The federal government gave states the OK to vaccinate anyone over 65 on Tuesday . A number of states have now opened mass vaccination sites in an effort to get more people inoculated, CBS News reports .

And that’s all! Hope you enjoyed this post. If you like my work, feel free to take a look at my other articles, and stay tuned for more to come :)
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1] Abstractive Text Summarization, Papers With Code
[2] 20 Applications of Automatic Summarization in the Enterprise, Fraise
[3] Transformers Github, Huggingface
[4] Transformers Documentation, Huggingface
[5] Pytorch Official Website, Facebook AI Research
[6] Lewis, Mike, et al. “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.” arXiv preprint arXiv:1910.13461 (2019).
[7] Raffel, Colin, et al. “Exploring the limits of transfer learning with a unified text-to-text transformer.” arXiv preprint arXiv:1910.10683 (2019).
[8] Tensorflow Datasets, Google
[9] U.S. COVID Vaccine Rollout Nears 1 Million a Day, MedicineNet