Simple abstractive text summarization with pretrained T5 — Text-To-Text Transfer Transformer | by Ramsri Goutham | Towards Data Science

T5 is a new transformer model from Google that is trained in an end-to-end manner with text as input and modified text as output. You can read more about it here.
It achieves state-of-the-art results on multiple NLP tasks like summarization, question answering, machine translation etc using a text-to-text transformer trained on a large text corpus.
Today we will see how we can use huggingface’s transformers library to summarize any given text. T5 is an abstractive summarization algorithm. It means that it will rewrite sentences when necessary than just picking up sentences directly from the original text.
Install these libraries in your jupyter notebook or conda environment before you begin :
!pip install transformers==2.8.0!pip install torch==1.4.0
The US has "passed the peak" on new coronavirus cases, President Donald Trump said and predicted that some states would reopen this month.The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.At the daily White House coronavirus briefing on Wednesday, Trump said new guidelines to reopen the country would be announced on Thursday after he speaks to governors."We'll be the comeback kids, all of us," he said. "We want to get our country back."The Trump administration has previously fixed May 1 as a possible date to reopen the world's largest economy, but the president said some states may be able to return to normalcy earlier than that.
The us has over 637,000 confirmed Covid-19 cases and over 30,826 deaths. President Donald Trump predicts some states will reopen the country in april, he said. "we'll be the comeback kids, all of us," the president says.
If you see the algorithm has intelligently summarized with mentioning April although it was never mentioned in the original story. It also replaced he with the president. Cut short and removed the extra information after the comma in the sentence “over 30,826 deaths ....”
The same code inline as some users have reported that the Github Gist wasn’t loading :
import torchimport json from transformers import T5Tokenizer, T5ForConditionalGeneration, T5Configmodel = T5ForConditionalGeneration.from_pretrained('t5-small')tokenizer = T5Tokenizer.from_pretrained('t5-small')device = torch.device('cpu')text ="""The US has "passed the peak" on new coronavirus cases, President Donald Trump said and predicted that some states would reopen this month.The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.At the daily White House coronavirus briefing on Wednesday, Trump said new guidelines to reopen the country would be announced on Thursday after he speaks to governors."We'll be the comeback kids, all of us," he said. "We want to get our country back."The Trump administration has previously fixed May 1 as a possible date to reopen the world's largest economy, but the president said some states may be able to return to normalcy earlier than that."""preprocess_text = text.strip().replace("\n","")t5_prepared_Text = "summarize: "+preprocess_textprint ("original text preprocessed: \n", preprocess_text)tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors="pt").to(device)# summmarize summary_ids = model.generate(tokenized_text,                                    num_beams=4,                                    no_repeat_ngram_size=2,                                    min_length=30,                                    max_length=100,                                    early_stopping=True)output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)print ("\n\nSummarized text: \n",output)# Summarized output from above ::::::::::# the us has over 637,000 confirmed Covid-19 cases and over 30,826 deaths. # president Donald Trump predicts some states will reopen the country in april, he said. # "we'll be the comeback kids, all of us," the president says.
The summarized output as mentioned above is -
The us has over 637,000 confirmed Covid-19 cases and over 30,826 deaths. President Donald Trump predicts some states will reopen the country in april, he said. "we'll be the comeback kids, all of us," the president says.
The key point to note in the code above is that we prepend our text with “summarize: ” before passing it to the T5 summarizer.
If you look in the paper, all we need to do for other tasks is to prepend our text with their corresponding string eg: “translate English to German: ” for translation task.
I am running a 4-week cohort-based course on “Practical Introduction to NLP” with Maven, the world’s best platform for cohort-based learning. If you wish to transform yourself from a Python developer to a junior NLP developer with practical project experience in 4 weeks, grab your spot now!
I launched a very interesting Udemy course titled “Question generation using NLP” expanding on some of the techniques discussed in this blog post. If you would like to take a look at it, here is the link.