PEGASUS Simple Abstractive Summarization using 🤗 Hugging Face & Python
By: Parth Chokhra
Tags: Programming, Artificial Intelligence, Data Science, Technology, Machine Learning

PEGASUS Simple Abstractive Summarization using 🤗 Hugging Face & Python

Demo for State-of-the-art Abstractive Summarization with Google’s PEGASUS

Photo by Mati Mango from Pexels

Google AI recently released PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization which achieved state-of-the-art results on 12 diverse summarization datasets.

Since the released of BERT, pretrained Transformers with self-supervised objectives on large text corpora is the go-to way to solve nearly every NLP task. Also, it has been found that the closer the pre-training self-supervised objective is to the final down-stream task, the better the fine-tuning performance. Therefore in PEGASUS important sentences are masked from the input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary.

To read more you can check the Google AI blog.😁( It’s an amazing blog). Below is a python demo using Huggigface Library.

Install Library

!pip install transformers==3.5.0

!pip install torch==1.7.0

Text input:

About 749 words input text from Wikipedia Titanic Page. I have taken the first four paragraphs as input.

Text Output

The sinking of the passenger liner Titanic 100 years ago on 15 April 1912 remains one of the world's worst maritime disasters, one of the most infamous tragedies in human history, and one of the most famous tragedies in the history of the film and theatre world, with a reputation as one of the greatest disasters in the history of the entertainment industry, with more than 1,500 people lost when the Titanic hit an iceberg on its maiden voyage from England to the United States, but also more than 200 people who died when the ship capsized off the coast of Newfoundland, Canada.

The code :

from transformers import PegasusTokenizer,PegasusForConditionalGeneration mname = "google/pegasus-xsum"

titanic_text = '''RMS Titanic was a British passenger liner operated by the White Star Line that sank in the North Atlantic Ocean in the early morning hours of 15 April 1912, after striking an iceberg during her maiden voyage from Southampton to New York City. Of the estimated 2,224 passengers and crew aboard, more than 1,500 died, making the sinking at the time the deadliest of a single ship in the West and the deadliest peacetime sinking of a superliner or cruise ship to date.With much public attention in the aftermath the disaster has since been the material of many artistic works and a founding material of the disaster film genre.RMS Titanic was the largest ship afloat at the time she entered service and was the second of three Olympic-class ocean liners operated by the White Star Line. She was built by the Harland and Wolff shipyard in Belfast. Thomas Andrews, chief naval architect of the shipyard at the time, died in the disaster.Titanic was under the command of Captain Edward Smith, who also went down with the ship. The ocean liner carried some of the wealthiest people in the world, as well as hundreds of emigrants from Great Britain and Ireland, Scandinavia and elsewhere throughout Europe, who were seeking a new life in the United States. The first-class accommodation was designed to be the pinnacle of comfort and luxury, with a gymnasium, swimming pool, libraries, high-class restaurants, and opulent cabins. A high-powered radiotelegraph transmitter was available for sending passenger "marconigrams" and for the ship's operational use.The Titanic had advanced safety features, such as watertight compartments and remotely activated watertight doors. The ship carried 16 lifeboat davits which could lower three lifeboats each, for a total of 48 boats. However, Titanic carried only a total of 20 lifeboats, four of which were collapsible and proved hard to launch during the sinking.The carried lifeboats were enough for 1,178 people—about half the number on board, and one third of her total capacity—due to the maritime safety regulations of those days. Though at the time of the sinking the lowered lifeboats were only filled about half.After leaving Southampton on 10 April 1912, Titanic called at Cherbourg in France and Queenstown (now Cobh) in Ireland, before heading west to New York.On 14 April, four days into the crossing and about 375 miles (600 km) south of Newfoundland, she hit an iceberg at 11:40 p.m. ship's time. The collision caused the hull plates to buckle inwards along her starboard (right) side and opened five of her sixteen watertight compartments to the sea; she could only survive four flooding. Meanwhile, passengers and some crew members were evacuated in lifeboats, many of which were launched only partially loaded. A disproportionate number of men were left aboard because of a "women and children first" protocol for loading lifeboats.At 2:20 a.m., she broke apart and foundered with well over one thousand people still aboard. Just under two hours after Titanic sank, the Cunard liner RMS Carpathia arrived and brought aboard an estimated 705 survivors.The disaster was met with worldwide shock and outrage at the huge loss of life, as well as the regulatory and operational failures that led to it. Public inquiries in Britain and the United States led to major improvements in maritime safety. One of their most important legacies was the establishment of the International Convention for the Safety of Life at Sea (SOLAS) in 1914, which still governs maritime safety. Several new wireless regulations were passed around the world in an effort to learn from the many missteps in wireless communications—which could have saved many more passengers.The wreck of Titanic was discovered in 1985 (73 years after the disaster) during a Franco-American expedition and United States Military mission.The ship was split in two and is gradually disintegrating at a depth of 12,415 feet (2,069.2 fathoms; 3,784 m). Thousands of artefacts have been recovered and displayed at museums around the world. Titanic has become one of the most famous ships in history, depicted in numerous works of popular culture, including books, folk songs, films, exhibits, and memorials. Titanic is the second largest ocean liner wreck in the world, only being surpassed by her sister ship HMHS Britannic, however, she is the largest sunk while in service as a liner, as Britannic was in use as a hospital ship at the time of her sinking. The final survivor of the sinking, Millvina Dean, aged two months at the time, died in 2009 at the age of 97.''' # download model

model = PegasusForConditionalGeneration.from_pretrained(mname) # download tokenizer

tok = PegasusTokenizer.from_pretrained(mname) batch = tok.prepare_seq2seq_batch(src_texts=[titanic_text]) # Hyperparameter Tuning gen = model.generate(

**batch,max_length = 200, # max length of summary

min_length = 100, # min length of summary

do_sample = True,

temperature = 3.0,

top_k =30,

top_p=0.70,

repetition_penalty = 1.2,

length_penalty = 5, # if more than 1 encourage model to generate #larger sequences

num_return_sequences=1 # no of summary you want to generate) # for forward pass: model(**batch)

summary = tok.batch_decode(gen, skip_special_tokens=True) print(summary)

The output produced is

The sinking of the passenger liner Titanic 100 years ago on 15 April 1912 remains one of the world's worst maritime disasters, one of the most infamous tragedies in human history, and one of the most famous tragedies in the history of the film and theatre world, with a reputation as one of the greatest disasters in the history of the entertainment industry, with more than 1,500 people lost when the Titanic hit an iceberg on its maiden voyage from England to the United States, but also more than 200 people who died when the ship capsized off the coast of Newfoundland, Canada.

References