Automatically Summarize Trump’s State of the Union Address | by Susan Li | Towards Data Science

Automatic text summarization, is the process of creating a short, concise and coherent version of a longer document. It is one of the most interesting and challenging problems in the field of NLP.
Since Trump delivered his State of the Union address last night, “Key takeaways”, “Fact check”, “Analysis”, “Reactions” are all over the news media. If you are like me, don’t want to listen through the entire speech for 82 minutes, or read the entire address, and you do not want to miss any important stuff, then we will try to explore the realms of text summarization and build a text summarizer. Hopefully, the summary we get from our text summerizer is as close to the original speech as possible, but much shorter. Let’s get started!
TextRank is an unsupervised text summarization technique that uses the intuition behind the PageRank algorithm to rank sentences. When using NLTK for our project, we have the following steps:
Fetch the State of Union Address on the internet.
Basic text cleaning.
Find vector representation (word embeddings) for each and every sentence.
Similarities between sentence vectors are then calculated and stored in a matrix.
The similarity matrix is converted into a graph, with sentences as vertices and similarity scores as edges.
Apply the PageRank algorithm over this sentence graph for sentence rank calculation.
Print out several top ranked sentences.
The data can be found on The White House website and it was issued today.
def get_only_text(url):    """     return the title and the text of the article    at the specified url    """    page = urlopen(url)    soup = BeautifulSoup(page, "lxml")    text = ' '.join(map(lambda p: p.text, soup.find_all('p')))      print ("=====================")    print (text)    print ("=====================")     return soup.title.text, text          url="https://www.whitehouse.gov/briefings-statements/remarks-president-trump-state-union-address-2/"text = get_only_text(url)
First, we have a peek a few sentences:
sentences = []for s in text:    sentences.append(sent_tokenize(s))sentences = [y for x in sentences for y in x]sentences[30:40]
Sounds about right.
We will use pre-trained word vectors to create vectors for the sentences in the State of Union address. I have downloaded the data from Glove and saved in my working directory.
word_embeddings = {}f = open('glove.6B.100d.txt', encoding='utf-8')for line in f:    values = line.split()    word = values[0]    coefs = np.asarray(values[1:], dtype='float32')    word_embeddings[word] = coefsf.close()
Some basic text preprocessing, such as remove stop words and remove special characters.
clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")clean_sentences = [s.lower() for s in clean_sentences]stop_words = stopwords.words('english')def remove_stopwords(sen):    sen_new = " ".join([i for i in sen if i not in stop_words])    return sen_newclean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]
In the following code scripts, we create vectors for the sentences. We first fetch vectors (each of size 100 elements) for the constituent words in a sentence and then take average of those vectors to arrive at a consolidated vector for the sentence. And we create an empty similarity matrix, and populate it with cosine similarities of the sentences. Finally we initialize the matrix with cosine similarity scores and print out top 15 ranked sentences as the summary representation.
Sumy is a Python library for extracting summary from HTML pages or plain texts. It was developed by Miso-Belica. We will apply the following summarization methods to the State of Union Address, and print out 10 sentences for each methods:
LsaSummarizer. Latent Semantic Analysis which combines term frequency with singular value decomposition.
LANGUAGE = "english"SENTENCES_COUNT = 10url="https://www.whitehouse.gov/briefings-statements/remarks-president-trump-state-union-address-2/"parser = HtmlParser.from_url(url, Tokenizer(LANGUAGE))print ("--LsaSummarizer--")    summarizer = LsaSummarizer()summarizer = LsaSummarizer(Stemmer(LANGUAGE))summarizer.stop_words = get_stop_words(LANGUAGE)for sentence in summarizer(parser.document, SENTENCES_COUNT):    print(sentence)
LuhnSummarizer. A naive approach based on TF-IDF and looking at the “window size” of non-important words between words of high importance. It also assigns higher weights to sentences occurring near the beginning of a document.
print ("--LuhnSummarizer--")     summarizer = LuhnSummarizer() summarizer = LuhnSummarizer(Stemmer(LANGUAGE))summarizer.stop_words = ("I", "am", "the", "you", "are", "me", "is", "than", "that", "this")for sentence in summarizer(parser.document, SENTENCES_COUNT):    print(sentence)
LexRankSummarizer. Unsupervised approach inspired by algorithms PageRank. It finds the relative importance of all words in a document and selects the sentences which contain the most of those high-scoring words.
print ("--LexRankSummarizer--")   summarizer = LexRankSummarizer()summarizer = LexRankSummarizer(Stemmer(LANGUAGE))summarizer.stop_words = ("I", "am", "the", "you", "are", "me", "is", "than", "that", "this")for sentence in summarizer(parser.document, SENTENCES_COUNT):    print(sentence)
EdmundsonSummarizer. When using EdmundsonSummarizer, we need to enter bonus_words which are the words we want to see in summary and are significant, stigma_words which are unimportant, null_words which are stop words.
print ("--EdmundsonSummarizer--")     summarizer = EdmundsonSummarizer() words1 = ("economy", "fight", "trade", "china")summarizer.bonus_words = words1     words2 = ("another", "and", "some", "next")summarizer.stigma_words = words2    words3 = ("another", "and", "some", "next")summarizer.null_words = words3for sentence in summarizer(parser.document, SENTENCES_COUNT):    print(sentence)
This is EdmundsonSummarizer outputs after I set the above words criteria:
Seems there are a lot of parameters to tweak before we determine which method/methods are the best for summarizing Trump’s State of Union Address. Anyway, I personally enjoyed this learning journey. Hope you too.
Jupyter notebook can be found on Github. Enjoy the rest of the week!