How to Create Simple News Summarization from Scratch using Python | by Fahmi Nurfikri | Towards Data Science

If you are new to the world of machine learning, specifically Natural Language Processing (NLP) and you hear about Automatic News Summarization and you are interested in it. You must be wondering, like “How to make a good model?”, “What should I learn?” and “Where should I start?”.
Then you start searching for what methods are good for automatic summaries and find methods such as Bart, then you try the pre-trained model, and impressed about its result. Then you are interested in training the model for your native language. But there’s a problem, you live in 3rd words country and no pre-trained models are available or you don’t have any supercomputer to run or you don’t even find a dataset for your language and you’re too lazy to labelled it alone.
So in this post, I’ll give a simple solutions for you to make your simple news summarization. First, let’s talk about approaches in news summarization.
In general, the news summarization approach is divided into two, namely extractive and abstractive. Extractive summarization means identifying important parts of the text and making it verbatim produce a subset of sentences from the original text; while abstractive summarization reproduces important material in new ways after the interpretation and examination of texts using advanced natural language techniques to produce new, shorter texts that convey the most critical information from the original.
Since our initial goal is to make a simple summarizer, here we will use the Extractive summarization approach. Let’s get started, if you want to see the full code of this article, please visit my github.
First, import package that will be used
import numpy as npimport nltkimport re
Here we use the NumPy, nltk, and re libraries. NumPy is a library for numerical calculations, nltk is a library that is widely used for NLP, and re is a library for regular expressions.
After that, we define the preprocessing functions that will be performed. It aims to concentrate words and eliminate characters that are not needed. Here there are 3 preprocessing functions, namely:
Case folding: change all letters to lowercase
Cleaning: removes all punctuation and numbers, leaving only the alphabet characters
Tokenization: converts sentences into tokens
Here is the code for the preprocessing process
def casefolding(sentence):    return sentence.lower()def cleaning(sentence):    return re.sub(r'[^a-z]', ' ', re.sub("’", '', sentence))def tokenization(sentence):    return sentence.split()
In addition, you can also add other functions such as stopwords removal, stemming, or lemmatization. However, because the usage can be different for each language, so the function will not be explained in this article, but an example is in the Github link that I have provided.
Next, we also need a function to turn the whole story into a collection of sentences
def sentence_split(paragraph):    return nltk.sent_tokenize(paragraph)
Next we will define a function to count the number of each word in the document. This process is carried out to give weight to words that aim to determine whether the word has an effect or not. Here is the code for the process
def word_freq(data):    w = []    for sentence in data:        for words in sentence:            w.append(words)    bag = list(set(w))    res = {}    for word in bag:        res[word] = w.count(word)    return res
Then, we will make a function to calculate the weight of each sentence. This process is carried out to determine which sentence is considered to best represent the whole story.
def sentence_weight(data):    weights = []    for words in data:        temp = 0        for word in words:            temp += wordfreq[word]        weights.append(temp)    return weights
Now, all the functions we will use are defined. Let’s take a case example. For example, I will use an article from Politico entitled “The New York Times Surrendered to an Outrage Mob. Journalism Will Suffer For It“.
We will start by inserting news on the site into a variable by doing copy-paste manually. If you don’t want to do copy-paste and are interested in web scraping, you can read my previous article titled Web Scraping News with 4 lines using Python.
towardsdatascience.com
news = """IIn a time in which even a virus has become the subject of partisan disinformation and myth-making, it’s essential that mainstream journalistic institutions reaffirm their bona fides as disinterested purveyors of fact and honest brokers of controversy. In this regard, a recent course of action by the New York Times is cause for alarm.On December 27, 2019, the Times published a column by their opinion journalist Bret Stephens, “The Secrets of Jewish Genius,” and the ensuing controversy led to an extraordinary response by the editors.Stephens took up the question of why Ashkenazi Jews are statistically overrepresented in intellectual and creative fields. This disparity has been documented for many years, such as in the 1995 book Jews and the New American Scene by the eminent sociologists Seymour Martin Lipset and Earl Raab. In his Times column, Stephens cited statistics from a more recent peer-reviewed academic paper, coauthored by an elected member of the National Academy of Sciences. Though the authors of that paper advanced a genetic hypothesis for the overrepresentation, arguing that Ashkenazi Jews have the highest average IQ of any ethnic group because of inherited traits, Stephens did not take up that argument. In fact, his essay quickly set it aside and argued that the real roots of Jewish achievement are culturally and historically engendered habits of mind.Nonetheless, the column incited a furious and ad hominem response. Detractors discovered that one of the authors of the paper Stephens had cited went on to express racist views, and falsely claimed that Stephens himself had advanced ideas that were “genetic” (he did not), “racist” (he made no remarks about any race) and “eugenicist” (alluding to the discredited political movement to improve the human species by selective breeding, which was not remotely related to anything Stephens wrote).It would have been appropriate for the New York Times to acknowledge the controversy, to publish one or more replies, and to allow Stephens and his critics to clarify the issues. Instead, the editors deleted parts of the column—not because anything in it had been shown to be factually incorrect but because it had become controversial.Worse, the explanation for the deletions in the Editors’ Note was not accurate about the edits the paper made after publication. The editors did not just remove “reference to the study.” They expurgated the article’s original subtitle (which explicitly stated “It’s not about having higher IQs”), two mentions of Jewish IQs, and a list of statistics about Jewish accomplishment: “During the 20th century, [Ashkenazi Jews] made up about 3 percent of the U.S. population but won 27 percent of the U.S. Nobel science prizes and 25 percent of the ACM Turing awards. They account for more than half of world chess champions.” These statistics about Jewish accomplishments were quoted directly from the study, but they originated in other studies. So, even if the Times editors wanted to disavow the paper Stephens referenced, the newspaper could have replaced the passage with quotes from the original sources.The Times’ handling of this column sets three pernicious precedents for American journalism.First, while we cannot know what drove the editors’ decision, the outward appearance is that they surrendered to an outrage mob, in the process giving an imprimatur of legitimacy to the false and ad hominem attacks against Stephens. The Editors’ Note explains that Stephens “was not endorsing the study or its authors’ views,” and that it was not his intent to “leave an impression with many readers that [he] was arguing that Jews are genetically superior.” The combination of the explanation and the post-publication revision implied that such an impression was reasonable. It was not.Unless the Times reverses course, we can expect to see more such mobs, more retractions, and also preemptive rejections from editors fearful of having to make such retractions. Newspapers risk forfeiting decisions to air controversial or unorthodox ideas to outrage mobs, which are driven by the passions of their most ideological police rather than the health of the intellectual commons.Second, the Times redacted a published essay based on concerns about retroactive moral pollution, not about accuracy. While it is true that an author of the paper Stephens mentioned, the late anthropologist Henry Harpending, made some deplorable racist remarks, that does not mean that every point in every paper he ever coauthored must be deemed radioactive. Facts and arguments must be evaluated on their content. Will the Times and other newspapers now monitor the speech of scientists and scholars and censor articles that cite any of them who, years later, say something offensive? Will it crowdsource that job to Twitter and then redact its online editions whenever anyone quoted in the Times is later “canceled”?Third, for the Times to “disappear” passages of a published article into an inaccessible memory hole is an Orwellian act that, thanks to the newspaper’s actions, might now be seen as acceptable journalistic practice. It is all the worse when the editors’ published account of what they deleted is itself inaccurate. This does a disservice to readers, historians and journalists, who are left unable to determine for themselves what the controversy was about, and to Stephens, who is left unable to defend himself against readers’ worst suspicions.We strongly oppose racism, anti-Semitism and all forms of bigotry. And we believe that the best means of combating them is the open exchange of ideas. The Times’ retroactive censoring of passages of a published article appears to endorse a different view. And in doing so, it hands ammunition to the cynics and obfuscators who claim that every news source is merely an organ for its political coalition."""
After that we will process the news by cutting it into sentence form, then for each sentence the preprocessing is done using the function that we have defined above.
sentence_list = sentence_split(news)data = []for sentence in sentence_list:    data.append(tokenization(cleaning(casefolding(sentence))))data = (list(filter(None, data)))
After that, we will count the number of each word in the document from the function we have defined.
wordfreq = word_freq(data)
Then, from the results of word calculation we only need to calculate the weight of each sentence.
rank = sentence_rank(data)
After that we determine how many main sentences we want to output as a representation of the news. For example I will output 2 sentences, so here I fill n with 2. Then the system will take the top 2 sentences which have the highest weight. It depends on you to choose many sentences to appear.
n = 2result = ''sort_list = np.argsort(ranking)[::-1][:n]for i in range(n):    result += '{} '.format(sentence_list[sort_list[i]])
To display the results, use the following code
print(result)
So results will appear like this.
The editors did not just remove “reference to the study.” They expurgated the article’s original subtitle (which explicitly stated “It’s not about having higher IQs”), two mentions of Jewish IQs, and a list of statistics about Jewish accomplishment: “During the 20th century, [Ashkenazi Jews] made up about 3 percent of the U.S. population but won 27 percent of the U.S. Nobel science prizes and 25 percent of the ACM Turing awards. Detractors discovered that one of the authors of the paper Stephens had cited went on to express racist views, and falsely claimed that Stephens himself had advanced ideas that were “genetic” (he did not), “racist” (he made no remarks about any race) and “eugenicist” (alluding to the discredited political movement to improve the human species by selective breeding, which was not remotely related to anything Stephens wrote).
Not bad, right? Although it may not be as good as if we use state-of-the-art methods, we can quite understand the essence of our brief news. And tips from me, by using this method, you can generate your own dataset, although it still has limitations because the results obtained are not always the way you want.
Good luck!
If you like this post, I want to recommend an article that inspired me.