Find Common Words in Article with Python Module Newspaper and NLTK | by Khuyen Tran | Towards Data Science

You want to extract essential information from an interesting article, but find the article is too long to read with your limited amount of time. Before you dive into the whole article and end up disappointed with the irrelevant content, you should look at the summary and keywords beforehand.
newpaper3k and nltk are the two effective tools that could be used for scraping and summarizing news articles. Let’s analyze the interesting article that I came across today on Medium: It Took Me 2 Years To Get 1000 Followers — Life Lessons I’ve Learned Throughout The Journey using these 2 tools.
pip install newspaper3kfrom newspaper import Article url = 'https://mystudentvoices.com/it-took-me-2-years-to-get-1000-followers-life-lessons-ive-learned-throughout-the-journey-9bc44f2959f0'article = Article(url)article.download()
Find the publish date
article.publish_date
Extract the top image and take a quick look at the image
image_url = article.top_imagefrom IPython.display import Imagefrom IPython.core.display import HTML Image(url=image_url)
Output:
Extract author’s name
article.authors
Output:
['William Cho']
Extract article keywords:
article.keywords
Output:
['soon', 'ive', 'work', 'journey', '1000', 'took', 'followers', 'started', 'truth', 'life', 'writing', 'wasnt', 'lessons', 'doing', 'read', 'maybe', 'learned', 'youre']
Summarize the article
article.summary
Output:
'But I’ve only been writing on Medium for a total of maybe 4.5 months.\nI started writing passionately, and you could tell from my writing that I thought I was enlightened and speaking from authority.\nI’ve noticed that it comes in moments where I put in effort to improve myself — working out, writing, and reading.\nYou’re just doing it to put yourself on a higher ground than your friends, to judge them from a higher platform and deem everything you’re doing more virtuous than what they’re doing.\nI would sidestep and avoid the truth — the truth that would hurt but ultimately set me free.'
With the summary, we could quickly grasp the key takeaways of the article:
To become a better writer, he constantly improved his writing skills by writing, reading, and working out.
His motivation was once from external sources such as friends.
There was struggling in his journey. But by evaluating his approach, he sets himself free.
Let see if we could extract more insights from the article using NLTK
Start with extracting the text from the newspaper
text = article.text
pip install nltk

Requirement already satisfied: nltk in ./opt/anaconda3/lib/python3.7/site-packages (3.4.5)
Requirement already satisfied: six in ./opt/anaconda3/lib/python3.7/site-packages (from nltk) (1.12.0)
Note: you may need to restart the kernel to use updated packages.

import nltk

nltk.download()

showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml

True
Remove punctuation
from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
text = tokenizer.tokenize(text)

text = ' '.join(word for word in text)

Tokenize words
from nltk.tokenize import word_tokenize
tokenized_word=word_tokenize(text)

Lowercase
tokenized_word = [word.lower() for word in tokenized_word]

Remove stopwords
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
print(stop_words)

{'t', 'under', 'will', "should've", 'such', 'has', 'other', 'do', 'shan', 'was', 'his', "wouldn't", 'he', 'if', 'ours', 'did', 'which', 'my', 'had', 'after', 'through', 'off', 'the', 'most', 'o', 'aren', 'when', 'into', 'being', 'her', 'each', 'this', 'been', 'about', 'until', 'won', 'what', 'be', 'ourselves', 'doesn', 'they', 'hers', 'am', "it's", 'i', 'down', 'y', 'have', 'who', 'their', 'any', 'with', 'both', 'because', 'from', 'before', 'or', 'hasn', 'haven', 'doing', 'few', 'don', "shouldn't", 'for', 'ain', 'by', 'having', 'some', 'our', 'm', 'no', 'weren', 'yourself', 've', 'there', 'were', "you'll", 's', 'should', "won't", 'over', 'against', 'to', "wasn't", "aren't", 'on', 'him', 'are', "mightn't", 'as', "hadn't", 'how', 'you', 'it', "shan't", 'hadn', 'and', 'than', 'that', "couldn't", 'where', 'then', 'its', 'whom', 'below', 'here', "hasn't", 'all', "didn't", "isn't", 'wasn', 'an', 'yourselves', 'again', "don't", 'why', 'theirs', 'up', 'couldn', 'just', 'can', "doesn't", 'so', 'own', 'd', 'wouldn', 'we', 'she', "mustn't", 'these', 'between', 'once', 'didn', "you've", "haven't", "that'll", 'but', 'needn', 'itself', "you'd", 'me', 'those', 'very', 'isn', 'himself', 'during', "she's", 'does', 'nor', 'them', 'not', 'mustn', 'now', 'same', 'your', 'll', 'yours', 'ma', 'myself', 'only', 'in', 'a', 'while', 'above', 'out', 're', 'shouldn', 'mightn', 'more', "needn't", 'at', "weren't", 'is', "you're", 'themselves', 'of', 'herself', 'too', 'further'}

filtered_word = []
for word in tokenized_word:
  if word not in stop_words:
    filtered_word.append(word)
    

Stemming the word
from nltk.stem import PorterStemmer

ps = PorterStemmer()

stemmed_words = []
for w in filtered_word:
  stemmed_words.append(ps.stem(w))

#See how stemming works
for word in ['thinking', 'felt', 'asked','challenging','devoted']:
    print(ps.stem(word))

think
felt
ask
challeng
devot

Try with lemmatization and compare it with stemming
from nltk.stem.wordnet import WordNetLemmatizer
lem = WordNetLemmatizer()

lem_words = []
for w in filtered_word:
  lem_words.append(lem.lemmatize(w,'v'))
  

for word in ['thinking', 'felt', 'asked','challenging','devoted']:
    print(lem.lemmatize(word,'v'))

think
felt
ask
challenge
devote

from nltk.probability import FreqDist
fdist = FreqDist(lem_words)

most_common = fdist.most_common(20)
most_common

[('write', 26),
 ('read', 11),
 ('maybe', 8),
 ('start', 8),
 ('one', 7),
 ('work', 7),
 ('know', 6),
 ('soon', 6),
 ('time', 6),
 ('think', 6),
 ('doubt', 6),
 ('would', 6),
 ('medium', 5),
 ('make', 5),
 ('truth', 5),
 ('day', 4),
 ('everyday', 4),
 ('keep', 4),
 ('could', 4),
 ('fail', 4)]
pip install matplotlib

Requirement already satisfied: matplotlib in ./opt/anaconda3/lib/python3.7/site-packages (3.1.1)
Requirement already satisfied: cycler>=0.10 in ./opt/anaconda3/lib/python3.7/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./opt/anaconda3/lib/python3.7/site-packages (from matplotlib) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./opt/anaconda3/lib/python3.7/site-packages (from matplotlib) (2.4.2)
Requirement already satisfied: python-dateutil>=2.1 in ./opt/anaconda3/lib/python3.7/site-packages (from matplotlib) (2.8.0)
Requirement already satisfied: numpy>=1.11 in ./opt/anaconda3/lib/python3.7/site-packages (from matplotlib) (1.17.2)
Requirement already satisfied: six in ./opt/anaconda3/lib/python3.7/site-packages (from cycler>=0.10->matplotlib) (1.12.0)
Requirement already satisfied: setuptools in ./opt/anaconda3/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib) (41.4.0)
Note: you may need to restart the kernel to use updated packages.

import matplotlib.pyplot as plt

plt.figure(figsize=(20,5))
plt.plot([word[0] for word in most_common], [word[1] for word in most_common])
plt.xlabel('Words')
plt.ylabel('Frequency')

Text(0, 0.5, 'Frequency')
 

By looking at the plot of the most frequent words, we have a better idea of what the article about.
Let’s utilize another tool called WordClouds to create a more interesting visualization of our article
From this list of words, I could guess the general idea of the article: Becoming a better writer takes effort. You have to consistently write and read every day to improve your skills. There may be doubts and fear, but things will eventually improve if you trust the journey and keep your eyes on the bigger vision. Now, I could dive into reading this article since I know it will give me practical lessons for improving my writing skills.
newspaper3k andnltk are the great combination for extracting the essential information from an online article. You could also utilize this tool to grasp the general ideas of the article to decide whether the article is worthy of reading or to analyze several articles at once.
Feel free to fork and play with the code for this article in this Github repo.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
medium.com
[1]https://care.pressreader.com/hc/en-us/articles/203210155-Advanced-Keyword-Search
[2] William Cho, It Took Me 2 Years To Get 1000 Followers — Life Lessons I’ve Learned Throughout The Journey (2018), My Student Voices