Scrape and Summarize News Python | Towards Data Science

In today’s digital world, we’re bombarded with endless information. We’ve got infinitely scrolling social media feeds and a 24-hour news cycle. So there’s plenty of news to stay aware of and we’ve got to be able to digest it quickly!
So let’s go through an exercise to shrink news articles to a more easily digestible size.
We’ll scrape an example article using the requests and BeautifulSoup packages, then we’ll summarize it using the excellent gensim library. You can follow along interactively by downloading this Jupyter Notebook available on my Github.
Let’s get right to it!
# Importsimport requestsfrom bs4 import BeautifulSoupfrom gensim.summarization import summarize
Now let’s pick an interesting article:
Now that we have an article, we’ll retrieve its content:
# Retrieve page texturl = 'https://www.npr.org/2019/07/10/740387601/university-of-texas-austin-promises-free-tuition-for-low-income-students-in-2020'page = requests.get(url).text
Now let’s scrape!
First, we’ll turn the page content into a BeautifulSoup object, which will allow us to parse the HTML tags.
# Turn page into BeautifulSoup object to access HTML tagssoup = BeautifulSoup(page)
Then, we’ll need to figure out which HTML tags contain the headline and the main text of the article. For a great HTML primer, check out HTML.com.
To do this, we’ll use Google Chrome’s excellent Inspect tool. Open the news article we chose in a new tab, right-click on the page and choose Inspect from the drop-down menu. That will bring up the Inspect tool which looks like this:
Click on the little button outlined above to find the HTML tags corresponding to anything you see on the page.
When you hover over the text on the page you want to select, in this case the headline and the main text of the article, you’ll see which HTML tags are used to identify that text.
The headline of the article is surrounded by the <h1> tag. We’ll select the first <h1> tag on the page like this:
# Get headlineheadline = soup.find('h1').get_text()
The main text of the article is surrounded by the <p> tag. This time we’ll have to find all of the <p> tags contained on the page since the paragraphs of the article are each contained in a <p> tag.
# Get text from all <p> tags.p_tags = soup.find_all('p’)# Get the text from each of the “p” tags and strip surrounding whitespace.p_tags_text = [tag.get_text().strip() for tag in p_tags]
If you examine the text we’ve retrieved in the ‘p_tags_text’ variable, you’ll notice there are some bits of text that aren’t from the main article, like the author’s name and some image captions. Since they are also delineated using <p> tags, we’ve selected them too. To clean up the text we have, we can use a couple of quick list comprehensions to filter out types of text we know aren’t part of the main article.
In this article, the image captions contain the newline ‘\n’ character to add whitespace around them. Since we know that an actual sentence from the article wouldn’t have a random line break, we can safely drop these. Similarly, we can drop out bits of text that don’t contain a period, since we know any proper sentence in an article would contain a period. That will drop out the author’s name and some other irrelevant bits.
# Filter out sentences that contain newline characters '\n' or don't contain periods.sentence_list = [sentence for sentence in p_tags_text if not '\n' in sentence]sentence_list = [sentence for sentence in sentence_list if '.' in sentence]# Combine list items into string.article = ' '.join(sentence_list)
Now that we have the text of the article, we can summarize it.
Gensim is an excellent Python package for a variety of NLP tasks. It includes a fairly robust summarization function that is easy to use. It’s a variation of the TextRank algorithm based on the findings of this paper (documentation). And as you’ll see, we can use it in one line of code:
summary = summarize(article_text, ratio=0.3)
There you have it! We’ve retrieved the headline of the article and created a summary of the content. Now you can get the gist of the article in about 1/3rd of the time and move on with your day.
For my next post, I’ll extend this functionality by creating a command-line utility to scrape and summarize an article and automatically email the summary to yourself. Stay tuned!