Use Python To Fetch Google News
By: Farhad Malik
Tags: Technology, Fintech, Data Science, Programming, Asyncio

Use Python To Fetch Google News

Outlining the steps to download news from Google via Python code

All successful projects revolve around clean data. Google is by far the best search engine. Furthermore, Google News is a fantastic source that combines news from various media and publishers.

I will demonstrate how we can build data sets for data science projects by using news from Google News. I wanted to be able to fetch Google News via Python code so that I can get the latest news daily without any manual intervention.

This article will demonstrate the steps that are required to retrieve google news for our chosen topics, words, and locations via Python code.

I will present two methodologies that can be utilized to get news from Google News.

Google News via Python. Image By Author

There are several paid services that are essentially a wrapper around GoogleNews but I wanted to explore the free libraries first.

Let’s jump to the code.

Approach 1: PygoogleNews

This approach demonstrates how we can get Google News by using the library pygooglenews. This library is a wrapper around the Google News RSS feed.

Key Features

We can use the library for many uses to interact with the Google News RSS feed, including:

Getting the top news

Getting location specific headlines

Getting stories for a topic. The acceptable topics are: world, nation, business, technology, entertainment, science, sports, and health.

Searching for stories using a query whereby we can pass any text as an argument

Implementation Steps

This section will outline the steps that can be followed to implement the google news fetcher using pygooglenews library

Step 1: Install the library

pip install pygooglenews

Step 2: Create a file: google_extractor.py

Import the libraries in the google_extractor.py

import asyncio

from pygooglenews import GoogleNews

Note, I am importing the asyncio and pygooglenews libraries. The asyncio library will help us execute the functions asynchronously without blocking the application. It will be faster to execute the functions asynchronously as they are IO bound in nature.

Step 3: Create a class GoogleApi and add the following three private functions in the file google_extractor.py :

_get_api: Returns an instance of the GoogleNews

_get_news_for_word: calls the search() function to get last 1 week news for your target word

_clean: Cleans the text and removes any noise

class GoogleApi:

def __init__(self):

self._api = self._get_api() def _get_api(self):

return GoogleNews() def _get_news_for_word(self, word, top):

text = self._api.search(word, when='1w')['entries']

news = list(map(lambda x: self._clean(x['title']) + ' @ (' + x['published'] + ')', text))[:top]

return news def _clean(self, text):

clean_text = re.sub(r'http\S+', '', text, flags=re.MULTILINE)

regrex_pattern = re.compile(pattern="["

u"\U0001F600-\U0001F64F" # emoticons

u"\U0001F300-\U0001F5FF" # symbols & pictographs

u"\U0001F680-\U0001F6FF" # transport & map symbols

u"\U0001F1E0-\U0001F1FF" # flags (iOS)

u"\U0001f17a-\U0001f97a" # flags (iOS)

"]+", flags=re.UNICODE)

return regrex_pattern.sub(r'', clean_text)

To understand how concurrency and parallelism work in detail, I highly recommend reading this article:

Step 4: Add the following public functions to the file google_extractor.py :

get_news_for_locations: This function gets the news for our chosen locations.

get_news_for_words: This function gets the news for our chosen words.

get_topic_headlines: This function gets the news for our chosen topics.

async def get_news_for_locations(self, target_locations, top=3):

final = {}

for location in target_locations:

text = self._api.geo_headlines(location)['entries']

news = list(map(lambda x: self._clean(x['title']) + ' @ (' + x['published'] + ')', text))[:top]

final[location] = news

return final async def get_news_for_words(self, target_words, top=3):

text = {}

for word in target_words:

ts = self._get_news_for_word(word, top)

text[word] = ts

return text async def get_topic_headlines(self, topics, top=3):

topic_news = {}

for topic in topics:

try:

topic_headlines = self._api.topic_headlines(topic)['entries']

topic_news[topic] = list(

map(lambda x: self._clean(x['title']) + ' @ (' + x['published'] + ')', topic_headlines))[:top]

except Exception as e:

print(topic)

print(e)

topic_news[topic] = []

return topic_news async def get_top_news(self):

return self._api.top_news()

Step 5: Create the main function that creates an instance of GoogleApi() and executes its public functions to get location, word and user news in the file google_extractor.py :

async def main():

"""

Main method, call the appropriate methods

:return:

"""

google_api = GoogleApi()

trends = google_api.get_news_for_locations(['United States', 'United Kingdom', 'Worldwide'])

words = google_api.get_topic_headlines(['BUSINESS'])

users = google_api.get_news_for_words(['bloomberg', 'cnbc'], 2)

trends_text, words_text, news_users = await asyncio.gather(trends, words, users) print('LOCATION NEWS')

for location, news in trends_text.items():

print(f'Location: {location}

{news}') print('

WORD NEWS')

for word, news in words_text.items():

print(f'Word: {word}

{news}') print('

USER NEWS')

for user, news in news_users.items():

print(f'User: {user}

{news}')

Step 6: Finally execute the main function

Place this snippet of code at the bottom of the file google_extractor.py

if __name__ == "__main__":

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait([main()]))

The results will be printed as shown below:

LOCATION NEWS

Location: United States

['Tensions linger between Biden and Obama camps throughout 2020 primary campaign: report - Fox News @ (Fri, 14 Aug 2020 20:09:38 GMT)', 'Postal Service warns 46 states their voters could be disenfranchised by delayed mail-in ballots - The Washington Post @ (Fri, 14 Aug 2020 20:44:00 GMT)', 'Ex-F.B.I. Lawyer Expected to Plead Guilty in Review of Russia Inquiry - The New York Times @ (Fri, 14 Aug 2020 21:26:38 GMT)']

Location: United Kingdom

['Coronavirus: France to be added to UK quarantine countries - BBC News @ (Fri, 14 Aug 2020 10:01:36 GMT)', 'UK crashes into deepest recession of any major economy - CNN @ (Wed, 12 Aug 2020 14:10:00 GMT)', 'EU settlement scheme: Two million granted full rights to stay - BBC News @ (Fri, 14 Aug 2020 03:56:17 GMT)']

Location: Worldwide

['Tensions linger between Biden and Obama camps throughout 2020 primary campaign: report - Fox News @ (Fri, 14 Aug 2020 20:09:38 GMT)', 'Postal Service warns 46 states their voters could be disenfranchised by delayed mail-in ballots - The Washington Post @ (Fri, 14 Aug 2020 20:44:00 GMT)', 'Ex-F.B.I. Lawyer Expected to Plead Guilty in Review of Russia Inquiry - The New York Times @ (Fri, 14 Aug 2020 21:26:38 GMT)'] WORD NEWS

Word: BUSINESS

["Dow Jones Edges Higher In New Bid For 28000; Tesla, AMAT Stock Lead Nasdaq - Investor's Business Daily @ (Fri, 14 Aug 2020 19:01:00 GMT)", 'FDA finds new toxic hand-sanitizer ingredient, expands warning to 157 products - Ars Technica @ (Fri, 14 Aug 2020 19:35:00 GMT)', 'U.S. retail sales slow in July; obstacles mount for nascent economic recovery - Reuters @ (Fri, 14 Aug 2020 12:40:00 GMT)', 'Trump says McKesson Corp. to distribute coronavirus vaccine under Operation Warp Speed - MarketWatch @ (Fri, 14 Aug 2020 18:07:00 GMT)', 'Warren Buffett made a bunch of stock moves last quarter — Here are the highlights - CNBC @ (Fri, 14 Aug 2020 20:32:00 GMT)'] USER NEWS

User: bloomberg

['Singapore Leaves Little to Chance in Covid Election - Bloomberg @ (Tue, 30 Jun 2020 07:00:00 GMT)', 'Bloomberg advocates targeting minorities in audio from 2015 - The San Diego Union-Tribune @ (Tue, 11 Feb 2020 08:00:00 GMT)']

User: cnbc

["India's interior minister tests positive for coronavirus, hospitalized - CNBC @ (Sun, 02 Aug 2020 07:00:00 GMT)", "WHO says coming weeks are 'critical' as Europe's cases near 1 million - CNBC @ (Wed, 15 Apr 2020 07:00:00 GMT)"]

For more information on the library, read it here

In the library’s link, the author mentioned that “if you want to integrate Google News data to your production then I would advise you to use one of the 3 methods described below. Why? Because you do not want your servers IP address to be locked by Google”. Do keep this mind and it’s very important.

I really like pygooglenews . However, there must be another approach.

Approach 2:

This section will demonstrate how we can use the GoogleNews library to get news from Google News. The code follows a similar pattern as above

Step 1: Create a Python file: google_news_extractor.py

Step 2: Install the library googlenews

Step 3: Copy and paste the following code:

import asyncio

from GoogleNews import GoogleNews class GoogleApi:

def __init__(self):

self._api = self._get_api() def _get_news_for_word(self, word, top):

self._api.search(word)

text = self._api.result()

news = list(map(lambda x: self._clean(x['title']) + ' @ (' + x['date'] + ')', text))[:top]

return news async def get_news_for_locations(self, target_locations, top=3):

final = {}

for location in target_locations:

self._api.search(location)

text = self._api.result()

news = list(map(lambda x: self._clean(x['title']) + ' @ (' + x['date'] + ')', text))[:top]

final[location] = news

return final def _get_api(self):

return GoogleNews(period="1w") async def get_news_for_words(self, target_words, top=3):

text = {}

for word in target_words:

ts = self._get_news_for_word(word, top)

text[word] = ts

return text async def get_topic_headlines(self, topics, top=5):

topic_news = {}

for topic in topics:

try:

self._api.search(topic)

topic_headlines = self._api.result()

topic_news[topic] = list(

map(lambda x: self._clean(x['title']) + ' @ (' + x['date'] + ')', topic_headlines))[:top]

except Exception as e:

print(topic)

print(e)

topic_news[topic] = []

return topic_news async def get_top_news(self):

return self._api.top_news() async def main():

"""

Main method, call the appropriate methods

:return:

"""

google_api = GoogleApi()

trends = google_api.get_news_for_locations(['United States', 'United Kingdom', 'Worldwide'])

words = google_api.get_topic_headlines(['Coronavirus'])

users = google_api.get_news_for_words(['bloomberg', 'cnbc'], 2)

trends_text, words_text, news_users = await asyncio.gather(trends, words, users) print('LOCATION NEWS')

for location, news in trends_text.items():

print(f'Location: {location}

{news}') print('

WORD NEWS')

for word, news in words_text.items():

print(f'Word: {word}

{news}') print('

USER NEWS')

for user, news in news_users.items():

print(f'User: {user}

{news}') if __name__ == "__main__":

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait([main()]))

Summary

Thank you for reading. Image By Author

This article demonstrated the steps that are required to retrieve google news for our chosen topics, words, and locations via Python code. We can use Google news and Python to build data sets for our Python projects.