How to get the Daily News using Python | by Mukundan Sankar | Towards Data Science

Did you ever wonder how you can get the latest news programmatically? News API is one such module in Python which helps you in this. It has data from about 120 sources including Reuters, BBC News, ABC News, ESPN.
newsapi.org
Firstly, get the API key from NewsAPI to begin extracting news articles. You can see more on how to get one here.
newsapi.org
Import the relevant libraries.
from newsapi import NewsApiClient
This is for the NewsAPI import. Now initializing a variable for the NewsAPI
newsapi = NewsApiClient(api_key=api_key)from pandas.io.json import json_normalizeimport pandas as pd
We will use Beautiful Soup package in Python to help clean up the HTML tags from the data.
from bs4 import BeautifulSoup
Now, let’s see the practical use.
Latest News
To get the latest news:
def top_headlines():       country=input("Which country are you interested in?")    category=input("""Which category are you interested in? \nHere    are the categories to choose from:\nbusiness\nentertainment      \ngeneral\nhealth\nscience\ntechnology""")           top_headlines =newsapi.get_top_headlines(category=category,   language='en',country=country)        top_headlines=json_normalize(top_headlines['articles'])      newdf=top_headlines[["title","url"]]       dic=newdf.set_index('title')['url'].to_dict()
Let’s talk about what this piece of code does. I have written a function called top_headlines. I take in two inputs, country and category. When I supply an input to the country you would have to write it in a specific format. The country parameter in the News API follows this code system:
en.wikipedia.org
USA is “us”, the United Kingdom is “gb”, India is “in”. For symbols of other countries refer to the link above.
In the code itself, I have mentioned how we can supply an input to the category parameter. “general” for general news, “sports” for sports news, and so on.
Using get_top_headlines you can get the latest news. Now, the output is in JSON so I wanted to convert it into a Dataframe. So, I used the json_normalize method which I imported earlier in this tutorial. I am then storing that into a new Dataframe with only a few select columns from the list of columns returned in the JSON output.
Then I decided that I want to make sense of this news information. I took a used case to see how the opinion of the media changes over time for a particular timeframe. News API has a paid version which offers up to 2 years worth of information. In the free version, you can retrieve up to 30 days worth of news which is the timeframe that I chose as well.
Search for a particular topic
To do this analysis, we need to use this method:
get_everything(query,language,sort_by='relevancy',from_param,to)
What this means is it gets everything related to a particular search query. Let’s say you want to try to get every piece of information regarding the hottest topic in the world today, COVID-19 or Coronavirus, you would specify so in the “query” parameter in the above function.
The “from_param” parameter and “to” parameter in the above function are the date parameters. It is asking you to specify the timeframe for which you want to get results for the search query. Since I mentioned, I want to get all results for a period of 30 days which is the maximum allowed timeframe for the free version, I will specify the from_param as the day 30 days before today’s date. However, I need to write this in a loop so that I can ensure I am getting every day.
So, I built a date function first which helps me with this.
import datetimefrom datetime import datetime, timedeltadef date(base):        date_list=[]        yr=datetime.today().year        if (yr%400)==0 or ((yr%100!=0) and (yr%4==0)):                  numdays=366                date_list.append([base - timedelta(days=x) for x in            range(366)])       else:                numdays=365                date_list.append([base - timedelta(days=x) for x in            range(365)])            newlist=[]            for i in date_list:                   for j in sorted(i):                           newlist.append(j)            return newlist def last_30(base):         date_list=[base - timedelta(days=x) for x in range(30)]          return sorted(date_list)  def from_dt(x):        from_dt=[]        for i in range(len(x)):                  from_dt.append(last_30(datetime.today())[i-1].date())             return from_dt        def to_dt(x):        to_dt=[]        for i in range(len(x)):                to_dt.append(last_30(datetime.today())[i].date())        return to_dtfrom_list=from_dt(last_30(datetime.today()))to_list=to_dt(last_30(datetime.today()))
The first date function (date) that I have built is a more general date function where I want to use it for a large timeframe if needed. I am getting a list of all days using the first date function. I need only 30 days so I made sure to do that using last_30. This will give me a list of the last 30 days from the current day and time. Moving on to the from_dt function, it is specifically for the from_param for the get_everything function mentioned above. This is the start of the timeframe for which you need all the news articles and to_dt is the end date.
I now have two separate lists for start and end date respectively.
Now we will work on the query parameter which is the search term for the get_everything function.
def func(query):     newd={}        newdf=pd.DataFrame()        for (from_dt,to_dt) in zip(from_list,to_list):                   all_articles =           newsapi.get_everything(q=query,language='en',        sort_by='relevancy',from_param=from_dt,to=to_dt)                  d=json_normalize(all_articles['articles'])                 newdf=d[["url","publishedAt","source.name","author"]]        dic=newdf.set_index(["source.name","publishedAt","author"])         ["url"].to_dict()                for (k,v) in dic.items():                        page = requests.get(v)                        html = page.content                        soup = BeautifulSoup(html, "lxml")                        text = soup.get_text()                        d2=soup.find_all("p")                        newd[k]=re.sub(r'<.+?>',r'',str(d2))         return newd
Looping through the zipped version of the from and to lists for the dates, we get a list of all articles for the 30-day timeframe. The reason I chose to loop this is so that I can do analysis for every single day. I then store it in a data frame. I then get the relevant columns and store them back in a dictionary. I want to parse the content in the news URL so that I get the content only from the articles. I used the BeautifulSoup package in Python here to parse the content and get relevant <p> tags where the information resides. Since we are primarily focused on text data here, we need information between these paragraph tags. You can find more information on BeautifulSoup here:
www.crummy.com
Once we have cleaner data, we can now proceed towards the Wordclouds to analyze a pattern.
News Analysis using WordCloud
Now, we move on to the Wordcloud. The part where we can make sense of the data from the news. What are some of the commonly occurring patterns?
def wordcld(dictionary):            newd={}        for (k,v) in dictionary.items():            if v!='[]':                    wordcloud = WordCloud().generate(str(dictionary[k]))                        fig, axes= plt.subplots(figsize=(20,12),clear=True)                             plt.imshow(wordcloud, interpolation='bilinear')                    plt.show()                     else:                    print(str(k[0])+"_"+str(k[1][5:10])+"_"+str(k[1][11:13])                      +"_"+str(k[1][14:16]) +"_"+str(k[1][17:19])+"_"+str(k[2]))                     print("Wordcloud Not applicable")dic=func("Indian Economy")wordcld(dic)
I am searching for some commonly occurring words for the Indian Economy and I get the below image. The image I got is for one day. The code I have above will get you results for 30 days.
Try this out and feel free to let me know if you have any questions.
As a next step, you can send this as a text message/email notification to yourself using Twilio API/SendGrid Email API which I will cover in another tutorial or you could see some of the existing ones like this:
medium.com
sendgrid.com
To summarize:
Feel free to comment here or send me an email if you have any questions. You can also see this code on GitHub: https://github.com/mukund14/News/tree/news_api_Python