Package: pygooglenews
pygooglenewsIf Google News had a Python libraryCreated by Artem fromnewscatcherapi.combut you do not need anything from us or from anyone else to get the software going, it just works out of the box.Myblog post about how I did itDemoYou might also like to check ourGoogle News APIorFinancial Google News APITable of ContentsAboutExamples of Use CasesWorking with Google News in ProductionMotivationInstallationQuickstartDocumentationAdvanced Query Search ExamplesAbout meChange LogAboutA python wrapper of the Google News RSS feed.Top stories, topic related news feeds, geolocation news feed, and an extensive full text search feed.This work is more of a collection of all things I could find out about how Google News functions.How is it different from other Pythonic Google News libraries?URL-escaping user input helper for the search functionExtensive support for the search function that makes it simple to use:exact matchin title match, in url match, etcsearch by date range (from_&to_), latest published (when)Parsing of the sub articles. Almost always, all feeds except the search one contain a subset of similar news for each article in a feed. This package takes care of extracting those sub articles. This feature might be highly useful to ML task when you need to collect a data of similar article headlinesExamples of Use CasesIntegrating a news feed to your platform/application/websiteCollecting data by topic to train your own ML modelSearch for latest mentions for your new productMedia monitoring of people/organizations — PRWorking with Google News in ProductionBefore we start, if you want to integrate Google News data to your production then I would advise you to use one of the 3 methods described below. Why? Because you do not want your servers IP address to be locked by Google. Every time you call any function there is an HTTPS request to Google's servers.Don't get me wrong, this Python package still works out of the box.NewsCatcher'sGoogle News API— all code is written for you, clean & structured JSON output. Low price. You can test it yourself with no credit card. Plus,financial version of APIis also available.ScrapingBee APIwhich handles proxy rotation for you. Each function in this package hasscraping_beeparameter where you paste your API key. You can also try it for free, no credit card required. SeeexampleYour own proxy — already have a pool of proxies? Each function in this package hasproxiesparameter (python dictionary) where you just paste your own proxies.MotivationI love working with the news data. I love it so much that I createdmy own company that crawls for hundreds of thousands of news articles, and allow you to search it via a news API. But this time, I want to share with the community a Python package that makes it simple to get the news data from the best search engine ever created -Google.Most likely, you know already that Google has its ownnews service. It is different from the usual Google search that we use on a daily basis (sorryDuckDuckGo, maybe next time).This package uses the RSS feed of the Google News. Thetop stories page, for example.RSS is an XML page that is already well structured. I heavily rely on Feedparser package to parse the RSS feed.Google News used to have an API but it was deprecated many years ago. (Unofficial) information
about RSS syntax is decentralized over the web. There is no official documentation. So, I tried my best
to collect all this informaion in one place.Installation$pipinstallpygooglenews--upgradeQuickstartfrompygooglenewsimportGoogleNewsgn=GoogleNews()Top Storiestop=gn.top_news()Stories by Topicbusiness=gn.topic_headlines('business')Geolocation Specific Storiesheadquaters=gn.geo_headlines('San Fran')Stories by a Query Search# search for the best matching articles that mention MSFT and# do not mention AAPL (over the past 6 monthsearch=gn.search('MSFT -APPL',when='6m')Documentation - Functions & ClassesGoogleNews ClassfrompygooglenewsimportGoogleNews# default GoogleNews instancegn=GoogleNews(lang='en',country='US')To get the access to all the functions, you first have to initiate theGoogleNewsclass.It has 2 required variables:langandcountryYou can try any combination of those 2, however, it does not exist for all. Only the combinations that are supported by GoogleNews will work. Check the official Google News page to check what is covered:On the bottom left side of the Google News page you may find aLanguage & regionsection where you can find all of the supported combinations.For example, forcountry=UA(Ukraine), there are 2 languages supported:lang=ukUkrainianlang=ruRussianTop Storiestop=gn.top_news(proxies=None,scraping_bee=None)top_news()returns the top stories for the selected country and language that are defined inGoogleNewsclass. The returned object containsfeed(FeedParserDict) andentrieslist of articles found with all data parsed.Stories by Topicbusiness=gn.topic_headlines('BUSINESS',proxies=None,scraping_bee=None)The returned object containsfeed(FeedParserDict) andentrieslist of articles found with all data parsed.Accepted topics are:WORLDNATIONBUSINESSTECHNOLOGYENTERTAINMENTSCIENCESPORTSHEALTHHowever, you can find some other topics that are also supported by Google News.For example, if you search forcoronain the search tab ofen+USyou will findCOVID-19as a topic.The URL looks like this:https://news.google.com/topics/CAAqIggKIhxDQkFTRHdvSkwyMHZNREZqY0hsNUVnSmxiaWdBUAE?hl=en-US&gl=US&ceid=US%3AenWe have to copy the text aftertopics/and before?, then you can use it as an input for thetop_news()function.frompygooglenewsimportGoogleNewsgn=GoogleNews()covid=gn.topic_headlines('CAAqIggKIhxDQkFTRHdvSkwyMHZNREZqY0hsNUVnSmxiaWdBUAE')However, be aware that this topic will be unique for each language/country combination.Stories by Geolocationgn=GoogleNews('uk','UA')kyiv=gn.geo_headlines('kyiv',proxies=None,scraping_bee=None)# orkyiv=gn.geo_headlines('kiev',proxies=None,scraping_bee=None)# orkyiv=gn.geo_headlines('киев',proxies=None,scraping_bee=None)# orkyiv=gn.geo_headlines('Київ',proxies=None,scraping_bee=None)The returned object containsfeed(FeedParserDict) andentrieslist of articles found with all data parsed.All of the above variations will return the same feed of the latest news about Kyiv, Ukraine:geo['feed'].title# 'Київ - Останні - Google Новини'It is language agnostic, however, it does not guarantee that the feed for any specific place will exist. For example, if you want to find the feed onLAorLos Angelesyou can do it withGoogleNews('en', 'US').The main (en,US) Google News client will most likely find the feed about the most places.Stories by a Querygn.search(query:str,helper=True,when=None,from_=None,to_=None,proxies=None,scraping_bee=None)The returned object containsfeed(FeedParserDict) andentrieslist of articles found with all data parsed.Google News search itself is a complex function that has inherited some features from the standard Google Search.The official reference on what could be insertedThe biggest obstacle that you might have is to write the URL-escaping input. To ease this process,helper = Trueis turned on by default.helperusesurllib.parse.quote_plusto automatically convert the input.For example:'New York metro opening'-->'New+York+metro+opening''AAPL -MSFT'-->'AAPL+-MSFT''"Tokyo Olimpics date changes"'-->'%22Tokyo+Olimpics+date+changes%22'You can turn it off and write your own query in case you need it byhelper = Falsewhenparameter (str) sets the time range for the published datetime. I could not find any documentation regarding this option, but here is what I deducted:hfor hours.(For me, worked for up to101h).when=12hwill search for only the articles matching thesearchcriteri and published for the last 12 hoursdfor daysmfor month (For me, worked for up to48m)I did not set any hard limit here. You may try put here anything. Probably, it will work. However, I would like to warn you that wrong inputs will not lead to an error. Instead, thewhenparameter will be ignored by the Google.from_andto_accept the following format of date:%Y-%m-%dFor example,2020-07-01Google's Special Query TermsCheat SheetMany Google's Special Query Terms have been tested one by one. Most of the core ones have been inherited by Google News service. At first, I wanted to integrate all of those as thesearch()function parameters. But, I realised that it might be a bit confusing and difficult to make them all work correctly.Instead,I decided to write some kind of a cheat sheet that should give you a decent understanding of what you could do.Boolean OR Search [ OR ]frompygooglenewsimportGoogleNewsgn=GoogleNews()s=gn.search('boeing OR airbus')print(s['feed'].title)# "boeing OR airbus" - Google NewsExclude Query Term [-]"The exclude (-) query term restricts results for a particular search request to documents that do not contain a particular word or phrase. To use the exclude query term, you would preface the word or phrase to be excluded from the matching documents with "-" (a minus sign)."Include Query Term [+]"The include (+) query term specifies that a word or phrase must occur in all documents included in the search results. To use the include query term, you would preface the word or phrase that must be included in all search results with "+" (a plus sign).The URL-escaped version of+(a plus sign) is%2B."Phrase Search"The phrase search (") query term allows you to search for complete phrases by enclosing the phrases in quotation marks or by connecting them with hyphens.The URL-escaped version of"(a quotation mark) is%22.Phrase searches are particularly useful if you are searching for famous quotes or proper names."allintext"Theallintext:query term requires each document in the search results to contain all of the words in the search query in the body of the document. The query should be formatted asallintext:followed by the words in your search query.If your search query includes theallintext:query term, Google will only check the body text of documents for the words in your search query, ignoring links in those documents, document titles and document URLs."intitle"Theintitle:query term restricts search results to documents that contain a particular word in the document title. The search query should be formatted asintitle:WORDwith no space between the intitle: query term and the following word."allintitle"Theallintitle:query term restricts search results to documents that contain all of the query words in the document title. To use theallintitle:query term, include "allintitle:" at the start of your search query.Note: Puttingallintitle:at the beginning of a search query is equivalent to puttingintitle:in front of each word in the search query."inurl"Theinurl:query term restricts search results to documents that contain a particular word in the document URL. The search query should be formatted asinurl:WORDwith no space between the inurl: query term and the following word"allinurlTheallinurl:query term restricts search results to documents that contain all of the query words in the document URL. To use theallinurl:query term, include allinurl: at the start of your search query.List of operators that do not work (for me, at least):Most (probably all) of theas_*terms do not work for Google Newsallinlinks:related:Tip. If you want to build a near real-time feed for a specific topic, usewhen='1h'. If Google captured fewer than 100 articles over the past hour, you should be able to retrieve all of them.Check theUseful Linkssection if you want to dig into how Google Search works.Especially,Special Query Termssection of Google XML reference.Plus, I will provide some more examples under theFull-Text Search ExamplessectionOutput BodyAll 4 functions return thedictionarythat has 2 sub-objects:feed- contains the information on the feed metadataentries- contains the parsed articlesBoth are inherited from theFeedparser. The only change is that each dictionary underentriesalso containssub_articleswhich are the similar articles found in the description. Usually, it is non-empty fortop_news()andtopic_headlines()feeds.TipTo check what is the found feed's name just check thetitleunder thefeeddictionaryHow to use pygooglenews withScrapingBeeEvery function hasscrapingbeeparameter. It accepts yourScrapingBeeAPI key that will be used to get the response from Google's servers.You can take a look at what exactly is happening in the source code: check for__scaping_bee_request()function under GoogleNews classPay attention to the concurrency of each plan atScrapingBeeHow to use example:gn=GoogleNews()# it's a fake API key, do not try to use itgn.top_news(scraping_bee='I5SYNPRFZI41WHVQWWUT0GNXFMO104343E7CXFIISR01E2V8ETSMXMJFK1XNKM7FDEEPUPRM0FYAHFF5')How to use pygooglenews with proxiesSo, if you have your own HTTP/HTTPS proxy(s) that you want to use to make requests to Google, that's how you do it:gn=GoogleNews()gn.top_news(proxies={'https':'34.91.135.38:80'})Advanced Querying Search ExamplesExample 1. Search for articles that mentionboeingand do not mentionairbusfrompygooglenewsimportGoogleNewsgn=GoogleNews()s=gn.search('boeing -airbus')print(s['feed'].title)# "boeing -airbus" - Google NewsExample 2. Search for articles that mentionboeingin titlefrompygooglenewsimportGoogleNewsgn=GoogleNews()s=gn.search('intitle:boeing')print(s['feed'].title)# "intitle:boeing" - Google NewsExample 3. Search for articles that mentionboeingin title and got published over the past hourfrompygooglenewsimportGoogleNewsgn=GoogleNews()s=gn.search('intitle:boeing',when='1h')print(s['feed'].title)# "intitle:boeing when:1h" - Google NewsExample 4. Search for articles that mentionboeingorairbusfrompygooglenewsimportGoogleNewsgn=GoogleNews()s=gn.search('boeing OR airbus',when='1h')print(s['feed'].title)# "boeing AND airbus when:1h" - Google NewsUseful LinksStack Overflow thread from which it all beganGoogle XML reference for the search queryGoogle News Search parameters (The Missing Manual)Built WithFeedparserBeutifulsoup4About meMy name is Artem. I ❤️ working with news data. I am a co-founder ofNewsCatcherAPI-Ultra-fast API to find news articles by any topic, country, language, website, or keywordIf you are interested in hiring me, please, contact me by email -bugara.artem@gmail.comorartem@newscatcherapi.comFollow me on 🖋Twitter- I write about data engineering, python, entrepreneurship, and memes.Want to read about how it all was done? Subscribe toCODARIUMthx to KizyChange Logv0.1.1 -- fixed language-country issues