Package: google-search-results
Google Search Results in PythonThis Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home Depot, eBay and more, usingSerpApi.The following services are provided:Search APISearch Archive APIAccount APILocation API(Google Only)SerpApi provides ascript builderto get you started quickly.InstallationPython 3.7+pipinstallgoogle-search-resultsLink to the python package pageQuick startfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffee","location":"Austin,Texas","api_key":"<your secret api key>"})result=search.get_dict()This example runs a search for "coffee" using your secret API key.The SerpApi service (backend)Searches Google using the search: q = "coffee"Parses the messy HTML responsesReturns a standardized JSON response
The GoogleSearch classFormats the requestExecutes a GET http request against SerpApi serviceParses the JSON response into a dictionaryEt voilà...Alternatively, you can search:Bing using BingSearch classBaidu using BaiduSearch classYahoo using YahooSearch classDuckDuckGo using DuckDuckGoSearch classeBay using EbaySearch classYandex using YandexSearch classHomeDepot using HomeDepotSearch classGoogleScholar using GoogleScholarSearch classYoutube using YoutubeSearch classWalmart using WalmartSearchApple App Store using AppleAppStoreSearch classNaver using NaverSearch classSee theplayground to generate your code.SummaryGoogle Search Results in PythonInstallationQuick startSummaryGoogle Search API capabilityHow to set SERP API keyExample by specificationLocation APISearch Archive APIAccount APISearch BingSearch BaiduSearch YandexSearch YahooSearch EbaySearch Home depotSearch YoutubeSearch Google ScholarGeneric search with SerpApiClientSearch Google ImagesSearch Google NewsSearch Google ShoppingGoogle Search By LocationBatch Asynchronous SearchesPython object as a resultPython paginate using iteratorError managementChange logConclusionGoogle Search API capabilitySource code.params={"q":"coffee","location":"Location Requested","device":"desktop|mobile|tablet","hl":"Google UI Language","gl":"Google Country","safe":"Safe Search Flag","num":"Number of Results","start":"Pagination Offset","api_key":"Your SERP API Key",# To be match"tbm":"nws|isch|shop",# To be search"tbs":"custom to be search criteria",# allow async request"async":"true|false",# output format"output":"json|html"}# define the search searchsearch=GoogleSearch(params)# override an existing parametersearch.params_dict["location"]="Portland"# search format return as raw htmlhtml_results=search.get_html()# parse results#  as python Dictionarydict_results=search.get_dict()#  as JSON using json packagejson_results=search.get_json()#  as dynamic Python objectobject_result=search.get_object()Link to the full documentationSee below for more hands-on examples.How to set SERP API keyYou can get an API key here if you don't already have one:https://serpapi.com/users/sign_upThe SerpApiapi_keycan be set globally:GoogleSearch.SERP_API_KEY="Your Private Key"The SerpApiapi_keycan be provided for each search:query=GoogleSearch({"q":"coffee","serp_api_key":"Your Private Key"})Example by specificationWe love true open source, continuous integration and Test Driven Development (TDD).
We are using RSpec to testour infrastructure around the clockto achieve the best Quality of Service (QoS).The directory test/ includes specification/examples.Set your API key.exportAPI_KEY="your secret key"Run testmaketestLocation APIfromserpapiimportGoogleSearchsearch=GoogleSearch({})location_list=search.get_location("Austin",3)print(location_list)This prints the first 3 locations matching Austin (Texas, Texas, Rochester).[{'canonical_name':'Austin,TX,Texas,United States','country_code':'US','google_id':200635,'google_parent_id':21176,'gps':[-97.7430608,30.267153],'id':'585069bdee19ad271e9bc072','keys':['austin','tx','texas','united','states'],'name':'Austin, TX','reach':5560000,'target_type':'DMA Region'},...]Search Archive APIThe search results are stored in a temporary cache.
The previous search can be retrieved from the cache for free.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas"})search_result=search.get_dictionary()assertsearch_result.get("error")==Nonesearch_id=search_result.get("search_metadata").get("id")print(search_id)Now let's retrieve the previous search from the archive.archived_search_result=GoogleSearch({}).get_search_archive(search_id,'json')print(archived_search_result.get("search_metadata").get("id"))This prints the search result from the archive.Account APIfromserpapiimportGoogleSearchsearch=GoogleSearch({})account=search.get_account()This prints your account information.Search BingfromserpapiimportBingSearchsearch=BingSearch({"q":"Coffee","location":"Austin,Texas"})data=search.get_dict()This code prints Bing search results for coffee as a Dictionary.https://serpapi.com/bing-search-apiSearch BaidufromserpapiimportBaiduSearchsearch=BaiduSearch({"q":"Coffee"})data=search.get_dict()This code prints Baidu search results for coffee as a Dictionary.https://serpapi.com/baidu-search-apiSearch YandexfromserpapiimportYandexSearchsearch=YandexSearch({"text":"Coffee"})data=search.get_dict()This code prints Yandex search results for coffee as a Dictionary.https://serpapi.com/yandex-search-apiSearch YahoofromserpapiimportYahooSearchsearch=YahooSearch({"p":"Coffee"})data=search.get_dict()This code prints Yahoo search results for coffee as a Dictionary.https://serpapi.com/yahoo-search-apiSearch eBayfromserpapiimportEbaySearchsearch=EbaySearch({"_nkw":"Coffee"})data=search.get_dict()This code prints eBay search results for coffee as a Dictionary.https://serpapi.com/ebay-search-apiSearch Home DepotfromserpapiimportHomeDepotSearchsearch=HomeDepotSearch({"q":"chair"})data=search.get_dict()This code prints Home Depot search results for chair as Dictionary.https://serpapi.com/home-depot-search-apiSearch YoutubefromserpapiimportHomeDepotSearchsearch=YoutubeSearch({"q":"chair"})data=search.get_dict()This code prints Youtube search results for chair as Dictionary.https://serpapi.com/youtube-search-apiSearch Google ScholarfromserpapiimportGoogleScholarSearchsearch=GoogleScholarSearch({"q":"Coffee"})data=search.get_dict()This code prints Google Scholar search results.Search WalmartfromserpapiimportWalmartSearchsearch=WalmartSearch({"query":"chair"})data=search.get_dict()This code prints Walmart search results.Search YoutubefromserpapiimportYoutubeSearchsearch=YoutubeSearch({"search_query":"chair"})data=search.get_dict()This code prints Youtube search results.Search Apple App StorefromserpapiimportAppleAppStoreSearchsearch=AppleAppStoreSearch({"term":"Coffee"})data=search.get_dict()This code prints Apple App Store search results.Search NaverfromserpapiimportNaverSearchsearch=NaverSearch({"query":"chair"})data=search.get_dict()This code prints Naver search results.Generic search with SerpApiClientfromserpapiimportSerpApiClientquery={"q":"Coffee","location":"Austin,Texas","engine":"google"}search=SerpApiClient(query)data=search.get_dict()This class enables interaction with any search engine supported by SerpApi.comSearch Google ImagesfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe","tbm":"isch"})forimage_resultinsearch.get_dict()['images_results']:link=image_result["original"]try:print("link: "+link)# wget.download(link, '.')except:passThis code prints all the image links,
and downloads the images if you un-comment the line with wget (Linux/OS X tool to download files).This tutorial covers more ground on this topic.https://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-trainingSearch Google NewsfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe",# search search"tbm":"nws",# news"tbs":"qdr:d",# last 24h"num":10})foroffsetin[0,1,2]:search.params_dict["start"]=offset*10data=search.get_dict()fornews_resultindata['news_results']:print(str(news_result['position']+offset*10)+" - "+news_result['title'])This script prints the first 3 pages of the news headlines for the last 24 hours.Search Google ShoppingfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe",# search search"tbm":"shop",# news"tbs":"p_ord:rv",# last 24h"num":100})data=search.get_dict()forshopping_resultindata['shopping_results']:print(shopping_result['position'])+" - "+shopping_result['title'])This script prints all the shopping results, ordered by review order.Google Search By LocationWith SerpApi, we can build a Google search from anywhere in the world.
This code looks for the best coffee shop for the given cities.fromserpapiimportGoogleSearchforcityin["new york","paris","berlin"]:location=GoogleSearch({}).get_location(city,1)[0]["canonical_name"]search=GoogleSearch({"q":"best coffee shop",# search search"location":location,"num":1,"start":0})data=search.get_dict()top_result=data["organic_results"][0]["title"]Batch Asynchronous SearchesWe offer two ways to boost your searches thanks to theasyncparameter.Blocking - async=false - more compute intensive because the search needs to maintain many connections. (default)Non-blocking - async=true - the way to go for large batches of queries  (recommended)# Operating systemimportos# regular expression libraryimportre# safe queue (named Queue in python2)fromqueueimportQueue# Time utilityimporttime# SerpApi searchfromserpapiimportGoogleSearch# store searchessearch_queue=Queue()# SerpApi searchsearch=GoogleSearch({"location":"Austin,Texas","async":True,"api_key":os.getenv("API_KEY")})# loop through a list of companiesforcompanyin['amd','nvidia','intel']:print("execute async search: q = "+company)search.params_dict["q"]=companyresult=search.get_dict()if"error"inresult:print("oops error: ",result["error"])continueprint("add search to the queue where id: ",result['search_metadata'])# add search to the search_queuesearch_queue.put(result)print("wait until all search statuses are cached or success")# Create regular searchwhilenotsearch_queue.empty():result=search_queue.get()search_id=result['search_metadata']['id']# retrieve search from the archive - blockerprint(search_id+": get search from archive")search_archived=search.get_search_archive(search_id)print(search_id+": status = "+search_archived['search_metadata']['status'])# check statusifre.search('Cached|Success',search_archived['search_metadata']['status']):print(search_id+": search done with q = "+search_archived['search_parameters']['q'])else:# requeue search_queueprint(search_id+": requeue search")search_queue.put(result)# wait 1stime.sleep(1)print('all searches completed')This code shows how to run searches asynchronously.
The search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed.
The current thread that executes the search is now non-blocking, which allows it to execute thousands of searches in seconds. The SerpApi backend will do the processing work.
The actual search result is deferred to a later call from the search archive using get_search_archive(search_id).
In this example the non-blocking searches are persisted in a queue: search_queue.
A loop through the search_queue allows it to fetch individual search results.
This process can easily be multithreaded to allow a large number of concurrent search requests.
To keep things simple, this example only explores search results one at a time (single threaded).See example.Python object as a resultThe search results can be automatically wrapped in dynamically generated Python object.
This solution offers a more dynamic, fully Oriented Object Programming approach over the regular Dictionary / JSON data structure.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas"})r=search.get_object()asserttype(r.organic_results),listassertr.organic_results[0].titleassertr.search_metadata.idassertr.search_metadata.google_urlassertr.search_parameters.q,"Coffee"assertr.search_parameters.engine,"google"Pagination using iteratorLet's collect links across multiple search results pages.# to get 2 pagesstart=0end=40page_size=10# basic search parametersparameter={"q":"coca cola","tbm":"nws","api_key":os.getenv("API_KEY"),# optional pagination parameter#  the pagination method can take argument directly"start":start,"end":end,"num":page_size}# as proof of concept# urls collectsurls=[]# initialize a searchsearch=GoogleSearch(parameter)# create a python generator using parameterpages=search.pagination()# or set custom parameterpages=search.pagination(start,end,page_size)# fetch one search result per iteration# using a basic python for loop# which invokes python iterator under the hood.forpageinpages:print(f"Current page:{page['serpapi_pagination']['current']}")fornews_resultinpage["news_results"]:print(f"Title:{news_result['title']}\nLink:{news_result['link']}\n")urls.append(news_result['link'])# check if the total number pages is as expected# note: the exact number if variable depending on the search engine backendiflen(urls)==(end-start):print("all search results count match!")iflen(urls)==len(set(urls)):print("all search results are unique!")Examples to fetch links with pagination:test file,online IDEError managementSerpApi keeps error management simple.backend service error or search failclient errorIf it's a backend error, a simple error message is returned as string in the server response.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas","api_key":"<secret_key>"})data=search.get_json()assertdata["error"]==NoneIn some cases, there are more details available in the data object.If it's a client error, then a SerpApiClientException is raised.Change log2023-03-10 @ 2.4.2Change long description to README.md2021-12-22 @ 2.4.1add more search engineyoutubewalmartapple_app_storenaverraise SerpApiClientException instead of raw string in order to follow Python guideline 3.5+add more unit error tests for serp_api_client2021-07-26 @ 2.4.0add page size support using num parameteradd youtube search engine2021-06-05 @ 2.3.0add pagination support2021-04-28 @ 2.2.0add get_response method to provide raw requests.Response object2021-04-04 @ 2.1.0Add home depot search engineget_object() returns dynamic Python object2020-10-26 @ 2.0.0Reduce class name to SearchAdd get_raw_json2020-06-30 @ 1.8.3simplify importimprove package for python 3.5+add support for python 3.5 and 3.62020-03-25 @ 1.8add support for Yandex, Yahoo, Ebayclean-up test2019-11-10 @ 1.7.1increase engine parameter priority over engine value set in the class2019-09-12 @ 1.7Change  namespace "from lib." instead: "from serpapi import GoogleSearch"Support for Bing and Baidu2019-06-25 @ 1.6New search engine supported: Baidu and BingConclusionSerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more..
To enable a type of search, the field tbm (to be matched) must be set to:isch: Google Images API.nws: Google News API.shop: Google Shopping API.any other Google service should work out of the box.(no tbm parameter): regular Google search.The fieldtbsallows to customize the search even more.The full documentation is available here.