Package: google-search-results-async
Google Search Results Async in PythonThis Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home depot, Ebay and more.. usingSerpApi.The following services are provided:Search APISearch Archive APIAccount APILocation API(Google Only)SerpApi provides ascript builderto get you started quickly.InstallationPython 3.7+pipinstallgoogle-search-resultsLink to the python package pageQuick startfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffee","location":"Austin,Texas","api_key":"<your secret api key>"})result=search.get_dict()This example runs a search about "coffee" using your secret api key.The SerpApi service (backend)searches on Google using the search: q = "coffee"parses the messy HTML responsesreturn a standardizes JSON response
The GoogleSearch classFormat the requestExecute GET http request against SerpApi serviceParse JSON response into a dictionary
Et voila..Alternatively, you can search:Bing using BingSearch classBaidu using BaiduSearch classYahoo using YahooSearch classduckduckgo using DuckDuckGoSearch classEbay using EbaySearch classYandex using YandexSearch classHomeDepot using HomeDepotSearch classGoogleScholar using GoogleScholarSearch classYoutube using YoutubeSearch classWalmart using WalmartSearchApple App Store using AppleAppStoreSearch classNaver using NaverSearch classSee theplayground to generate your code.SummaryGoogle Search Results in PythonInstallationQuick startSummaryGoogle Search API capabilityHow to set SERP API keyExample by specificationLocation APISearch Archive APIAccount APISearch BingSearch BaiduSearch YandexSearch YahooSearch EbaySearch Home depotSearch YoutubeSearch Google ScholarGeneric search with SerpApiClientSearch Google ImagesSearch Google NewsSearch Google ShoppingGoogle Search By LocationBatch Asynchronous SearchesPython object as a resultPython paginate using iteratorError managementChange logConclusionGoogle Search API capabilitySource code.params={"q":"coffee","location":"Location Requested","device":"desktop|mobile|tablet","hl":"Google UI Language","gl":"Google Country","safe":"Safe Search Flag","num":"Number of Results","start":"Pagination Offset","api_key":"Your SERP API Key",# To be match"tbm":"nws|isch|shop",# To be search"tbs":"custom to be search criteria",# allow async request"async":"true|false",# output format"output":"json|html"}# define the search searchsearch=GoogleSearch(params)# override an existing parametersearch.params_dict["location"]="Portland"# search format return as raw htmlhtml_results=search.get_html()# parse results#  as python Dictionarydict_results=search.get_dict()#  as JSON using json packagejson_results=search.get_json()#  as dynamic Python objectobject_result=search.get_object()Link to the full documentationsee below for more hands on examples.How to set SERP API keyYou can get an API key here if you don't already have one:https://serpapi.com/users/sign_upThe SerpApiapi_keycan be set globally:GoogleSearch.SERP_API_KEY="Your Private Key"The SerpApiapi_keycan be provided for each search:query=GoogleSearch({"q":"coffee","serp_api_key":"Your Private Key"})Example by specificationWe love true open source, continuous integration and Test Drive Development (TDD).
We are using RSpec to testour infrastructure around the clockto achieve the best QoS (Quality Of Service).The directory test/ includes specification/examples.Set your api key.exportAPI_KEY="your secret key"Run testmaketestLocation APIfromserpapiimportGoogleSearchsearch=GoogleSearch({})location_list=search.get_location("Austin",3)print(location_list)it prints the first 3 location matching Austin (Texas, Texas, Rochester)[{'canonical_name':'Austin,TX,Texas,United States','country_code':'US','google_id':200635,'google_parent_id':21176,'gps':[-97.7430608,30.267153],'id':'585069bdee19ad271e9bc072','keys':['austin','tx','texas','united','states'],'name':'Austin, TX','reach':5560000,'target_type':'DMA Region'},...]Search Archive APIThe search result are stored in temporary cached.
The previous search can be retrieve from the the cache for free.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas"})search_result=search.get_dictionary()assertsearch_result.get("error")==Nonesearch_id=search_result.get("search_metadata").get("id")print(search_id)Now let retrieve the previous search from the archive.archived_search_result=GoogleSearch({}).get_search_archive(search_id,'json')print(archived_search_result.get("search_metadata").get("id"))it prints the search result from the archive.Account APIfromserpapiimportGoogleSearchsearch=GoogleSearch({})account=search.get_account()it prints your account information.Search BingfromserpapiimportBingSearchsearch=BingSearch({"q":"Coffee","location":"Austin,Texas"})data=search.get_dict()this code prints baidu search results for coffee as a Dictionary.https://serpapi.com/bing-search-apiSearch BaidufromserpapiimportBaiduSearchsearch=BaiduSearch({"q":"Coffee"})data=search.get_dict()this code prints baidu search results for coffee as a Dictionary.https://serpapi.com/baidu-search-apiSearch YandexfromserpapiimportYandexSearchsearch=YandexSearch({"text":"Coffee"})data=search.get_dict()this code prints yandex search results for coffee as a Dictionary.https://serpapi.com/yandex-search-apiSearch YahoofromserpapiimportYahooSearchsearch=YahooSearch({"p":"Coffee"})data=search.get_dict()this code prints yahoo search results for coffee as a Dictionary.https://serpapi.com/yahoo-search-apiSearch EbayfromserpapiimportEbaySearchsearch=EbaySearch({"_nkw":"Coffee"})data=search.get_dict()this code prints ebay search results for coffee as a Dictionary.https://serpapi.com/ebay-search-apiSearch Home depotfromserpapiimportHomeDepotSearchsearch=HomeDepotSearch({"q":"chair"})data=search.get_dict()this code prints home depot search results for chair as Dictionary.https://serpapi.com/home-depot-search-apiSearch YoutubefromserpapiimportHomeDepotSearchsearch=YoutubeSearch({"q":"chair"})data=search.get_dict()this code prints youtube search results for chair as Dictionary.https://serpapi.com/youtube-search-apiSearch Google ScholarfromserpapiimportGoogleScholarSearchsearch=GoogleScholarSearch({"q":"Coffee"})data=search.get_dict()this code prints Google Scholar search results.Search WalmartfromserpapiimportWalmartSearchsearch=WalmartSearch({"query":"chair"})data=search.get_dict()this code prints Google Scholar search results.Search YoutubefromserpapiimportYoutubeSearchsearch=YoutubeSearch({"search_query":"chair"})data=search.get_dict()this code prints Google Scholar search results.Search Apple StorefromserpapiimportAppleAppStoreSearchsearch=AppleAppStoreSearch({"term":"Coffee"})data=search.get_dict()this code prints Google Scholar search results.Search NaverfromserpapiimportNaverSearchsearch=NaverSearch({"query":"chair"})data=search.get_dict()this code prints Google Scholar search results.Generic search with SerpApiClientfromserpapiimportSerpApiClientquery={"q":"Coffee","location":"Austin,Texas","engine":"google"}search=SerpApiClient(query)data=search.get_dict()This class enables to interact with any search engine supported by SerpApi.comSearch Google ImagesfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe","tbm":"isch"})forimage_resultinsearch.get_dict()['images_results']:link=image_result["original"]try:print("link: "+link)# wget.download(link, '.')except:passthis code prints all the images links,
and download image if you un-comment the line with wget (linux/osx tool to download image).This tutorial covers more ground on this topic.https://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-trainingSearch Google NewsfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe",# search search"tbm":"nws",# news"tbs":"qdr:d",# last 24h"num":10})foroffsetin[0,1,2]:search.params_dict["start"]=offset*10data=search.get_dict()fornews_resultindata['news_results']:print(str(news_result['position']+offset*10)+" - "+news_result['title'])this script prints the first 3 pages of the news title for the last 24h.Search Google ShoppingfromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"coffe",# search search"tbm":"shop",# news"tbs":"p_ord:rv",# last 24h"num":100})data=search.get_dict()forshopping_resultindata['shopping_results']:print(shopping_result['position'])+" - "+shopping_result['title'])this script prints all the shopping results order by review order.Google Search By LocationWith SerpApi, we can build Google search from anywhere in the world.
This code is looking for the best coffee shop per city.fromserpapiimportGoogleSearchforcityin["new york","paris","berlin"]:location=GoogleSearch({}).get_location(city,1)[0]["canonical_name"]search=GoogleSearch({"q":"best coffee shop",# search search"location":location,"num":1,"start":0})data=search.get_dict()top_result=data["organic_results"][0]["title"]Batch Asynchronous SearchesWe do offer two ways to boost your searches thanks toasyncparameter.Blocking - async=false - it's more compute intensive because the search would need to hold many connections. (default)Non-blocking - async=true - it's way to go for large amount of query submitted by batch  (recommended)# Operating systemimportos# regular expression libraryimportre# safe queue (named Queue in python2)fromqueueimportQueue# Time utilityimporttime# SerpApi searchfromserpapiimportGoogleSearch# store searchessearch_queue=Queue()# SerpApi searchsearch=GoogleSearch({"location":"Austin,Texas","async":True,"api_key":os.getenv("API_KEY")})# loop through a list of companiesforcompanyin['amd','nvidia','intel']:print("execute async search: q = "+company)search.params_dict["q"]=companyresult=search.get_dict()if"error"inresult:print("oops error: ",result["error"])continueprint("add search to the queue where id: ",result['search_metadata'])# add search to the search_queuesearch_queue.put(result)print("wait until all search statuses are cached or success")# Create regular searchwhilenotsearch_queue.empty():result=search_queue.get()search_id=result['search_metadata']['id']# retrieve search from the archive - blockerprint(search_id+": get search from archive")search_archived=search.get_search_archive(search_id)print(search_id+": status = "+search_archived['search_metadata']['status'])# check statusifre.search('Cached|Success',search_archived['search_metadata']['status']):print(search_id+": search done with q = "+search_archived['search_parameters']['q'])else:# requeue search_queueprint(search_id+": requeue search")search_queue.put(result)# wait 1stime.sleep(1)print('all searches completed')This code shows how to run searches asynchronously.
The search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed.
The current thread that executes the search is now non-blocking which allows to execute thousand of searches in seconds. The SerpApi backend will do the processing work.
The actual search result is defer to a later call from the search archive using get_search_archive(search_id).
In this example the non-blocking searches are persisted in a queue: search_queue.
A loop through the search_queue allows to fetch individual search result.
This process can be easily multithreaded to allow a large number of concurrent search requests.
To keep thing simple, this example does only explore search result one at a time (single threaded).See example.Python object as a resultThe search results can be automatically wrapped in dynamically generated Python object.
This solution offers a more dynamic solution fully Oriented Object Programming approach over the regular Dictionary / JSON data structure.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas"})r=search.get_object()asserttype(r.organic_results),listassertr.organic_results[0].titleassertr.search_metadata.idassertr.search_metadata.google_urlassertr.search_parameters.q,"Coffee"assertr.search_parameters.engine,"google"Pagination using iteratorLet's collect links accross multiple search result pages.# to get 2 pagesstart=0end=40page_size=10# basic search parametersparameter={"q":"coca cola","tbm":"nws","api_key":os.getenv("API_KEY"),# optional pagination parameter#  the pagination method can take argument directly"start":start,"end":end,"num":page_size}# as proof of concept# urls collectsurls=[]# initialize a searchsearch=GoogleSearch(parameter)# create a python generator using parameterpages=search.pagination()# or set custom parameterpages=search.pagination(start,end,page_size)# fetch one search result per iteration# using a basic python for loop# which invokes python iterator under the hood.forpageinpages:print(f"Current page:{page['serpapi_pagination']['current']}")fornews_resultinpage["news_results"]:print(f"Title:{news_result['title']}\nLink:{news_result['link']}\n")urls.append(news_result['link'])# check if the total number pages is as expected# note: the exact number if variable depending on the search engine backendiflen(urls)==(end-start):print("all search results count match!")iflen(urls)==len(set(urls)):print("all search results are unique!")Examples to fetch links with pagination:test file,online IDEError managementSerpAPI keeps error mangement very basic.backend service error or search failclient errorIf it's a backend error, a simple message error is returned as string in the server response.fromserpapiimportGoogleSearchsearch=GoogleSearch({"q":"Coffee","location":"Austin,Texas","api_key":"<secret_key>"})data=search.get_json()assertdata["error"]==NoneIn some case, there is more details availabel in the data object.If it's client error, then a SerpApiClientException is raised.Change log2021-12-22 @ 2.4.1add more search engineyoutubewalmartapple_app_storenaverraise SerpApiClientException instead of raw string in order to follow Python guideline 3.5+add more unit error tests for serp_api_client2021-07-26 @ 2.4.0add page size support using num parameteradd youtube search engine2021-06-05 @ 2.3.0add pagination support2021-04-28 @ 2.2.0add get_response method to provide raw requests.Response object2021-04-04 @ 2.1.0Add home depot search engineget_object() returns dynamic Python object2020-10-26 @ 2.0.0Reduce class name to SearchAdd get_raw_json2020-06-30 @ 1.8.3simplify importimprove package for python 3.5+add support for python 3.5 and 3.62020-03-25 @ 1.8add support for Yandex, Yahoo, Ebayclean-up test2019-11-10 @ 1.7.1increase engine parameter priority over engine value set in the class2019-09-12 @ 1.7Change  namespace "from lib." instead: "from serpapi import GoogleSearch"Support for Bing and Baidu2019-06-25 @ 1.6New search engine supported: Baidu and BingConclusionSerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more..
To enable a type of search, the field tbm (to be matched) must be set to:isch: Google Images API.nws: Google News API.shop: Google Shopping API.any other Google service should work out of the box.(no tbm parameter): regular Google search.The fieldtbsallows to customize the search even more.The full documentation is available here.