Package: search-engine-parser
Search Engine Parser"If it is a search engine, then it can be parsed"- some random guysearch-engine-parser is a package that lets you query popular search engines and scrape for result titles, links, descriptions and more. It aims to scrape the widest range of search engines.
View all supported engineshere.Search Engine ParserPopular Supported EnginesInstallationDevelopmentCode DocumentationRunning the testsUsageCodeCommand lineFAQCode of ConductContributionLicense (MIT)Popular Supported EnginesPopular search engines supported include:GoogleDuckDuckGoGitHubStackOverflowBaiduYouTubeView all supported engineshere.InstallationInstall from PyPi:# install only package dependenciespipinstallsearch-engine-parser# Installs `pysearch` cli  toolpipinstall"search-engine-parser[cli]"or from master:pipinstallgit+https://github.com/bisoncorps/search-engine-parserDevelopmentClone the repository:gitclonegit@github.com:bisoncorps/search-engine-parser.gitThen create a virtual environment and install the required packages:mkvirtualenvsearch_engine_parserpipinstall-rrequirements/dev.txtCode DocumentationCode docs can be found onRead the Docs.Running the testspytestUsageCodeQuery results can be scraped from popular search engines, as shown in the example snippet below.importpprintfromsearch_engine_parser.core.engines.bingimportSearchasBingSearchfromsearch_engine_parser.core.engines.googleimportSearchasGoogleSearchfromsearch_engine_parser.core.engines.yahooimportSearchasYahooSearchsearch_args=('preaching to the choir',1)gsearch=GoogleSearch()ysearch=YahooSearch()bsearch=BingSearch()gresults=gsearch.search(*search_args)yresults=ysearch.search(*search_args)bresults=bsearch.search(*search_args)a={"Google":gresults,"Yahoo":yresults,"Bing":bresults}# pretty print the result from each enginefork,vina.items():print(f"-------------{k}------------")forresultinv:pprint.pprint(result)# print first title from google searchprint(gresults["titles"][0])# print 10th link from yahoo searchprint(yresults["links"][9])# print 6th description from bing searchprint(bresults["descriptions"][5])# print first result containing links, descriptions and titleprint(gresults[0])For localization, you can pass theurlkeyword and a localized url. This queries and parses the localized url using the same engine's parser:# Use google.de instead of google.comresults=gsearch.search(*search_args,url="google.de")If you need results in a specific language you can pass the 'hl' keyword and the 2-letter country abbreviation (here's ahandy list):# Use 'it' to receive italian resultsresults=gsearch.search(*search_args,hl="it")CacheThe results are automatically cached for engine searches. You can either bypass the cache by addingcache=Falseto thesearchorasync_searchmethod or clear the engine's cachefromsearch_engine_parser.core.engines.githubimportSearchasGitHubgithub=GitHub()# bypass the cachegithub.search("search-engine-parser",cache=False)#OR# clear cache before searchgithub.clear_cache()github.search("search-engine-parser")ProxyAdding a proxy entails sending details to the search functionfromsearch_engine_parser.core.engines.githubimportSearchasGitHubgithub=GitHub()github.search("search-engine-parser",# http proxies supported onlyproxy='http://123.12.1.0',proxy_auth=('username','password'))Asyncsearch-engine-parser supportsasync:results=awaitgsearch.async_search(*search_args)ResultsTheSearchResultsafter searching:>>>results=gsearch.search("preaching to the choir",1)>>>results<search_engine_parser.core.base.SearchResultobjectat0x7f907426a280># the object supports retrieving individual results by iteration of just by type (links, descriptions, titles)>>>results[0]# returns the first <SearchItem>>>>results[0]["description"]# gets the description of the first item>>>results[0]["link"]# gets the link of the first item>>>results["descriptions"]# returns a list of all descriptions from all resultsIt can be iterated like a normal list to return individualSearchItems.Command linesearch-engine-parser comes with a CLI tool known aspysearch. You can use it as such:pysearch--enginebing--typedescriptions"Preaching to the choir"Result:'Preaching to the choir'originatedintheUSAinthe1970s.Itisavariantoftheearlier'preaching to the converted',whichdatesfromEnglandinthelate1800sandhasthesamemeaning.Origin-thefullstory'Preaching to the choir'(alsosometimesspelledquire)isofUSorigin.usage:pysearch[-h][-V][-eENGINE][--show-summary][-uURL][-pPAGE][-tTYPE][-cc][-rRANK][--proxyPROXY][--proxy-userPROXY_USER][--proxy-passwordPROXY_PASSWORD]query

SearchEngineParser

positionalarguments:queryQuerystringtosearchengineforoptionalarguments:-h,--helpshowthishelpmessageandexit-V,--versionshowprogram'sversionnumberandexit-eENGINE,--engineENGINEEnginetouseforparsingthequerye.ggoogle,yahoo,bing,duckduckgo(default:google)--show-summaryShowsthesummaryofanengine-uURL,--urlURLAcustomlinktouseasbaseurlforsearche.ggoogle.de-pPAGE,--pagePAGEPageoftheresulttoreturndetailsfor(default:1)-tTYPE,--typeTYPETypeofdetailtoreturni.efull,links,desciptionsortitles(default:full)-cc,--clear-cacheClearcacheofenginebeforesearching-rRANK,--rankRANKIDofDetailtoreturne.g5(default:0)--proxyPROXYProxyaddresstomakeuseof--proxy-userPROXY_USERProxyusertomakeuseof--proxy-passwordPROXY_PASSWORDProxypasswordtomakeuseofCode of ConductMake sure to adhere to thecode of conductat all times.ContributionBefore making any contributions, please read thecontribution guide.License (MIT)This project is licensed under theMIT 2.0 Licensewhich allows very broad use for both academic and commercial purposes.Contributors ✨Thanks goes to these wonderful people (emoji key):Ed Luff💻Diretnan Domnan🚇⚠️🔧💻MeNsaaH🚇⚠️🔧💻Aditya Pal⚠️💻📖Avinash Reddy🐛David Onuh💻⚠️Panagiotis Simakis💻⚠️reiarthur💻Ashokkumar TA💻Andreas Teuber💻mi096684🐛devajithvs💻Geg Zakaryan💻🐛Hakan Boğan🐛NicKoehler🐛💻ChrisLin🐛💻Pietro💻🐛This project follows theall-contributorsspecification. Contributions of any kind welcome!