Package: yagooglesearch
yagooglesearch - Yet another googlesearchOverviewyagooglesearchis a Python library for executing intelligent, realistic-looking, and tunable Google searches.  It
simulates real human Google search behavior to prevent rate limiting by Google (the dreaded HTTP 429 response), and if
HTTP 429 blocked by Google, logic to back off and continue trying.  The library does not use the Google API and is
heavily based off thegooglesearchlibrary.  The features include:Tunable search client attributes mid searchingReturning a list of URLs instead of a generatorHTTP 429 / rate-limit detection (Google is blocking your IP for making too many search requests) and recoveryRandomizing delay times between retrieving paged search results (i.e., clicking on page 2 for more results)HTTP(S) and SOCKS5 proxy supportLeveragingrequestslibrary for HTTP requests and cookie managementAdds "&filter=0" by default to search URLs to prevent any omission or filtering of search results by GoogleConsole and file loggingPython 3.6+Terms and ConditionsThis code is supplied as-is and you are fully responsible for how it is used.  Scraping Google Search results may
violate theirTerms of Service.  Another Python Google search library had some
interesting information/discussion on it:Original issueA responseAuthor created a separateTerms and Conditions...that contained link to thisblogGoogle's preferred method is to use theirAPI.Installationpippipinstallyagooglesearchpyproject.tomlgitclonehttps://github.com/opsdisk/yagooglesearchcdyagooglesearch
virtualenv-ppython3.venv# If using a virtual environment.source.venv/bin/activate# If using a virtual environment.pipinstall.# Reads from pyproject.tomlUsageimportyagooglesearchquery="site:github.com"client=yagooglesearch.SearchClient(query,tbs="li:1",max_search_result_urls_to_return=100,http_429_cool_off_time_in_minutes=45,http_429_cool_off_factor=1.5,# proxy="socks5h://127.0.0.1:9050",verbosity=5,verbose_output=True,# False (only URLs) or True (rank, title, description, and URL))client.assign_random_user_agent()urls=client.search()len(urls)forurlinurls:print(url)Max ~400 results returnedEven though searching Google through the GUI will display a message like "About 13,000,000 results", that does not meanyagooglesearchwill find anything close to that.  Testing shows that at most, about 400 results are returned.  If you
set 400 <max_search_result_urls_to_return, a warning message will be printed to the logs.  Seehttps://github.com/opsdisk/yagooglesearch/issues/28for the discussion.Google is blocking me!Low and slow is the strategy when executing Google searches usingyagooglesearch.  If you start getting HTTP 429
responses, Google has rightfully detected you as a bot and will block your IP for a set period of time.yagooglesearchis not able to bypass CAPTCHA, but you can do this manually by performing a Google search from a browser and proving you
are a human.The criteria and thresholds to getting blocked is unknown, but in general, randomizing the user agent, waiting enough
time between paged search results (7-17 seconds), and waiting enough time between different Google searches (30-60
seconds) should suffice.  Your mileage will definitely vary though.  Using this library with Tor will likely get you
blocked quickly.HTTP 429 detection and recovery (optional)Ifyagooglesearchdetects an HTTP 429 response from Google, it will sleep forhttp_429_cool_off_time_in_minutesminutes and then try again.  Each time an HTTP 429 is detected, it increases the wait time by a factor ofhttp_429_cool_off_factor.The goal is to haveyagooglesearchworry about HTTP 429 detection and recovery and not put the burden on the script
using it.If you do not wantyagooglesearchto handle HTTP 429s and would rather handle it yourself, passyagooglesearch_manages_http_429s=Falsewhen instantiating the yagooglesearch object.  If an HTTP 429 is detected, the
string "HTTP_429_DETECTED" is added to a list object that will be returned, and it's up to you on what the next step
should be.  The list object will contain any URLs found before the HTTP 429 was detected.importyagooglesearchquery="site:twitter.com"client=yagooglesearch.SearchClient(query,tbs="li:1",verbosity=4,num=10,max_search_result_urls_to_return=1000,minimum_delay_between_paged_results_in_seconds=1,yagooglesearch_manages_http_429s=False,# Add to manage HTTP 429s.)client.assign_random_user_agent()urls=client.search()if"HTTP_429_DETECTED"inurls:print("HTTP 429 detected...it's up to you to modify your search.")# Remove HTTP_429_DETECTED from list.urls.remove("HTTP_429_DETECTED")print("URLs found before HTTP 429 detected...")forurlinurls:print(url)HTTP and SOCKS5 proxy supportyagooglesearchsupports the use of a proxy.  The provided proxy is used for the entire life cycle of the search to
make it look more human, instead of rotating through various proxies for different portions of the search.  The general
search life cycle is:Simulated "browsing" togoogle.comExecuting the search and retrieving the first page of resultsSimulated clicking through the remaining paged (page 2, page 3, etc.) search resultsTo use a proxy, provide a proxy string when initializing ayagooglesearch.SearchClientobject:client=yagooglesearch.SearchClient("site:github.com",proxy="socks5h://127.0.0.1:9050",)Supported proxy schemes are based off those supported in the Pythonrequestslibrary
(https://docs.python-requests.org/en/master/user/advanced/#proxies):httphttpssocks5- "causes the DNS resolution to happen on the client, rather than on the proxy server."  You likelydo
notwant this since all DNS lookups would source from whereyagooglesearchis being run instead of the proxy.socks5h- "If you want to resolve the domains on the proxy server, use socks5h as the scheme."  This is thebestoption if you are using SOCKS because the DNS lookup and Google search is sourced from the proxy IP address.HTTPS proxies and SSL/TLS certificatesIf you are using a self-signed certificate for an HTTPS proxy, you will likely need to disable SSL/TLS verification when
either:Instantiating theyagooglesearch.SearchClientobject:importyagooglesearchquery="site:github.com"client=yagooglesearch.SearchClient(query,proxy="http://127.0.0.1:8080",verify_ssl=False,verbosity=5,)or after instantiation:query="site:github.com"client=yagooglesearch.SearchClient(query,proxy="http://127.0.0.1:8080",verbosity=5,)client.verify_ssl=FalseMultiple proxiesIf you want to use multiple proxies, that burden is on the script utilizing theyagooglesearchlibrary to instantiate
a newyagooglesearch.SearchClientobject with the different proxy. Below is an example of looping through a list of
proxies:importyagooglesearchproxies=["socks5h://127.0.0.1:9050","socks5h://127.0.0.1:9051","http://127.0.0.1:9052",# HTTPS proxy with a self-signed SSL/TLS certificate.]search_queries=["python","site:github.com pagodo","peanut butter toast","are dragons real?","ssh tunneling",]proxy_rotation_index=0forsearch_queryinsearch_queries:# Rotate through the list of proxies using modulus to ensure the index is in the proxies list.proxy_index=proxy_rotation_index%len(proxies)client=yagooglesearch.SearchClient(search_query,proxy=proxies[proxy_index],)# Only disable SSL/TLS verification for the HTTPS proxy using a self-signed certificate.ifproxies[proxy_index].startswith("http://"):client.verify_ssl=Falseurls_list=client.search()print(urls_list)proxy_rotation_index+=1GOOGLE_ABUSE_EXEMPTION cookieIf you have aGOOGLE_ABUSE_EXEMPTIONcookie value, it can be passed intogoogle_exemptionwhen instantiating theSearchClientobject.&tbs= URL filter clarificationThe&tbs=parameter is used to specify either verbatim or time-based filters.Verbatim search&tbs=li:1time-based filtersTime filter&tbs= URL parameterNotesPast hourqdr:hPast dayqdr:dPast 24 hoursPast weekqdr:wPast monthqdr:mPast yearqdr:yCustomcdr:1,cd_min:1/1/2021,cd_max:6/1/2021See yagooglesearch.get_tbs() functionLimitationsCurrently, the.filter_search_result_urls()function will remove any url with the word "google" in it.  This is to
prevent the returned search URLs from being polluted with Google URLs.  Note this if you are trying to explicitly search
for results that may have "google" in the URL, such assite:google.com computerLicenseDistributed under the BSD 3-Clause License. SeeLICENSEfor more information.Contact@opsdiskProject Link:https://github.com/opsdisk/yagooglesearchAcknowledgementsMario Vilasfor his amazing work on the originalgooglesearchlibrary.ContributorsKennBro-https://github.com/opsdisk/yagooglesearch/pull/9ArshansGithub-https://github.com/opsdisk/yagooglesearch/pull/21pguridi-https://github.com/opsdisk/yagooglesearch/pull/38