Package: web-unblocker
Web UnblockerIntroductionGetting startedInsecureRequestWarningScraping Google Search ResultsSample Python ScriptUsing Web UnblockerUsing the Same IPUsing the Same LocationSending Custom HeadersRendering and ScreenshotsScraping AmazonSample Python ScriptUsing Web UnblockerSending Custom HeadersRenderingSetting the ZIP code or the locationConclusionIntroductionWeb Unblockeris an AI-powered proxy solution capable of bypassing sophisticated anti-bot systems.Web Unblocker has the following features:ML-driven proxy managementDynamic browser fingerprintingML-powered response recognitionAuto-retry functionalityJavaScript renderingGetting startedExecute the followingcurlcommand from your terminal:curl --insecure --proxy unblock.oxylabs.io:60000 --proxy-user "USERNAME:PASSWORD" https://ip.oxylabs.ioThe output should be a random IP.Notice a few things here:--proxyorxis used for proxy. Here, we use it for the URL of Web Unblocker--insecureor the equivalent-kis required for Web Unblocker to work--proxy-useror-Uis used for the proxy user and password. If you don't have one, sign up for afree trial.If you are observing low success rates or retrieve empty content, please try adding additional"x-oxylabs-render: html"header with your request.You can find the equivalent python code ingetting_started.py:importrequestsproxies={'http':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000','https':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000'}url='https://ip.oxylabs.io'response=requests.get(url,proxies=proxies,verify=False)# disable SSL certificate verificationprint(response.text)InsecureRequestWarningOne side effect of usingverify=Falseis that you may receive warnings forInsecureRequestWarning.Add these two lines to suppress these warnings:fromrequests.packages.urllib3.exceptionsimportInsecureRequestWarningrequests.packages.urllib3.disable_warnings(InsecureRequestWarning)Scraping Google Search ResultsSample Python ScriptThe following is asample scriptthat scrapes Google search engine results.importrequestsimportbs4keyword="shoes"url=f'https://google.com/search?q={keyword}'request_result=requests.get(url)soup=bs4.BeautifulSoup(request_result.text,"lxml")search_headings=soup.find_all('h3')forinfoinsearch_headings:print(info.getText())If you run this code with Python a few times, you’ll be blocked by Google.Using Web UnblockerThe easiest solution to bypass all possible bans is to use Web Unblocker. Add the following lines to use Web Unblocker, just like you would use proxies.proxies={'http':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000','https':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000'}# Sending the proxy informationresponse=requests.get(url,proxies=proxies,verify=False)For the complete code, seegoogle_unblocked.py.Using the Same IPIf you use Web Unblocker to scrape multiple pages, you may want to maintain the same IP or geographical region.To use the same IP, send a headerX-Oxylabs-Session-Idand use any random string its value.headers={"X-Oxylabs-Session-Id":"aRandomString"}response=requests.get('https://www.google.com/search?q=shoes',verify=False,proxies=proxies,headers=headers,)Using the Same LocationInstead of using the same IP, you can also rotate the IP from a specific location. To do that, send the headerx-oxylabs-geo-locationand set its value as a country, state, city, coordinates, or radius.headers={"x-oxylabs-geo-location":"New York,New York,United States"}response=requests.get('https://www.google.com/search?q=shoes',verify=False,proxies=proxies,headers=headers,)For a complete example, seegoogle_location.pySeedocumentationfor more details.Sending Custom HeadersYou can add standard or custom headers to the request.One of the possible use cases is getting device-specific search results.headers={"User-Agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Mobile/15E148 Safari/604.1","Accept-Language":"en-US",}response=requests.get(url,proxies=proxies,verify=False,headers=headers)Rendering and ScreenshotsWeb Unblockers fully supports rendering. If you want to render a page, add the customer header as shown below:headers={"X-Oxylabs-Render":"html"}In this particular example, Google doesn't need rendering. However, this can be used to take a screenshot. Send the same header, but change the value topng.headers={"X-Oxylabs-Render":"png"}response=requests.get(url,verify=False,proxies=proxies,headers=headers,)# Save screenshot as PNG filewithopen("google_rendered.png",'wb')asf:f.write(response.content)Seegoogle_screenshots.pyfor the complete source code.Scraping AmazonThis section shows you how to scrape Amazon with Web Unblocker.We start with a simple script and will add more features.Sample Python ScriptExamine the code inbasic_script.py. First, you will notice that Amazon will not even return a response without a valid user agent.Second, upon executing the following script a few times, Amazon will block you.importrequestsurl='https://www.amazon.com/Bose-QuietComfort-45-Bluetooth-Canceling-Headphones/dp/B098FKXT8L'custom_headers={'user-agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36','accept-language':'en-US,en;q=0.9',}response=requests.get(url,headers=custom_headers)print(response.text)Using Web UnblockerThe easiest solution to bypass all the bans is to use the Web Unblocker.Add the following lines to use the Web Unlcoker, just like you would use proxies.proxies={'http':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000','https':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000'}response=requests.get(url,proxies=proxies,verify=False,headers=custom_headers)Once you have the response, you can use BeautifulSoup to extract the product title and price as follows:soup=bs4.BeautifulSoup(response.text,"lxml")product_title=soup.find('span',id="productTitle")price_element=soup.select_one("div[role='radio'] [class='a-price-whole']")print(product_title.getText())ifprice_element:print(price_element.getText())For the complete code, seeamazon_unblocked.py.Sending Custom HeadersAs shown in the Sample Python Script, you cannot scrape Amazon without at least one header—user agent.Similarly, you can send any other custom header, which will be forwarded to Amazon by Web Unblocker.url='https://www.amazon.com/Bose-QuietComfort-45-Bluetooth-Canceling-Headphones/dp/B098FKXT8L'custom_headers={'user-agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36','accept-language':'en-US,en;q=0.9',}proxies={'http':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000','https':'http://USERNAME:PASSWORD@unblock.oxylabs.io:60000'}response=requests.get(url,proxies=proxies,verify=False,headers=custom_headers)For the complete code, seeamazon_unblocked.py.RenderingUsually, sending the user agent is enough for web scraping Amazon, as shown in the previous example. If you are still facing problems, you can render the page by sending the headerX-Oxylabs-Render:custom_headers={"X-Oxylabs-Render":"html"}#...response=requests.get(url,proxies=proxies,verify=False,headers=custom_headers)The rest of the code remains the same.Seeamazon_rendering.pyfor the complete code.Setting the ZIP code or the locationUsing thex-oxylabs-geo-locationparameter value for Amazon pages will yield a result with a corresponding delivery preference setting.You can use this parameter to get correctly-localized Amazon results in a few ways. For most Amazon domains, you can send a zip/postcode or a2-letter ISO 3166-1 alpha-2 country code.Note that this is different from scraping Google, where you cannot specify zip but instead can send a City. Seedocumentationfor more details.headers={"x-oxylabs-geo-location":"11001"#New York ZIP}response=requests.get(url,verify=False,proxies=proxies,headers=headers,)ConclusionYou should now be able to scrape Google and Amazon with Web Unblocker. We have provided many examples in theamazonandgooglefolders.Learn more aboutWeb Unblocker.If you face any problems, reach out to support.