The Lookout: Five Guides For Not Being Blocked While Scraping The Web
By: Abideen Opeyemi Bello Bideen
Tags: Programming, Web Scraping, Data Science, Machine Learning, Web Scrapers

1. Usage of Headless Browsers

Photo by Richard Pennystan on Unsplash

In order to decide whether or not the question came from an actual person, the trickiest websites to scrape will find subtle details like the Internet font, extension, browser cookies and execution of JavaScript. You may need your own headless browser to scrape these web sites (or have your own Scraper API!).

Tools such as Selenium and Puppeteer allow you to write a program to manipulate a real web browser which is exactly the same as a true user, in order to remove detection entirely. While this is a little bit of work rendering Selenium undetectable or Puppeteer undetectable, the easiest way to scrape websites is to make this easy.

Please note, these programming controllable browsers are highly cpu and memory intensive, and will crash occasionally, only for web scraping use where appropriate. You don’t have to use this platform for certain sites (where a quick GET request is done) so just hit certain tools if you’re blocked or you don’t have a real browser!

2. IP Rotation

Photo by Karsten Würth on Unsplash

The number one way that websites detect web scrapers is by analyzing their IP address, so most web scraping without being blocked uses a number of separate IP addresses to prevent any one IP address from being barred. To stop routing all your requests through the same IP address, you can use an IP rotation service such as the Scraper API or other proxy services to route your requests through a variety of separate IP addresses. This will cause you to scrap the majority of websites without any problems.

For places that use more sophisticated proxy blacklists, you may need to consider using residential or smartphone proxies if you are unfamiliar with what this entails you can find out our post on various forms of proxies here. At the end of the day the number of IP addresses in the world is set, and the vast majority of people surfing the Internet only get 1 (the IP address given to them by their Internet service provider for their home internet), so having said that 1 million IP addresses would allow you to surf as many as 1 million ordinary Internet users without giving rise to suspicion. This is by far the most popular method for websites to block web crawlers, so if you get blocked having more IP addresses is the first thing you can do.

3. Set Other Request Headers

Photo by Priscilla Du Preez on Unsplash

A whole host of headers would be set on actual web browsers, all of which can be tested by diligent websites to obstruct the web scraper. You should navigate to https:/httpbin.org/anything to make your scraper appear to be a real browser, then simply copy the headers you see there they are the headers the new web browser is using). Stuff like “Accept”, “Accept-Encoding”, “Accept-Language”, and “Upgrade-Insecure-Requests” being set would make your requests look like they are coming from a real server so you won’t have your web crawling interrupted.

For example, the headers from the latest Google Chrome is:

“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,

image/apng,*/*;q=0.8,application/signed-exchange;v=b3″,

“Accept-Encoding”: “gzip”,

“Accept-Language”: “en-US,en;q=0.9,es;q=0.8”,

“Upgrade-Insecure-Requests”: “1”,

“User-Agent”: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36”

You should be able to avoid being detected by 99 percent of websites by rotating through a series of IP addresses and setting proper HTTP request headers (especially User Agents).

4. Set A Real Agent

Photo by Austin Distel on Unsplash

User Agents are a special form of HTTP header that tells the website exactly what browser you are using while visiting. Any websites can review user agents and block user agent requests that are not part of a main browser. Many web scrapers do not try to customize the User Agent and are thus quickly found when looking for disabled User Agents.

Don’t be one of the makers of this! Remember to set your web crawler to be a popular user agent (you can find a list of popular user agents here). For advanced users, as most websites choose to be mentioned on Google and thus allow Googlebot in, you can even set your User Agent to the Googlebot User Agent.

Any new upgrade to Google Chrome, Safari, Firefox, etc. has a totally different user agent, but if you go years without updating the user agent on your crawlers, they can get more and more suspect. It’s crucial to note to keep the user agents you use somewhat up to date.

5. Avoid Traps by Honeypot

Photo by Priscilla Du Preez on Unsplash

Through bringing in invisible connections that only a robot can pursue, a lot of sites will attempt to find web crawlers. You need to detect when a link has the “display: zero” or “visibility: secret” CSS properties package, and else a site will be able to recognize you correctly as a programmatic scraper, fingerprint the properties of your requests and block you very quickly if they do not obey the link.

Honeypots are one of the simplest ways for smart webmasters to identify crawlers, so make sure that on each page you scrape, you conduct this search. Often, advanced webmasters may only set the color to white (or whatever color the page’s background color is), so you might want to verify whether the link has anything like “color: #fff;” or “color: #ffffff” set, as the link can also be essentially invisible.

6. Detect Website Changes

Photo by Markus Winkler on Unsplash

For various purposes, many websites modify templates and this can also cause scrapers to crack. In comparison, certain websites may have different templates (page 1 of the search results may have a different interface than page 4) in unusual areas.

This is true also for surprisingly big, less tech-savvy firms, such as large department stores that are now making the online transition. When building your scraper, you need to properly detect these changes and create constant tracking so that you know your crawler is still running (usually only counting the amount of active requests per crawl should do the trick).

Another simple way to set up testing is to write a unit test on the web on a single URL (or one URL of each kind, for example, you would want to write a unit test on the search results page on a review site, another unit test on the review page, another unit test on the main product page, etc.).

This way, using just a few requests every 24 hours or so, you can search for breaking site modifications without having to go through a complete crawl to find errors.