How to Protect Your Web Application From Web Scraping.
Tags: Web Scraping, Anti Scraping, Java, React, Tomcat

Ways to Prevent Scraping

Rate Limiting : Rate limiting is used to limit the rate at which a client can make requests to the server. For example, if you configure your service to only allow 100 requests per second and a particular client exceeds that limit, then an error will be triggered. See more at www.keycdn.com.

: Rate limiting is used to limit the rate at which a client can make requests to the server. For example, if you configure your service to only allow 100 requests per second and a particular client exceeds that limit, then an error will be triggered. See more at www.keycdn.com. Identifying Automated Traffic: A utomated traffic can be identified by finding non-humanlike patterns in web traffic. A couple of indicators of automated traffic can be traffic volume (e.g., making more than 10000 requests to an endpoint in a given day) and event periodicity (e.g., making a request every minute or making requests for all 24 hours of a day). Read more about identifying automated traffic here.

utomated traffic can be identified by finding non-humanlike patterns in web traffic. A couple of indicators of automated traffic can be traffic volume (e.g., making more than 10000 requests to an endpoint in a given day) and event periodicity (e.g., making a request every minute or making requests for all 24 hours of a day). Read more about identifying automated traffic here. Known Malicious Identifiers: A simple way to protect your web app from scraping is to reject requests from known malicious entities. For example, there are many sites that maintain a list of malicious IP addresses (see some here). You can also identify a client scraping from the User-Agent. If someone is trying to scrape your web app by using a popular scraping library, it is likely that their User-Agent contains at least one of these terms.

A simple way to protect your web app from scraping is to reject requests from known malicious entities. For example, there are many sites that maintain a list of malicious IP addresses (see some here). You can also identify a client scraping from the User-Agent. If someone is trying to scrape your web app by using a popular scraping library, it is likely that their User-Agent contains at least one of these terms. Honey-Potting: Another technique to protect your app from web scraping is known as honey-potting. This is a technique where you plant links or buttons in your webpage that only a bot can access — they are not rendered in a browser — and so if someone clicks on it, then you know they are a bot!

Rate Limiting Your API

Although there are various ways to rate-limit requests to your API, in this example we will implement a Java Servlet Filter that rate-limits requests using Bucket4J. Bucket4J is a rate-limiting library based on a token-bucket algorithm.

Our Java has one endpoint, which is used to get the users from the database. With no rate limiting, the user data from this endpoint can be scraped, and therefore collected at a massive scale. So, we will implement rate limiting on this endpoint.

First, we implement a RateLimitingFilterClass (using the example from https://github.com/vladimir-bukhtoyarov/bucket4j/blob/master/doc-pages/basic-usage.md). This filter will limit the rate an IP address can make requests to our endpoint to 50 requests per second. For simplicity, IP addresses are stored in a hashmap, with IP addresses as the key and buckets as the values. Although, a more scalable solution would implement this differently, such as caching entities that have recently visited.

This filter checks if an IP address has exceeded the rate limit. If it has, a 429 status code is sent with a message, “Too many requests.” If it hasn’t, the request is then sent down the chain, which in this example, is the UserServlet:

In a tomcat web application using Java servlets, the URL specifications are declared in the web.xml file:

Rate limiting is not a fool-proof technique to protect against web scraping. For example, you must decide the appropriate entities to rate-limit. In this example, we limit IP addresses, but a single user can rotate IP addresses using proxies, thereby surpassing the rate limit, and many users can share a single IP address, which can lead to users being enforced by our protections even if they are not scraping. For these reasons, it is important to carefully configure your rate limits and to use other techniques along with rate limiting.

Identifying Automated Traffic

Another way to protect your web application from web scraping is to identify automated traffic. An entity scraping your site may be trying to circumvent rate limits by slowing their rate of requests, but they are often still identifiable through their patterns of access. Automated traffic can be identified by analyzing the traffic to your application and identifying entities with access patterns that do not seem like how a normal, human user would access your application.

For example, Buehrer, G. et al. “Classification of Automated Web Traffic.” (2009) identifies patterns that normally indicate automated traffic. A couple of which are:

Traffic volume — Example: entities that make more than 1000 requests to an endpoint in a given time period.

— Example: entities that make more than 1000 requests to an endpoint in a given time period. Event periodicity — Example: entities that make requests every minute on the minute.

Identifying Automated Traffic by Volume

One way we can identify automated traffic is by identifying entities (in this example, IP addresses) with a high volume of requests.

In this example application, we log client activity to a table, and each time the application receives a request, determine whether the client has made a suspiciously high number of requests. A scraping entity could try to circumvent the rate limiter by making requests slower than 1 request per second, but if the entity has made more than 1000 requests in an hour, the requests will be blocked by our server. Of course, these configurations should be tuned specifically for each application to avoid blocking actual users. This implementation is used for the simplicity of the example, but there are other, more scalable solutions.

We use a HighVolumeFilter class to filter out requests made by entities who have made more than 1000 requests in a given hour. See code below:

Identifying Automated Traffic by Event Periodicity

Another way you can identify automated traffic to your web application is by finding patterns in traffic that a normal human user wouldn’t normally have. For example, it is pretty unlikely that a human user will make a request every minute on the minute for over 100 consecutive requests. While this pattern in access to the web application would not be caught by the HighVolumeFilter or RateLimitingFilter, it is likely that the entity in question is scraping. So we add another filter (EventPeriodicityFilter) to the application to enforce scraping protections on entities (IP addresses) that may be trying to circumvent our other enforcements in this way. See the filter below:

Known Malicious Identifiers

Another simple way to protect your application from web scraping is by blocking access to your site from known malicious entities, such as blacklisted IP addresses or User-Agents of popular web scraping libraries.

In this example, we will block known malicious entities by storing a table of known malicious User-Agents, and when a request is made to our application, verifying that the User-Agent does not match any of the values in the table. See the code for the MaliciousIdentifierFilter below:

Honey-Potting

One clever way to protect your application from web scraping is a sneaky technique called honey-potting. A honeypot is a computer security mechanism set to detect, deflect, or, in some manner, counteract attempts at unauthorized use of information systems. The way we will use honey-potting in this article is by adding links that are invisible in the web browser to a normal user and can only be clicked programatically. This can trick bots into exposing themselves, and we can then block their access to the application.

After trying different methods adding an invisible button to my application. I noticed that puppeteer — a headless version of Google Chrome — wouldn’t allow a scraper to click on a buton with the style display: none . This would prevent them from being caught in a honey-pot. So, instead, I used opacity: 0.0 . This makes the HTML elevent invisible to people, but is still technically rendered.

If an entity clicks on the invisible link, their IP address is then saved to a database. See the updated client code below:

The IP addresses caught with honey-potting are then blocked from making further requests using the MaliciousIdentifierFilter.

Conclusion

In this article we…

Talked about how scraping can be a risk for user privacy.

Learned how we can build protections against web scraping into our application, such as rate limiting, identifying automated traffic, identifying known malicious entities, and honey-potting.

Saw examples of simple implementations of these techniques using Java Tomcat server and a React client.

See the example code on GitHub.