From Resource Management to Data Science: Building Real-Time Crawler
By: Julius Cerniauskas
Tags: Web Crawling, Data Scraping, Web Crawler, Real Time Crawler, Web Scraping

From Resource Management to Data Science: Building Real-Time Crawler

Interested in building and implementing a large-scale web crawler? Learn from our mistakes, experience, and advice, which will help you to create a robust tool for web data acquisition.

Photo by Markus Spiske on Unsplash

At Oxylabs we happened to build a large-scale web crawler almost by accident. As with any development project that people just happen to stumble upon — we made a lot of mistakes and learned a lot of lessons along the way. We’re hoping our story could be of use to anyone interested in creating tools for data acquisition.

We started out as proxy providers offering services to all who wanted to perform large-scale data gathering projects. As those who have delved into this industry know all too well, proxies are prone to being banned when they are used to scrape search engine results and complex e-commerce websites.

We noticed that some users had better success rates and proxy longevity than others even when scraping identical targets. Over time, a lot of knowledge on web scraping was acquired. In turn, we sought to provide value by sharing any relevant information. Just like we hope to do so now.

Humble beginnings

Unfortunately, even the best guidelines cannot protect against the constantly evolving nature of anti-scraping measures and the need for a consistent flow of data. Proxies still ended up getting blocked. Replacing them drove the costs up for both sides of the bargain.

Blocked proxies became a painful experience for both sides. Due to blocked IPs, companies needed to slow down their data acquisition operations while we needed to constantly procure more proxies. Luckily, a way to resolve our woes appeared in the most unforeseen manner: one of our data analysts had learned the basics of Python.

Photo by Divide By Zero on Unsplash

Not this type of Python.

Building the prototype

One user had had a long-standing struggle with e-commerce websites. Whether they were just unlucky or something had gone awry during the development process of their crawler, the proxies we provided seemed to get consistently blocked. Internally, an idea to develop our own scraper had been floating about for a rather long time but no one ever sat down and actually developed it. Until one of our data analysts had decided to take matters into his own hands.

As with any project developed with a very basic understanding of the toolkit yet a lot of passion, it was a mess, held together by Flask API, the request library, and prayers to ancient snake gods. Sure, it was a mess, but it worked. In fact, our prototype did the job quite well although it did have hiccups that only became apparent later on. Thus, one late evening in the year 2017 our newest solution was born and we went on to call it Real-Time Crawler.

Lessons in planning and development practices

Unfortunately, while our solution worked, the client for whom our Real-Time Crawler was initially developed did not take us up on the offer. But we did have a working prototype which showed a lot of promise. Our thinking was that we could further optimize this tool to get data from large e-commerce websites and search engines more efficiently than before. By upholding best practices that we developed with time, we could squeeze the most out of each IP address and avoid potential blocks. What if instead of offering resources to acquire a product we were to offer the product itself? Hopefully, no one would have to struggle with the tricky and complex process of data acquisition anymore.

While our prototype functioned well, it was still unfit for widespread use: one solitary server hosted the entire process, could only handle a few requests at a time and, worst of all, the acquired data was stored on the server file system as html files. When our tool was first developed, we didn’t focus on long-term large-scale implementation as it arose out of hope that we could resolve one particular challenge.

We realized that if we were to put through any amount of stress it would break its back in seconds. But the hope was there and it was enough. Our C-level executives and senior development teams sat down with our data analyst and started drawing plans for the future. We would begin anew with a better plan.

“Once more unto the breach, dear friends, once more”

And so serious development of our Real-Time Crawler began. Several features needed to be implemented:

A better request and delivery handling

A queue system

A better server infrastructure

Fortunately, few challenges arose with the last two points. Our queue system was developed and Real-Time Crawler was deployed into many different servers through the use of APIs and MySQL. Of course, these changes did not happen overnight. While initial implementation was rather quick, a good part of nearly 3 years had been dedicated to ironing out the details and adding additional features.

A better request and data delivery system did not come as easily. Yet, we had to develop one quickly as users had troubles implementing our initial build.

Queries and delivery: an uphill battle of our own making

We started off with only one way to query and retrieve data — callback. Users would send a payload detailing their request to our servers and would be able retrieve the requested data after the job is finished. Each payload would be assigned a unique job ID for retrieval purposes. Additionally, we implemented callback URLs which we would ping (if provided) in order to smooth out the process.

Source: Oxylabs’ design team

Callback, however, had one issue we were not able to predict: it was fairly difficult to implement for testing purposes. To no one’s surprise, it is a hard bargain to convince someone that your tool will be mutually beneficial if the intended audience cannot easily test it. But we wanted to deliver the data requested in the most efficient way possible. What’s more, most businesses in this industry already have in-house web crawling solutions. Providing value to these companies was going to be significantly more difficult than we had anticipated.

Our sales and marketing teams had to work twice as hard. A lot of effort and hours could have been saved if we had paid slightly more attention to the development culture around us — what languages, APIs, frameworks and such were widely used.

Reliability versus e-commerce

We knew we would be butting heads with one endless challenge — layout and anti-botting changes. Scraping the most challenging and most wanted search engines and e-commerce websites wasn’t going to be easy. Neither of these would use the same layouts forever or be willing to give up their data easily. All of this means that a large part of our resources were and still are dedicated to understanding how anti-scraping measures were implemented and what we could do about it.

We had acquired a lot of experience and know-how on the data acquisition industry just by providing our usual business services. A strong technical foundation allowed us to foresee certain challenges and implement workarounds before they arose. Yet, there were two issues our experience could not resolve: data storage and ease-of-implementation.

Dealing with data storage issues

Data storage became a larger issue than expected. A part of our blunder was starting out with just one method — callback. Callback seems like a nice way to handle data at the outset as it is theoretically very simple. Unfortunately, the queried data has to go somewhere and stay there until someone retrieves it. Retrieval, as is often the case, will be delayed. Therefore that data has to be stored for days until someone comes to pick it up.

We had implemented a 24 hour (often, users have even more time) policy as it seemed reasonable. Inevitably, storing large swaths of data for 24+ hours puts a lot of strain on any storage infrastructure. For this reason we decided to remove redundant data queries by creating caches, optimize databases and simply gear up on better hardware. But these solutions can only go so far. Additionally, e-commerce data is time-sensitive and 24-hour old data might already be past its expiry date. We needed to find better ways for users to send queries and receive data.

Turning our focus towards providing better delivery methods and easier ways to implement our tool brought us great success. We were certain that Real-Time Crawler would be mutually beneficial but we needed a way to reduce the barrier to entry.

Trials, ease-of-implementation and new delivery methods

Trial issues branched out of the same initial development process. As mentioned earlier, implementing Real-Time Crawler was rather difficult which made trials and testing painful. One of our solutions was to develop additional methods for our tool. As it says on the tin of our tool, Real-Time was the second method to be created. Its implementation is slightly easier but it relies on an open connection to constantly relay queried data.

Source: Oxylabs’ design team

Real-Time brought us two benefits — it was slightly easier to test and reduced the strain on data storage facilities. Real-Time brought with it a revolution as it was possible for others to see it in action without rebuilding their entire infrastructure. But it still wasn’t good enough as the long-term implementation still required users to modify their current crawling solutions. Another solution was needed.

SuperAPI was the final form of Real-Time Crawler. It allowed users to implement our web scraping solution without reworking anything. They would set our tool as their proxy server in their own web crawling tool. Everything else would work just as if it had been implemented through any other method. Finally, users could implement our solution without putting them through a complex process of changing what was already working for them.

Three delivery methods were now available:

Callback

Real-Time

SuperAPI

Each of them had their own benefits and drawbacks. Most users still utilize the old-school callback as it is reliable and simple to use, if a little difficult to get started with. We developed even more delivery methods later on as we focused more on providing better testing opportunities. Now, improving stability would be our newest goal.

Maintaining stability

Our final challenge is maintaining stability. Every tool breaks down every once in a while and most other processes are prepared to handle it. In data acquisition, especially e-commerce related, breakdown in the process can be catastrophic.

Stability in data acquisition is a constant tug-of-war between websites and developers. Websites might update their layouts both for user experience reasons and as a practice against data scraping. Each layout change breaks the data acquisition process and requires some time for redevelopment.

Anti-scraping algorithm improvements are worse. Just like layout changes, they will break any tool but the redevelopment process will be significantly more difficult. Reverse engineering will be required which means that without an awesome team of developers and a little luck, returning back to working order might take a lot of time and resources. Unfortunately, there are no long-term solutions that allow anyone to maintain stability indefinitely, only best practices.

Over time we had learned two things about maintaining stability. First, there has to be a dedicated crisis response team where everyone is ready to work around the clock until the issue is resolved. Every hour is precious. Second, after the crisis has been resolved, look for patterns in how the data acquisition process breaks down. Patch according to your predictions and maybe you will avoid similar issues down the road.

Finalizing the concept

While we are always building up and improving our web crawling solution, the main concept remains the same over time. Here’s a basic outline of how our Real-Time Crawler works:

Payload sent to API. Job logged and ID set. Set into the queue system. Each queue can be consumed by many different RTC instances/servers. The job is acquired by a specific instance. Fingerprint set and checked (according to initial request). Before the job is initiated, our [master] proxy rotator handles and assigns suitable IPs. Job initiated and HTML downloaded. Used IPs are sent into cooldown. If the job fails, Real-Time Crawler retries with different settings until successful. Data is parsed and prepared for retrieval.

Steps to success: lessons in data gathering

Our tool has been up and running for quite some time now. Along the way from its humble beginnings where string and sinew held everything together to the Skynet-like machine overlord that Real-Time Crawler is now, we learned many lessons:

Create tools with one eye set upon the future, the other towards the markets. Analyze which languages, APIs, frameworks are the most popular. It will save you a lot of time and costs.

Building industry-leading software takes time. We faced (and still do) hundreds of challenges over the years and there were times when we thought Real-Time Crawler might not be as great as we first thought.

Find a way to offer a simple testing environment. Often you will have to convince businessmen, not developers.

Write simple to understand tutorials and provide plenty of success examples.

Ensure that your tool is scalable and its features do not cause issues in other areas (e.g. data storage).

Data storage will inevitably become an issue. Create processes to save space — build caches, acquire more storage centers.

Have a dedicated crisis response team as breakdowns are inescapable.

Analyze each break down. Every crisis will, at the very least, provide clues for possible future breakdowns. Patch according to the clues.

The end?

Large-scale data gathering is an endless process as the landscape of the Internet keeps changing. We are still improving our Real-Time Crawler by adding more targets, increasing stability and applicability. Our newest fork, Universal Real-Time Crawler, is still a budding project that we hope to scale to the same heights as its predecessor.

It took us many years and numerous mistakes to build a successful large scale crawling tool. We invite everyone to learn from us. By following the lessons we learned along the way, you will make any data acquisition project less costly and time-consuming.