Examining the Code That Built a Web Scraping Startup
By: Mat Steininger
Tags: Startup Lessons, Python, Web Scraping, Startup, Data Science

Examining the Code That Built a Web Scraping Startup

Startups have taken the world by storm over the past few decades. We’ve seen companies expand to colossal valuations overnight just by solving basic everyday problems. AirBnb, Uber, Robinhood, and many more started just by solving simple problems with even simpler solutions. Here in this article, I’m going to discuss not only what my simple solution was, but also how I built a startup with the exact code I used to do it.

As a savvy teenager, I often found myself hustling for money anywhere I could. From fixing broken Xboxes on eBay to reselling vintage games from GameStop, I constantly found myself following the old adage of buying low and selling high.

The Problem

Now insert the world of limited edition sneakers and clothing, where resell profits are anywhere from 2 to 10 times those at retail. This is because online stores often release very popular products with a limited quantity. Then, immediately after, those same items are resold on third party platforms for much higher markups for those who couldn’t purchase in time.

Example of the reselling market with sellout times

It was no question a savvy kid like myself would find themself involved in this classic form of arbitrage. But there remained one looming problem, items released often sell out and restock unexpectedly, and especially before we get a chance to checkout. The age old solution was to mindlessly refresh the page, but what if there was a better, more time efficient solution?

The Solution

Enter my company, crepkitchen, which solved this problem by using very basic web scraping and automation. In fact web scraping has recently become an invaluable resource most companies use to gather data on competition, product improvement, and in our case notifying users about ecommerce store inventory.

Promotional image of all our customers’ successful order confirmations in the background

The solution is rather simple. Every action a user does to look at or even purchase a product on an online store can easily be automated. Here in this article, I will provide the very simple code I used to do both of the things that helped establish my company.

To just add some context and scale of this potential solution, crepkitchen generated over $90,000 in annual profit by solving this specific issue. To learn more about this business model, I gave a short business pitch at a competition at the University of Maryland and other similar links can be found here. Eventually, I ended up selling some of the bigger technology and resources, so the company is no more, but it was a valuable experience nonetheless.

Me giving a pitch at my university

The Web Scraping Code

To start off, I’ll be using Python because of the vast frameworks and libraries that are extremely simple to implement. First, we want to simulate a user actually going to the website to view a specific product and saving the HTML, for example, this sample product.

import requests

request = requests.get('https://mathewsteininger.com/sample-product')

html = request.content

Then we can use BeautifulSoup to get the specific HTML tag of the item we want. Here this will be the exact area of the site that lets the user know whether an item is in stock or not (price could also be something to monitor).

soup = BeautifulSoup(html, 'html.parser')

# Monitor this tag to check stock changes

stock = soup.find("button", {"id": "availability"})

# Monitor this tag to check price changes

price = soup.find("div", {"id": "price"})

That’s most of the work! Now we can just package that up into a while true statement, add some time delays and a notification API, and we’re done.

Basic web scraping monitor

Preventing Rate Limiting

If you ran the code continuously by now, you might run into a significant issue called rate-limiting. See companies don’t like that most of their web traffic and resources are used up by bots, so they start limiting access to IPs that access the site too often (they also check your request headers so make sure to set those). The most basic way around this is to change your IP through a series of rotating proxies (also randomly increasing your time between requests). Here, we’ll use a cool library called ProxyManager which makes our ability to switch IPs even easier.

More complex web scraping with proxies and headers

Achieving Order Automation

Now this is a bit trickier and involves using a cool tool called Selenium that allows us to simulate the actions of a user. Selenium is essentially the same as automating opening a web browser and filling out some predefined forms. Therefore, once we have determined an item is back in stock, we trigger a checkout method. Keep in mind this may be against the store’s Terms of Service, so proceed at your own risk. In our example I will just simulate navigating a simple website.

Using selenium to automate form submissions

What To Do Now?

And there you go, you’ve successfully built a series of web scrapers and automated checkout scripts that are so valuable when dealing with limited edition products that sell out very quickly. After adding many sites in your network of scrapers, you’ll soon have a viable service that users are willing to pay for. Keep in mind though that our network of web scrapers were a lot more complex and robust, but generally this is how they worked.

A Final Note

All of these tools may run into certain website specific problems because that’s the main issue with web scraping today, adapting your scripts for many sites.