A Quicker Way to Build Datasets through Web Scraping | by William Yang | Towards Data Science

If you want to skip the HTML tag digging and get straight to scraping, here’s the gist. Note that the scraper tries to do an exact match with each item in your wanted list. Otherwise, read on for a short background on webscraping, when it’s useful to scrape websites, and some challenges you may experience while scraping.
I’ve recently been doing some research into bubble tea trends in the United States. I wanted to look at changes in drink orders, when boutique and franchise bubble tea shops were opened, and customer reviews regarding these establishments. Naturally, I turned to Yelp. But a few limitations very quickly set me back; I was limited to the first 1000 businesses on the Yelp API, and I could only get three Yelp selected reviews per business.
This makes sense from a business perspective — you wouldn’t want other businesses easily snooping in on your successes and failures and iterating off that. But it also demonstrates the larger misfortunes about web scraping. On one hand, it’s a great way to obtain data for a side project on a topic we’re interested in. On the other, companies rarely want bots interacting on their websites. Even if they offer APIs, they don’t always expose every piece of information needed.
Security against webscraping has also increased. From header data verification, to CAPTCHA challenges, to machine-learning guided behavioral approaches, a lot of work has been invested to make it harder for bots to scourge websites. Lastly, most pages don’t immediately provide useful information in HTML. Instead, they are injected through JavaScript execution, and only when a “real user” has been detected.
Hence the trusty process of:
Manually navigating to a websiteHighlight a segment of interestRight click and inspect elementsCopy the HTML tag or XPath, and use a library like BeautifulSoup to parse the html
Manually navigating to a website
Highlight a segment of interest
Right click and inspect elements
Copy the HTML tag or XPath, and use a library like BeautifulSoup to parse the html
Here’s an example from my project. Let’s say you wanted to parse reviews from Yelp on a particular bubble tea shop https://www.yelp.com/biz/chun-yang-tea-flushing-new-york-flushing?osq=bubble%20tea&sort_by=date_desc (I am not affiliated with this bubble tea shop, nor with Yelp. This is a sample from my current project on analyzing bubble tea trends in the United States.)
You would first need to set up a headless browser, or emulate the header of a real browser, in order to execute any JavaScript on a dynamic webpage.
Then, connect it to a library that can pull data out of HTML and XML files.
In order to figure out the specific tag, lemon--span__373c0__3997G raw__373c0__3rKqk, that would lead me to the reviews, I followed the numbered list of steps above, highlighting a sample review, inspecting the elements, and snooping around for the tag of interest.
This actually ends up being quite brittle, since websites may change how they render their pages at any time. More importantly, this process is redundant, in the sense that we need to first find the desired tags manually before we can set up an automated webscraper.
Instead of searching for the HTML tag or the XPath, why not just automate the manual search using simple matching rules? This is the basic premise of the AutoScraper library by Alireza Mika. Let’s walk through the same task of scraping Yelp reviews, this time with the AutoScraper library.
The AutoScraper object #3 takes in a target url #1 and a list of wanted items #2, and builds some rules around the provided parameters #4. Note that it’ll only mark tags that exactly match the items of interest; thus, the review I included in the the wanted_list is the first review on the Yelp webpage in its entirety.
Upon printing results, you’ll get to see all the elements that the webscraper has found. You can fetch these results again on the same or another url with the following command.
scraper.get_result_similar(url)
You may notice that the scraper has picked up on some extraneous pieces of information. You can fine-tune its selections by first grouping them under the rules that the scraper has learned:
groups = scraper.get_result_similar(url, grouped=True)
Since groups is a dictionary, you can get the names of the rules by calling print(groups.keys())
You can then key into the dictionary using a particular rule:
groups['rule_io6e']
If the results of the specified rule accurately reflects the desired results, you can choose to keep the rule with the method call:
scraper.keep_rules('rule_io6e')
Lastly, you can persist the model to file for later use.
scraper.save('yelp-reviews')
Of course, this library is not foolproof. Click-intensive webpages, such as Google’s reviews of a business, would be rather challenging to automate without a browser emulator.
Since AutoScraper expects an exact match between items in the wanted list and items that appear on a website, it can be difficult to pinpoint a constantly updating value, such as the price of a stock.
However, as developers, part of our responsibilities is to tie solutions along a pipeline together to iterate on or create a new product. Hopefully, you can make use of this scraping library, whether in a framework like scrapy, or just to simplify your next scraping task.
Regardless of how you decide to scrape a website, do so responsibly!
Don’t bombard a website with requests. This is the number one mistake people new to webscraping do: they try to speed up scraping by sending multiple requests at a time, from multiple servers — this is a DDoS attack!Respect the robots.txt: you can usually find them by adding robots.txt to the index page. For example, https://www.yelp.com/robots.txtUse the provided API!
Don’t bombard a website with requests. This is the number one mistake people new to webscraping do: they try to speed up scraping by sending multiple requests at a time, from multiple servers — this is a DDoS attack!
Respect the robots.txt: you can usually find them by adding robots.txt to the index page. For example, https://www.yelp.com/robots.txt
Use the provided API!