Introduction to Web Scraping
Tags: Python, Web Scraping

Finding the perfect pasta recipe, planning ahead to avoid traffic, or doing research for an essay- all of these can be done with ease if one knows how to navigate the Internet. What if one wanted to find and aggregate large sums of data from the Internet? Enter web scraping: the automated process of finding multitudes of data from websites. This revolutionary tool has helped businesses, companies, and individuals acquire the information necessary to making the right choice. By gaining a deeper understanding of what it is, how it works, and what its use cases are, one can leverage their decision-making skills in the competitive world.

What is web scraping?

Web scraping is the automated process of collecting vast amounts of HTML data from a specific set of websites or subjects. This data can be extracted from the web in general or a (few) specific website(s).

To explain it more simply: assume one was assigned a research essay on King Louis XVI. The first thing to do after receiving the topic of the research essay is sit down at a computer, search relevant keywords, and sort through the results. Once something useful is found, it can be compiled in a list of relevant links and information.

Doing something like this, or any copying and pasting for that matter, is an example of web scraping at the most infinitesimal level. More specifically, it is an example of screen scraping. The point of this simple example is this: at its largest form, web scraping is the use of a coded program to aggregate heaps of HTML data from any public website. And, it is a skill anyone can learn today.

Web scraping basics

Web scraping involves two parts: the crawler and the scraper.

The crawler, also known as the spider, crawls around the web or designated URL in pursuit of more links to explore. After the spider finds its designated information, it sends these links to the scraper. The scraper then extracts the more nitty-gritty, specific pieces of data from these pages.

Here is how it would look in the context of the previously mentioned King Louis XVI research paper. If there were two parts to conducting research, they would be:

(1) collect a list of potentially useful links for the essay (say, Wikipedia and the list of references at the bottom of the page, “King Louis XVI”).

(2) go through each link and collect more specific information that pertains to the outline.

In this simplified explanation, the crawler perfectly enacts the former, and the scraper the latter. Both work hand-in-hand, and one process always follows the other.

How would web scraping look like if I wanted to do it by myself?

In four basic steps, it would look like this. One can also follow along with the included examples of Python code and get a head start.

(1) Choose a website and make a list of the URLs of the site that you want to to get data from. Follow these primary steps.

# to get started, install these essentials from your text editor: pip install beautifulsoup4

pip install requests # now, one can import external libraries and request content from the chosen website import requests

from bs4 import BeautifulSoup

(2) Acquire the HTML of the page(s).

(3) Use locators to locate specific data in the HTML.

# split the content with BeautifulSoup and then find links soup = BeautifulSoup(result.content) links = soup.find_all("a", "title")

(4) Save the collected data. Usually, it is saved as a JSON file or CSV file, but any other structured format will do.

data = {}

for link in links:

title = link.string

data[title] = link.attrs['href']

Web scraping is simple, especially if one is working on a small project. Yet, the larger the project, the more mishaps one can run into. What if, for example, the designated websites change? One would have to keep up with the changing HTML files, and that takes work. Nonetheless, it is possible, and its uses make it all the more worth it.

Uses for web scraping

The biggest use for web scraping is price intelligence and monitoring. The tool helps companies monitor the prices of their competitors and adjust accordingly. With the amount of amount of new companies that emerge every year, it is clear that web scraping will become increasingly prominent.

Following price intelligence, research is another big industry. Data science is on the rise, and different markets need to know everything they can to thrive, not just survive. This includes product development, market trend analysis, competitor research, and more.

These are just a few of the numerous uses for web scraping.

Learn more

There exist plenty of resources that teach anyone how to web scrape. Udemy and lots of other online courses are a great place to start. And, while there is a legal gray area for how web scraping can be done, paying close attention to the courses will help one maneuver the ropes. In the end, web scraping is a multifaceted tool: it helps businesses deliver products that buyers want, the internet curate results towards the user’s needs, and companies and individuals make more statistical sense of the world. The ultimate goal is to use data to drive more informed decision-making processes at every level.