The Ultimate Guide to Web Scraping with Node.js
By: Haider Imtiaz
Tags: Nodejs, Data, JavaScript, Web Scraping, Programming

JavaScript is the most famous web programming language and its famous js packages include NodeJs, AngularJs, ReactJs, and etc. Nodes is a widely use javaScript package and also as a compiler for JavaScript in command lines. We are going to use Node.js for doing the web scraping. Why web scraping is important and why we need that. Web scraping is basically used to scrap web page data. The website holds a lot of data daily, take an example of a bbc.com news site. That had news information update every minute and suppose we want to get the news in text form. Well doing manually scraping is not a solution, in that case, the web scraping tools come into action. Not only JavaScript can scrap webpage you can use any programming language and use an HTTP requests module to scrap.Getting started with web scraping can be broken down into two simple steps.

1:Requesting the web page source code using any Http requests module

2:Parsing the required data from the web page source code

This guide will walk you through the complete scraping process using the famous node.js library CheerioJs and request_process. Working through the example that is shown in this article will make you a pro in gathering data from websites. First thing first: If you don’t have NodeJs installed in your system, download the latest version from the official NodeJs site, after installing NodeJs successfully, install the required libraries using the following commands.Our target website is showing below in the image.

npm i cheerio

The first step in our coding is we need to fetch the HTML source code of the page, well by using the Axios module in node.js we can fetch the page and pass it to cheerio for parsing and extraction of the data. Check the following command to install the Axios with npm and Create a Js file, I’m making a js file with the name scrape.js.

npm i axios

In the first line of the code, we are loading the Axios module using the require(“axios”) function into a const variable. Next, we made a function name fetchHTML holding the code for fetching any website page. As you noticed inside the function, we had axios.get() function. That’s the main part of the fetch process. The get() method sends a request to the webpage for their source code and in response, the webpage returns an HTML source code that holds all the information that is currently present on the page. The next part is Grabbing all the data of the games present on the current page. Use the google chrome inspect mode to get a look at the source code of the webpage. For that just press the F12 from the keyboard.

As you saw each game data is inside a <a> tag element so for parsing this we will use the cheerio module that had a function for parsing and extracting the data we need.

If you had look at the code above, On the first line we are importing the cheerio module in our scrape.js and next we made a function name scrapSteam that will hold all the fetching part.

const searchResults = selector(“body”).find(“#search_result_container > #search_resultsRows > a”);

In that line, we are telling cheerio that the “<a>” collection is inside a div with id ‘search_resultsRows’ and this div is inside other with id ‘search_result_container’. So,’searchResults’ is an array of cheerio objects with “<a>” elements. Our next challenge is getting the title of every game, for this look for the Html code in inspect mode.

Well, now it's time to make our new function extractTitle in scraper.js check the code below.

Using the same method we can grab the release date of every game, use the inspect mode of google chrome and find the right element of date.

Our target data is inside the div tag with the class responsive_search_name_combined we had again used the find() function to find the div element with that specific class and after we find that class, we noticed that data is inside in another div so we again perform the same method and select the div with class=col search_released responsive_secondrow next, we used the text() and trim() method to grab the text form of the data. Below is a simple guide on how we grab the data inside the div element.

div[class=responsive_search_name_combined] → div[class=’col search_released responsive_secondrow’]→ Date Data

Now its time to grab the Prices of each game, Take a look at the inspect mode again and look for the element of the price of any game.

Well, the discount price and Original price is in the same div tags so that a new challenge for us lets think for a moment about how we handle that case. Let code a custom selector for the price div tag check the extended code below.

As we saw the price element is inside the two div tags, again we using the selector.find() method to select tag by tag, but now we need to get the original price and discount price separately.

Original price data is in the span tag so we just use the find() method to find that specific span tag and extract the price data. And finally, we will get the discounted price property. But notice that this value isn’t inside a specific HTML tag, so we have some different ways to get this value, but I will use a regular expression. Check the code below

First I’ll get the HTML from the cheerio object on line 1, After I’ll get the groups that match with this Regex, Then I’ll get the last group’s value. In the final, we scrap our required data from the steam website. Here is the full code below.

So this is the guide for web scraping using cheeio.js and node.js but the things do not end here. There is more we need to discover in the Data extraction world. I hope this article will help you someday. Feel free to share your opinion.