What is web Scraping? and how to scrap a website using node?
By: Sayyed Hammad Ali
Tags: Web Scraping, Mern Stack, Scrapping, Node, React

Web scraping is the technique of extracting data from the target websites. And a piece of code that help you to scrap data is known as scraper. In this article we will learn, with a real life example of a web scrapper. A working demo of this scraper is available for you with the source code. Let’s understand it first and then test it in real life.

For example we are searching for jobs and there are a lot of job search platforms available in market i.e. stepstone.de, indeed.com and monster.de. For example we are searching for PHP Developer jobs in Berlin. We don’t want to visit each of these job search platforms and search for the same jobs. Instead we create a job scrapper, which will do this job for us and will return us the results of these three target websites (stepstone.de, indeed.com and monster.de). How cool is that!

In this tutorial I will use Node.js as back-end and React.js as front-end application.

User Input form for searching jobs

User enters the job title and job location in the input from (shown above) and clicks the Find Jobs button. What happens behind the scenes is interesting to know. When user click the button a post request is been sent to Node server, which contain user inputs. In the node server code we create three custom URL’s using the user input for the three target websites, which looks something like that.

Link to File

Custom URL queries for Monster, Indeed and StepStone

These URL’s will only help us to scrap the job results listed on first page of target websites, for more results we can definitely research about the URL parameters which will give us more results on each page of target website.

Now we will use a npm package axios to send HTTP request to these three URL’s which will return the HTML body for the target websites as response. Now we will use an other package cheerio which is a subset of core jQuery and it will help us scrap the relevant data. And then adding the jobs into a global variable jobsData. After finish scraping from all target websites, node server return the jobsData to front-end react application. I used ES6 promises with axios to send three HTTP requests. It return queryHtml as response when all the three requests are completed. The response queryHtml is an array which contains three items queryHtml[0].data is the html of Monster and similarly queryHtml[1].data and queryHtml[2].data contains the html string of Indeed and StepStone. Then these 3 array items passed to cheerio.load to parse them into HTML, so we can scrap our required data in later steps.

Link to File

HTTP request to Monster, Indeed and StepStone

Just for reference I show you Monster.de job search results page. Here we can see the DOM elements structure for the title, post date, company and other relevant information in the browser inspect mode.

DOM element of Monster.de for Job Search Results

Function getMonsterJobs which will get two arguments. First argument $ is the DOM which is prepared by cherrio and the second argument source which will be the target website monster. You should have basic knowledge of jQuery to understand this function given below. After scraping the required information about job we Add this new Job into the jobsData global object. We will perform the same analysis for the Indeed and StepStone and will create function getIndeedJobs and getStepStoneJobs and add the jobs results in jobsData global object.

Link to File

DOM elements scraping functions for Monster, Indeed and StepStone

So lets go back to the front-end application and try to have a look how the response of the node server after scrapping is displayed in front-end. When user submit the form for job search with job title and job location the following code is executed.

Link to File

onSubmit = (e) => {

e.preventDefault();

const {jobTitle, jobLocation} = this.state;

const userInput = {jobTitle, jobLocation}

this.setState({loading: true});



axios

.post('/api/jobs', userInput)

.then(res => {

this.setState({

jobsData: res.data.jobsData,

loading: false

});

})

.catch(err => console.log(err));

}

The following code is executed in a react app in an class component named as JobSearch. It contain states jobsData and loading. On submitting the form, an HTTP post request is sent to the node server which is containing userInput (jobTitle and jobLocation). In response we get scraped jobs data. Then we update the state jobsData in our component. loading state is set to true to show the spinner for good user experience and is set back to false after we get back the response from node server. Now we have the jobsData in our state and we pass this as props to a child component JobCard (Link to File). This component displays the search results in the form of cards.

Demo and Source code are waiting for you to have a look on them and please let me know if It helped you understanding web scrapping.

About the author

I work as a Full stack developer at Lucid.Studio and I’m very interested to learn and share my knowledge with the community. If you liked my work, then connect with me on LinkedIn: Sayyed Hammad Ali.