Webscrape with Java, NodeJs & Python
By: Andrei Elekes
Tags: Java, Programming, Python, Nodejs, Software Development

What Is Web Scraping? 🤷‍♂️

Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites.

Webscraping software may access the World Wide Web directly using the Hypertext Transfer Protocol or through a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.

Web scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). Therefore, web crawling is the main component of web scraping, to fetch pages for later processing. Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be to find and copy names and phone numbers, or companies, and their URLs, to a list (contact scraping).

There are, however, some web scraping software that will automatically load and extract data from multiple pages of websites based on your requirements. It is either custom-built for a specific website or is one that can be configured to work with any website. With the click of a button, you can easily save the data available on the website to a file on your computer.

Many services offer web scraping like Scrapestorm Jp, Grepsr, and ScrapingHub. But today, I will be discussing how to build your own web scraper application using Java, NodeJs and Python.

Java WebScraper ☕️

The best library to use for Java webscraping is Jsoup.

jsoup is a Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods.

jsoup implements the WHATWG HTML5 specification, and parses HTML to the same DOM as modern browsers do.

scrape and parse HTML from a URL, file, or string

find and extract data, using DOM traversal or CSS selectors

manipulate the HTML elements, attributes, and text

clean user-submitted content against a safe white-list, to prevent XSS attacks

output tidy HTML

jsoup is designed to deal with all varieties of HTML found in the wild, from pristine and validating, to invalid tag-soup; jsoup will create a sensible parse tree.

Download the Jsoup JAR file from here and then create a java class containing the URL that you need to scrape:

import java.io.IOException;

import org.jsoup.Jsoup; public class JsoupFromStringEx { public static void main(String[] args) throws IOException { String webPage = "https://www.google.com/"; String html = Jsoup.connect(webPage).get().html(); System.out.println(html); }

}

After running the java class, the webpage data should be printed out. This is the most basic way of webscraping in Java. Of course, this does not separate the data; many functions need to be placed for the application to do so. To create a more elaborate webscraping application follow this.

NodeJs WebScraper 🕸

By using the superb tutorial here, we create a new scraper directory for this tutorial and initialize it with a package.json file by running npm init -y from the project root. Then run this command to install all the dependencies needed:

npm install axios cheerio puppeteer --save

Here’s what each one does:

Axios : Promise-based HTTP client for Node.js and the browser

: Promise-based HTTP client for Node.js and the browser Cheerio : jQuery implementation for Node.js. Cheerio makes it easy to select, edit, and view DOM elements.

: jQuery implementation for Node.js. Cheerio makes it easy to select, edit, and view DOM elements. Puppeteer: A Node.js library for controlling Google Chrome or Chromium.

When the installation is complete, create a new pl-scraper.js file in the root of your project directory and populate it with the following code:

// pl-scraper.js const axios = require('axios'); const url = 'https://www.premierleague.com/stats/top/players/goals?se=-1&cl=-1&iso=-1&po=-1?se=-1'; axios(url)

.then(response => {

const html = response.data;

console.log(html);

})

.catch(console.error);

If you run the code with node pl-scraper.js , a long string of HTML will be printed to the console.

And that’s it, you just retrieved all the data from a webpage using a NodeJs webscraper. But how can you parse the HTML for the exact data you need? Continue following Pusher’s tutorial here.

Python Webscraper 🐍

With reference to Python Docs found here, we start off by downloading lxml that is a pretty extensive library written for parsing XML and HTML documents very quickly, even handling messed up tags in the process. We will also be using the Requests module instead of the already built-in urllib2 module due to improvements in speed and readability. You can easily install both using pip install lxml and pip install requests .

Let’s start with the imports:

from lxml import html

import requests

Next, we will use requests.get to retrieve the web page with our data, parse it using the html module, and save the results in tree :

page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')

tree = html.fromstring(page.content)

(We need to use page.content rather than page.text because html.fromstring implicitly expects bytes as input.)

tree now contains the whole HTML file in a nice tree structure which we can go over two different ways: XPath and CSSSelect. In this example, we will focus on the former.

XPath is a way of locating information in structured documents such as HTML or XML documents. A good introduction to XPath is on W3Schools. There are also various tools for obtaining the XPath of elements such as FireBug for Firefox or the Chrome Inspector. If you’re using Chrome, you can right-click an element, choose ‘Inspect element’, highlight the code, right-click again, and choose ‘Copy XPath’.

After a quick analysis, we see that in our page the data is contained in two elements — one is a div with title ‘buyer-name’ and the other is a span with class ‘item-price’:

<div title="buyer-name">Carson Busses</div>

<span class="item-price">$29.95</span>

Knowing this we can create the correct XPath query and use the lxml xpath function like this:

#This will create a list of buyers:

buyers = tree.xpath('//div[@title="buyer-name"]/text()')

#This will create a list of prices

prices = tree.xpath('//span[@class="item-price"]/text()')

Let’s see what we got exactly:

print 'Buyers: ', buyers

print 'Prices: ', prices Buyers: ['Carson Busses', 'Earl E. Byrd', 'Patty Cakes',

'Derri Anne Connecticut', 'Moe Dess', 'Leda Doggslife', 'Dan Druff',

'Al Fresco', 'Ido Hoe', 'Howie Kisses', 'Len Lease', 'Phil Meup',

'Ira Pent', 'Ben D. Rules', 'Ave Sectomy', 'Gary Shattire',

'Bobbi Soks', 'Sheila Takya', 'Rose Tattoo', 'Moe Tell'] Prices: ['$29.95', '$8.37', '$15.26', '$19.25', '$19.25',

'$13.99', '$31.57', '$8.49', '$14.47', '$15.86', '$11.11',

'$15.98', '$16.27', '$7.50', '$50.85', '$14.26', '$5.68',

'$15.00', '$114.07', '$10.09']

Congratulations! We have successfully scraped all the data we wanted from a web page using lxml and Requests. We have it stored in memory as two lists. Now we can do all sorts of cool stuff with it: we can analyze it using Python, or we can save it to a file and share it with the world.

Caution ⚠️

So is it legal or illegal? Web scraping and crawling aren’t illegal by themselves. After all, you could scrape or crawl your own website, without a hitch…

In 2016, the US Congress passed its first legislation specifically to target bad bots — the Better Online Ticket Sales (BOTS) Act, which bans the use of software that circumvents security measures on ticket seller websites. Automated ticket scalping bots use several techniques to do their dirty work including web scraping that incorporates advanced business logic to identify scalping opportunities, input purchase details into shopping carts, and even resell inventory on secondary markets.

In other words, if you’re a venue, organization or ticketing software platform, it is still on you to defend against this fraudulent activity during your major on sales. But of course, this depends on where in the world you are:

The UK however, seems to have followed the US with its Digital Economy Act 2017 which achieved Royal Assent in April. The Act seeks to protect consumers in a number of ways in an increasingly digital society, including by “cracking down on ticket touts by making it a criminal offence for those that misuse bot technology to sweep up tickets and sell them at inflated prices in the secondary market.”

You can read more about this here.

To put that into perspective, companies themselves have the responsibility of protecting their own data from web scrapers as they have to invoke the law themselves. So before you go off and try to web scrape from a .gov webpage with your python program, think again!

Use Cases 〽️

Businesses use web scraping for different purposes and it varies on a case to case basis.

In eCommerce, Retailers/ marketplaces use web scraping to monitor their competitor prices and to improve their product attributes. Also, collect product reviews to do sentimental analysis. Lawyers use web scraping to see the past judgment report for their case reference. Lead generation companies use it to scrape the email address and phone numbers. Recruiters use it to collects user's profiles. Some travel companies collect data in real-time to provide live tracking details. Media companies collect trending topics and use hashtags to collect information from social media profiles. Business directories scrape complete information about the business profile, address, email, phone, products/services, working hours, Geocodes, etc.

Each business has competition in the present world, So companies scrape their competitor information regularly to monitor the movements. Government secret agencies also scrape for national securities purpose.

It's safe to say that webscaping is a big field, and you have just finished a brief tour of that field, using Java, NodeJs, and Python as your guide. You have also learned that it is illegal to scrape some sites, and you should check their terms and conditions before scraping. So do your webscraping wisely!

References 📖