Scraping HTML with Nokogiri and Ruby
Tags: Ruby, Web Scraping, Data, Udacity, Data Science

In this tutorial I’ll be using Ruby language and its popular library Nokogiri. So, if you do not have Ruby language and Nokogiri installed in your system then you can use Repl.it.

Repl.it allows users to write code and build apps and websites using a browser.

Let’s get started!!

What is Nokogiri?

Nokogiri is an open source software library to parse HTML and XML in Ruby. It is one of the most downloaded Ruby gems, having been downloaded over 300 million times from the rubygems.org repository.

I think that’s enough, right? Now let’s get our hands dirty with the code!

Opening a Web Page as HTML with Nokogiri and open-uri

Before we write our actual code, we need to have couple libraries included in our code in order to scrape the HTML

require 'nokogiri'

require 'open-uri'

These two lines will give us functionality of Nokogiri and Open-uri in our file.

Using Nokogiri to extract data

For this tutorial we will use Udacity’s specific page of Artificial Intelligence courses. Where all of the courses related to Artificial Intelligence are offered at Udacity.

Visit this Udacity Link and use your browser’s developer tools to inspect the page. (You can just right-click anywhere on the page and select “inspect element”.)

udacity = Nokogiri::HTML(open("https://www.udacity.com/courses/school-of-ai"))

Using this line of code will open Udacity’s website and wrap it in Nokogiri.

Nokogiri::HTML method takes the string of HTML returned by open-uri's open method and convert it into a NodeSet (aka, a bunch of nested "nodes") that we can easily play around with.

Udacity’s Artificial Intelligence courses list

You can see that title of every courses has a CSS class of card__title__nd-name we can grab all of titles using just one class name in our code. Let’s do it!!

courses = udacity.css(".card__title__nd-name")

In this line of code we are extracting all of values whose class is card__title__nd-name and all of values are stored in course variable. Here’s what the courses variable will output:

Using VS Code terminal I get this output

Woah!! 🤯

See? We just got every single line of HTML code which contains class of card__title__nd-name with only few lines of code. That’s so powerful, isn’t it?

Okay so, now we have HTML code of all courses. Let’s turn it into a readable form so we can read its values easily.

count = 1 courses.each do |course| puts "#{count}. #{course.text.strip}" count += 1 end

In these lines of code, first we are declaring variable count and making it equal to 1 so when we iterate over our array of HTML code the first value should be equal to 1.

And then we are iterating over our array courses using .each iterator, and printing out the values using puts , after every iteration we are increasing value of count using count += 1

And here is our output:

All of the courses of Udacity in the section of Artificial Intelligence

So now we have access to all of the courses of Udacity that are in the section of Artificial Intelligence.

Imagine you had task to write all of courses of Udacity that are related Artificial Intelligence, right now they only have 41 courses in this section but what if they had thousands or even hundreds? You would have to write all of those courses manually one by one. And it’d have taken you a lot of time, maybe a day to do such simple task. But with few lines of code you could you could achieve your task in matter of minutes.

Note: — the content we’re getting by scraping is all technically publicly available, as it all visible on public websites. Be careful, however, as some content may not be used without permission or licensing. Images, for instance, often belong to someone and can have a license attached — something you would probably want to look into before scraping any image src attributes from a site!

Scraping is a big topic, and it takes a lot of practice to get comfortable doing it.