Web Scrape Movie Database with Beautiful Soup
By: Khuyen Tran
Tags: Data Science, Beautifulsoup, Web Scraping

Web Scrape Movie Database with Beautiful Soup

Get your Database to Predict Next Trending Movie

Motivation

You want to predict what is the next trending movie with machine learning. You try to search for clean data for building a machine learning model but desperate to find one. So you think about creating your data. But you are hesitant to find your own data because you may not know much about HTML or web scraping.

Image from Pixabay

What if I tell you could scrape with Python without former knowledge of HTML? Plus it is easy to learn and implement? Beautiful Soup is the Python library for web scraping that would enable you to scrape HTML and XML files easily. You could find the documentary about the library here. As long as you know you to use Python, by following this tutorial, you will gain a significant understanding of how to create your own data.

I created a previous tutorial on how to web scrape on Wikipedia that you can find here. What is different from this tutorial is that I will introduce different functions of Beautiful Soup that would enable you to access the next element in case that the tag you want to access does not have unique attributes. You will also learn how to scrape multiple pages with the same structure at once.

Steps for Web Scraping:

Define what information you want to scrape from the website Inspect the page Start scraping with Beautiful Soup

Goals

I want to scrape The Movie Database of Ghibli Studio to find out what characteristics make a Ghibli movie better than another. After looking at the page, I decide on what data would help me to achieve my goal.

One the first page I could scrape:

Title

Url: for further web scraping

Image: maybe the animation could affect the audience’s choice?

Rank

Rating

I am also interested in scraping other information when clicking the movie link:

Summary: so I could use text analysis

Year: movies that are released earlier may not attract many audiences

Run Time: the duration matters

Director: the audience may have a higher tendency to watch the movie if the director is Hayao Miyazaki

Inspect the Page

Right-click on the page then go to inspect.

Then click on the icon shown above to choose the element on the website to inspect.

The text in HTML format can look confusing at first for those who are not familiar with it. But don’t worry, you will get the structure easily after working with a couple of them. After clicking on the icon I mentioned above, you could click on any information that you care about on the website. You want to scrape the rating first? Just click on where the rating is. You will know the exact location of the rating on HTML lines of code.

Get started with Beautiful Soup

import requests

import urllib.request

import time

from bs4 import BeautifulSoup

import numpy as np

import pandas as pd

from urllib.request import urlopen

bsObj = BeautifulSoup(html)

bsObj.prettify() html = urlopen(' https://www.themoviedb.org/list/4309' bsObj = BeautifulSoup(html)bsObj.prettify()

The outcome is a messy text in HTML format. But pay attention that the list of the movies is wrapped in unordered list <ul> tag with attribute id = “list_page_1”. Let’s extract data with find_all(tag, attribute) . Then retrieve a list of all <li> tags within <ul> tag.

movies = bsObj.find('ul', id = "list_page_1")

movies = movies.find_all('li')

len(movies)

for movie in movies:

print(movie, len(movie), "



")

The function find_all() returns a list of objects with tag specified in the parenthesis. From analyzing the length of each item in the list, we know that the length of the movie item is 3. Thus, the items with length > 0 would give us the information about the movies

movies = [movie for movie in movies if len(movie) > 0]

Scrape the First Page

In the first page, the data we want to scrape is: title, URL, image, rank, and rating

Each movie is wrapped in a different tag with the same structure. So we could start by extracting the information from the first movie to see if we get what we want. Then loop through all the movies using similar codes to extract all the information we care about.

Start with the first page. We find out that the image’s URL and title are wrapped inside the img tag. Our current tag is li . We want to find a way to access to img tag and get its attributes.

movie_1 = movies[0]

movie_1.img.attrs

Let’s break this down. .img returns the img tag that is wrapped inside the current tag. attrs allows you to access the attribute of the img tag.

Outcome:



'data-sizes': 'auto',

'data-src': 'https://image.tmdb.org/t/p/w185_and_h278_bestv2/hnYowHwLq0iUWriAHtiiCWsI2dP.jpg',

'data-srcset': 'https://image.tmdb.org/t/p/w185_and_h278_bestv2/hnYowHwLq0iUWriAHtiiCWsI2dP.jpg 1x,

'alt': 'Nausicaä of the Valley of the Wind'} {'class': ['poster', 'lazyload', 'fade'],'data-sizes': 'auto','data-src': 'https://image.tmdb.org/t/p/w185_and_h278_bestv2/hnYowHwLq0iUWriAHtiiCWsI2dP.jpg','data-srcset': 'https://image.tmdb.org/t/p/w185_and_h278_bestv2/hnYowHwLq0iUWriAHtiiCWsI2dP.jpg 1x, https://image.tmdb.org/t/p/w370_and_h556_bestv2/hnYowHwLq0iUWriAHtiiCWsI2dP.jpg 2x','alt': 'Nausicaä of the Valley of the Wind'}

Find the title

movie_1.img.attrs['alt']

Outcome:

'Nausicaä of the Valley of the Wind'

Find the link to the movie’s image and use Image to display the image to make sure the link is working

from IPython.display import Image

image_url = movie_1.find('img').attrs['data-src']

Image(url= image_url)

Extract URL by accessing to <a> tag then to its attribute href

a = movie_1.a.attrs

for value in a.values():

url = value

print(url)

Outcome:

/movie/81

The link to the movie is “https://www.themoviedb.org/movie/81". Thus, we concatenate the string “https://www.themoviedb.org" with “/movie/81” to get the URL of the first movie

Find rank by accessing div tag with class=’number’. Then access the span tag. .text returns the text inside the tag

int(movie_1.find('div', {'class':'number'}).span.text)

Perform the same steps to find the rating

float(movie_1.find_all('span',{'class':'rating'})[1].text)

Loop through all the movies using similar lines of code

Find Next Tag and Scrape Multiple Pages

Click on one of the movies in the movie list. Use the URL you found previously to extract the information about the movie such as summary, director, language, runtime, budget, revenue, and genre

If you want to access the <p> tag that includes the information you care about but the tag does not have the specific attribute to call, you could use .find_next_siblings() to access to the p tag by using the ul tag above with the same hierarchy.

Now we are ready to scrape multiple pages at once.

Put Data into DataFrame

import pandas as pd

ghibli = pd.DataFrame(list(zip(names, ranks, ratings, languages, runtimes, budgets, revenues, genres, summaries)),

columns=['name','rank','rating','language','runtime', 'budget','revenue','genre','summary'])

ghibli.head(10)

Outcome:

Awesome! Now we have the dataset that we could use for feature engineering and machine learning. There are multiple things we could do to preprocess the data, especially the genre. But I will leave this for another tutorial.

Conclusion

Congratulations! You have just learned the powerful tool that gives you the freedom to scrape most of the data. The best way to learn is to apply. Use the technique in this tutorial to scrape the data that you care about but hesitant to tackle. If you don’t know where to start yet, you could practice with The Movie Database on the category that you care about. You would be surprised by how easy it is to scrape. Plus, you will have fun when seeing what you can scrape by just a few lines of code.

You could find the Github repository here.

I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.

Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: