Web Scraping Metacritic Reviews using BeautifulSoup | by Adeline Ong | Towards Data Science

I decided to scrape Metacritic Pokemon reviews because I wanted to topic model negative game reviews. I was intrigued by how the games were praised by critics but evaluated badly by most gamers. Why do gamers hate Pokemon? More about that in this post. In the meantime, here’s how I scraped the reviews from the site.
**Full code at the bottom. Embedded codes (in the gray boxes) are for explanatory purposes***
Import packages. The main packages that you need are request, BeautifulSoup and pandas.
Import packages. The main packages that you need are request, BeautifulSoup and pandas.
#To get the url, and scrap the html page  import requestsfrom bs4 import BeautifulSoup#To save the reviews in a dataframe import pandas as pd
2. Figure out the web page’s Html structure. Html tags are used to mark page elements, and similar elements are usually contained within the same tags. You want to be familiar with the Html tag patterns, what info is contained within which tags, and the best way to find these tags.
The easiest way to find important tags is to use Chrome, select the text you’re interested in scraping, right-click and ‘Inspect’.
This will open a separate window containing the page’s Html. The text we highlighted is in the span tag. You can expand the span tag and see the text by clicking on the small triangle next to <span>.
BeautifulSoup (BS) can find reviews within span tags, but there are other page elements within span tags that are not reviews. A better way would be to tell BS to find an outer tag that is review-specific and then find a span tag within.
If you scroll through the page’s Html, you will notice that each review’s text is nested within div classes. These classes follow a fixed structure and repeat for every review on the page.
Take note of the review_content div class. It contains reviews contents such as username, rating score and review text. Also, note the other classes within review_content as we’ll use these to tell BS how to find review texts.
3. Parse the page using BS. Make a URL request, and parse the response into BS.
url = 'https://www.metacritic.com/game/switch/pokemon-sword/user-reviews?page=0'user_agent = {'User-agent': 'Mozilla/5.0'}response = requests.get(url, headers = user_agent)soup = BeautifulSoup(response.text, 'html.parser')
4. Create a dictionary to hold the info that you want to scrape. The keys are the feature names and the values are lists.
review_dict = {'name':[], 'date':[], 'rating':[], 'review':[]}
5. Tell BS how to find the info that you want, and append them to the dictionary’s lists. We’ll go through how to get the text review. The code for the other elements is at the bottom of this post.
#look for review_content tags since every user review has this tag for review in soup.find_all('div', class_='review_content'): #within review_content tag, look for the presence of longer reviewsif review.find('span', class_='blurb blurb_expanded'): review_dict['review'].append(review.find('span', class_=’blurb blurb_expanded').text) else: review_dict[‘review’].append(review.find('div',class_='review_body').find('span').text)
We use find all to get all review_content tags within the page. This is the same as finding all reviews on the page because each review is enclosed by this div class.
The if-else statement ensures that the text is pulled from the right tag, depending on whether the review is long or not.
Let’s look at the ‘else ’condition first. For each review_content tag, we look for a div class that is close to the span tag containing the text review. In this case, I’ve used review_body div class. Since there is only one span tag within this class, we can use find to look for the first span tag.
On to the ‘if’ condition. Longer reviews (that require users to click on ‘Expand’ to see the full text) are within a blurb blurb_expanded span class. Shorter reviews do not have this class. Since blurb blurb_expanded only appears for longer reviews, we can find it directly.
Since all reviews on Metacritic have the same elements, we can just append the desired info to lists, and they will be in-order.
6. Convert the dictionary into a dataframe.
sword_reviews = pd.DataFrame(review_dict)
Woohoo! And that’s it. If you want to scrape all the reviews, you just need to loop through all the pages, as I have done in the code below:
In this post, we created a simple web scraper using BS. We could do this because the page did not have dynamic content. For dynamic pages, you might want to try using Selenium, which can be very powerful when combined with BS.
Web scraping can be time-consuming, but it’s a gateway to online content. I personally find web scraping useful for NPL projects because it enables easy access to current conversations, and quick sensing of sentiments, which is useful for real-time evaluations of marketing/communication campaigns.
I hope this post has been helpful to you. Happy scraping!