Scrape Company Reviews & Ratings from Indeed in 2 Minutes | by Yasser Elsedawy | Towards Data Science

In this tutorial, I will show you how to perform web scraping using Anaconda Jupyter notebook and the BeautifulSoup library.
We’ll be scraping Company reviews and ratings from Indeed platform, and then we will export them to Pandas library dataframe and then to a .CSV file.
Let us get straight down to business, however, if you’re looking on a guide to understanding Web Scraping in general, I advise you of reading this article from Dataquest.
Let us start by importing our 3 libraries
from bs4 import BeautifulSoupimport pandas as pdimport requests
Then, let’s go to indeed website and examine which information we want, we will be targeting Ernst & Young firm page, you can check it from the following link
https://www.indeed.com/cmp/Ey/reviews?fcountry=IT
Based on my location, the country is indicated as Italy but you can choose and control that if you want.
In the next picture, we can see the multiple information that we can tackle and scrape:
1- Review Title
2- Review Body
3- Rating
4- The role of the reviewer
5- The location of the reviewer
6- The review date
However, you can notice that Points 4,5&6 are all in one line and will be scraped together, this can cause a bit of confusion for some people, but my advice is to scrape first then solve problems later. So, let’s try to do this.
After knowing what we want to scrape, we need to find out how much do we need to scrape, do we want only 1 review? 1 page of reviews or all pages of reviews? I guess the answer should be all pages!!
If you scrolled down the page and went over to page 2 you will find that the link for that page became as following:
https://www.indeed.com/cmp/Ey/reviews?fcountry=IT&start=20
Then try to go to page 3, you will find the link became as following:
https://www.indeed.com/cmp/Ey/reviews?fcountry=IT&start=4
Looks like we have a pattern here, page 2=20 , page 3 = 40, then page 4 = 60, right? All untill page 8 = 140
Let’s get back to coding, start by defining your dataframe that you want.
df = pd.DataFrame({‘review_title’: [],’review’:[],’author’:[],’rating’:[]})
In the next code I will make a for loop that starts from 0, jumps 20 and stops at 140.
1- Inside that for loop we will make a GET request to the web server, which will download the HTML contents of a given web page for us.
2- Then, We will use the BeautifulSoup library to parse this page, and extract the text from it. We first have to create an instance of the BeautifulSoup class to parse our document
3- Then by inspecting the html, we choose the classes from the web page, classes are used when scraping to specify specific elements we want to scrape.
4- And then we can conclude by adding the results to our DataFrame created before.
“I added a picture down for how the code should be in case you copied and some spaces were added wrong”
for i in range(10,140,20):     url = (f’https://www.indeed.com/cmp/Ey/reviews?fcountry=IT&start={i}')     header = {“User-Agent”:”Mozilla/5.0 Gecko/20100101 Firefox/33.0 GoogleChrome/10.0"}     page = requests.get(url,headers = header)     soup = BeautifulSoup(page.content, ‘lxml’)     results = soup.find(“div”, { “id” : ‘cmp-container’})     elems = results.find_all(class_=’cmp-Review-container’)     for elem in elems:         title = elem.find(attrs = {‘class’:’cmp-Review-title’})         review = elem.find(‘div’, {‘class’: ‘cmp-Review-text’})         author = elem.find(attrs = {‘class’:’cmp-Review-author’})         rating = elem.find(attrs = {‘class’:’cmp-ReviewRating-text’})         df = df.append({‘review_title’: title.text,          ‘review’: review.text,        ‘author’: author.text,          ‘rating’: rating.text            }, ignore_index=True)
DONE. Let’s check our dataframe
df.head()
Now, once scraped, let’s try solve the problem we have.
Notice the author coulmn had 3 differnt information seperated by (-)
So, let’s split them
author = df[‘author’].str.split(‘-’, expand=True)
Now, let’s rename the columns and delete the last one.
author = author.rename(columns={0: “job”, 1: “location”,2:’time’})del author[3]
Then let’s join those new columns to our original dataframe and delete the old author column
df1 = pd.concat([df,author],axis=1)del df1[‘author’]
let’s examine our new dataframe
df1.head()
Let’s re-organize the columns and remove any duplicates
df1 = df1[[‘job’, ‘review_title’, ‘review’, ‘rating’,’location’,’time’]]df1 = df1.drop_duplicates()
Then finally let’s save the dataframe to a CSV file
df1.to_csv(‘EY_indeed.csv’)
You should now have a good understanding of how to scrape and extract data from Indeed. A good next step for you if you are familiar a bit with web scraping it to pick a site and try some web scraping on your own.