Cleaning Web-Scraped Data With Pandas and Regex! (Part I) | by Rohan Gupta | Towards Data Science

Now that I’ve started programming on a daily basis, I decided to invest in a laptop that would allow me to multi-task smoothly and run all my desired applications the way I’m used to.
I own a Unix-based Macbook, but its M3 processor just doesn’t cut it when I’m working on projects that require processing a lot of images or videos. I also have a Windows notebook from work with a decent i5, but after using it, I’ve realized that my needs are much higher.
However, I’m not about to just drop $2000 any machine with good specs. I actually want to understand how the components/specs of a laptop (Size, Storage, RAM, Processor, etc.) contribute to form an overall representation of the Lapton in the market at a certain price. For example, I want to derive:
The relationship between different components of the laptop and its price.
The relationship between product ratings and specific components.
A comparison of different brands and their similar products.
There are a number of insights I could derive from scraping product data, and I will eventually develop an algorithm to find the right Laptop given a number of inputs.
Thanks to a friend, I found out about the “Web Scraper” extension on Google Chrome. You can find it here. The extension helped me collect the required data for Laptops on Amazon.
This web-scraper does what most others do: it collects information we want from the page source. Websites don’t always make it easy for you to extract data from their pages, therefore you would need to clean the extracted data before you can use it for any kind of analysis.
So what do I mean by “cleaning” the data?Often, the data will have some impurities, such as NaN (empty) values, unnecessary columns, unreadable characters or spaces ( я or \s). In the following examples of my laptop dataset, you’ll see how all our data gets stored in a single column and we would need to split each set of words into separate columns.
For the Web-Scraping, you can follow this guide on how to extract product data on Amazon using the Web Scraper extension on Google Chrome. I will specifically be working with product data on Amazon, although I believe you can scrape from a lot of websites. The important thing is to find the right JSON code for your extractions (as you can see in the guide). Before you move on to the next part, Learn How to Scrape the Data. If you already have the scraped data as a CSV file, then feel free to move on.
Now that you have your scraped data as a CSV, let’s load up a Jupyter notebook and import the following libraries:
#!pip install pandas, numpy, reimport pandas as pdimport numpy as npimport re #Regex
Then upload data and read it with df = pd.read_csv('amazon.csv') . The table should look like the output below.
So here’s what I thought of the data: Not completely useless because it does contain all the required information, especially in the product_specification column. However, when we look at one instance of this column, it looks like:
Looks like it’s totally possible to separate these components into different columns for each item in my dataset. To start off, first I’ll separate my data and copy only the required columns into a new DataFrame.
data = df[['listing', 'listing-href', 'brand', 'review_count', 'sales_price', 'product_description', 'product_specification']].copy()
Now that we have our columns in data, let’s start the cleaning process.
Think like a Data Scientist, but also.. think like Marie Kondo
First, we’re going to have to drop all the rows that contain Null/NaN values. This is because we can’t run regex functions on columns that don’t have any value, we would end up with unmatching index values and it would be hard to merge our cleaned columns to the dataset later.
data = data.dropna(how='any')data = data.reset_index(drop=True)
Now that we have dropped our null values, we can start to clean the dataset. The first step would be to remove the \s, \n, \t characters. As you see in the picture above, our information is clouded within \n characters.
data = data.replace(r'\n',' ', regex=True)data = data.replace(r'\t','', regex=True)data = data.replace(r'\s\s\s','', regex=True)
Note: I used \s\s\s because after some experimentation, I noticed a pattern of three-spaced tabs. However, you can also just use \s if you don’t have such a pattern.
This next step is also extremely important. We have to drop the specific rows that don’t contain information on the following. The only problem is that it will reduce the size of your dataset significantly. However, in my case, I would say that is better than having empty values. Empty values would have to be filled out by mean, median or mode, neither of which would make sense here. Thus, I will drop the rows which don’t have my required data.
#Screen Sizedata.drop(data[~data.product_specification.str.contains("Screen Size")].index, inplace=True)#Processor typedata.drop(data[~data.product_specification.str.contains("Processor (?<![a-zA-Z:])[-+]?\d*\.?\d+\s\w+",regex=True)].index, inplace=True)#Processor Speeddata.drop(data[~data.product_specification.str.contains("GHz (?:[A-Za-z]+\s){0,}[A-Za-z]+\w+",regex=True)].index, inplace=True)#RAMdata.drop(data[~data.product_specification.str.contains("RAM (\w+\s\w+){1,}",regex=True)].index, inplace=True)#Storagedata.drop(data[~data.product_specification.str.contains("Hard Drive (\w+\s\w+\s\w+){1,}",regex=True)].index, inplace=True)#Graphic Carddata.drop(data[~data.product_specification.str.contains("Chipset Brand (\w+){0,}",regex=True)].index, inplace=True)#Graphic Card Branddata.drop(data[~data.product_specification.str.contains("Card Description (\w+){0,}",regex=True)].index, inplace=True)data = data.reset_index(drop=True)
In the lines above, when I use the ~ operator, the function will do the opposite. So when I say ~data.product_specification.str.contains, I’m actually doing a “doesn’t-contain” function and will not drop strings that contain the value inside the bracket. Inside the bracket, I specify the regex string I’m looking for. For example, if I want 0 or more words after the word “Card Description”, I can say Card Description (\w+){0,}. I will explain it further in the section after this.
Now that I know the length of my data will not be compromised, I can convert the product_specification column into a list for the regex functionality.
l = []for i in data['product_specification']:    l.append(i)
This is where I’ll explain the regex syntax and how I extracted the textual data from each column. In the lines below, I’m extracting the screen sizes of each laptop in my list and storing them in a separate list called “screen_size”.
screensize = []for words in l:    screensize.append(re.findall(r"^Screen Size (?<![a-zA-Z:])[-+]?\d*\.?\d+", words))    screen_size = []for sublist in screensize:    for item in sublist:        screen_size.append(item)
The image below shows us what screen_size looks like after the regex loop.
As you can see, for each laptop I have the word “Screen Size” along with the size (15.6, 17.3, etc) stored in the list.
So if you closely look at the regex syntax, you can see I asked for a float number after the word “Screen Size ”
The reason we can’t use \d is that these aren’t integers, but float types. Therefore I have to specify that there will be a “.” between 2 integers with \d*\.?\d+.
Let’s see another example with extraction of the processor speed for each laptop.
We repeat the steps, writing similar lines while changing the regex statement.
processors = []for words in l:    processors.append(re.findall(r"Processor (?<![a-zA-Z:])[-+]?\d*\.?\d+\s\w+", words))    processor = []for sublist in processors:    for item in sublist:        processor.append(item)
The image below shows us what processor looks like after the regex loop.
Now, if you look at the difference between this one and the last one, you can see that I added \s\w+ after the \d*\.?\d+ because now we have another word “GHz” after the float number.
Thus, it’s important to look at the data and know what type of information you’ll be collecting for each component.
We would need to perform the same kind of loop for each of the components. If you look at my gist at the end, you will find all the examples already done in there.
Note: This is only the first part! In the next post, which will come out shortly, I will store all these lists into my DataFrame df and perform more sanitary operations such as cleaning names, replacing infrequent categories, removing $ signs and units such as “GHz” or “GB”.
Below, you can find my notebook with all the code to clean this data.
Stay tuned for more — Part 2 Coming Soon.
Follow Rohan Gupta for other Data Science content and tutorials!