Breaking Down Goodreads Dataset using Python | by Shivangi Sareen | Towards Data Science

I love reading books and am always looking out for the next one to read, even before I start the one recently bought. So, I decided to mess around with this Goodreads dataset I happened to stumble upon on Kaggle and see what book recommendations I would end up with.
It was super fun and packed with learning!
Like always, I used Jupyter notebook and set up a virtual environment using virtualenv. Check out this post for all the steps.
import syssys.path.append('./lib/python3.7/site-packages')import pandas as pdimport reimport mathdata = pd.read_csv("books.csv")
When reading the csv, we get the following error:
ParserError: Error tokenizing data. C error: Expected 10 fields in line 4012, saw 11
This is happening because, in certain rows, there are commas that lead to all the values being shifted one place to the right and hence an extra column being added.
Possible things to do:
Skip those rows entirely. But we do not know beforehand what those rows are. The Kaggle discussion area for this dataset lists those rows, however, doing away with them will end in a loss of data.
Using a text editor, replace all commas with another delimiter like ; or | and then manually go to the rows with more than one author and put commas there. This is a cumbersome process and one which involves too much manual effort. If the number of rows corrupted are too many, this solution is not feasible.
What I ended up doing — using a text editor and adding a column in the end to avoid the ParseError. I named the column extra and read the csv file using pandas.
Now we can view and data.
data.describe(include = "all")
bookID: unique identification number for each book;title: the name of the book;authors: names of the authors of the book. Multiple authors are delimited with -;average_rating: the average rating of the book;isbn: unique number to identify the book, the International Standard Book Number;isbn13: a 13-digit ISBN to identify the book, instead of the standard 11-digit ISBN;language_code: the primary language of the book;# num_pages: number of pages of the book;ratings_count: total number of ratings the book received;text_reviews_count: total number of written text reviews the book received.
5 rows out of 13719 are corrupted, i.e., they have a value in the extra field.
To find out these rows, we check which do not have value NaN in that field, i.e., have an integer value. So, I stored the index of those rows that had an integer value in the field extra.
corrupted_rows = []for index, i in data.iterrows():     if -math.inf <= i['extra'] <= math.inf:         print("index number:", index)         print("field value:", i['extra'])         print("...")corrupted_rows.append(index)
data.loc[corrupted_rows, :]
Let’s combine the multiple authors into a single field. I join them using a ; in the authors field:
data['authors'] = data['authors']+ ";" +data['average_rating']
We get a warning: SettingWithCopyWarning. This isn't an error so nothing is broken. However, it's always good to know what the warning is trying to tell us instead of brushing it off. Here's an explanation.
The authors have been combined in one column. What’s left to do now is to shift the column values, from the third column onwards (including third since we start from index 0), one space to the left. This is achieved using the shift operator where we specify how many spaces to the left (or right) need to be shifted and whether along the row or column axis.
#-1 indicates one space to the left and axis=1 specifies column axisdata.iloc[corrupted_rows, 3:] = data.iloc[corrupted_rows, 3:].shift(-1, axis = 1) 
The result of this shift is missing values from two columns: # num_pages and text_reviews_count. This is because the data types of the columns do not match those of the final shifted values.
The solution is to temporarily convert the data type of all columns to str, perform the shift and then convert them back to their original data types.
data.iloc[corrupted_rows, 3:] = data.iloc[corrupted_rows, 3:].astype(str) #convert to strdata.iloc[corrupted_rows, 3:] = data.iloc[corrupted_rows, 3:].shift(-1, axis = 1) #the shiftdata.describe(include="all")
We can now delete the last column.
 del data['extra']
We now have to convert the data types of the columns back to their original types.
pd.to_numeric automatically configures float or int data types.
data["average_rating"] = pd.to_numeric(data.average_rating) data["ratings_count"] = pd.to_numeric(data.ratings_count) data["# num_pages"] = pd.to_numeric(data.["# num_pages"])data["text_reviews_count"] = pd.to_numeric(data.["text_reviews_count"])
I want to only keep those books that are in English. First, we need a list of distinct language codes that are being used.
language_code_unique = data.language_code.unique() 
When we print the list, we see that the English language codes start with en. So, simple use of regex will help us filter the data.
#list to store the different English language codesenglish_lang_code = [] language_code_regex = re.compile(r'^en')for code in language_code_unique:    mo2 = language_code_regex.search(code)    if mo2 != None:        english_lang_code.append(code)
The output is 5 language codes that correspond to English. We then use define a function to be used directly on the dataframe.
def check_lang_code(row):     if row.language_code in english_lang_code:         return rowdata = data[data.language_code.isin(english_lang_code)]
This leaves us with books in English and reduces the number of rows to 12651. To further decrease the data size, we extract only relevant columns: title, authors, average_rating, ratings_count.
data_filtered = data.iloc[:, [1, 2, 3, 8]] 
To get a higher quality list of books, we need to set a threshold of average_rating and ratings_count. I decided to use the mean for both.
ratings_count_mean = data_filtered["ratings_count"].mean()average_rating_mean = data_filtered["average_rating"].mean()data_filtered = data_filtered[(data_filtered.average_rating > average_rating_mean)]data_filtered = data_filtered[(data_filtered.ratings_count > ratings_count_mean)]
This leaves us with a dataset of 1013 rows × 4 columns.
Using average_rating as the only factor to get a list of top rated books isn’t enough.
data_filtered.sort_values(by=['average_rating'], ascending=False) 
The list below is not right.
We need to factor in ratings_count. I created a new column weighted_rating as the multiplication of ratings_count and average_rating.
data_filtered['weighted_rating'] = data_filtered['average_rating'] * data_filtered['ratings_count']#sort in descending order of 'weighted_rating'data_filtered = data_filtered.sort_values(by='weighted_rating', ascending=False) 
Looking much better!
Clearly, Harry Potter takes the lead. Now, I’ve read the entire series and so I don’t want to include any Harry Potter books. This is an easy case of regex.
match_regex = re.compile(r’Harry Potter’)#list to store all 'Harry Potter' titlesmatched_titles = []for index, row in data_filtered.iterrows():    mo1 = match_regex.search(row.title)    if mo1 != None:        matched_titles.append(row.title)
data_without_hp = data_filtered[~data_filtered.title.isin(matched_titles)]
The top 20 books in the list:
There are books in the list that I’ve read and can easily filter out by adding to match_regex but I decided to leave it at that. The book that stood out was “The Secret Life of Bees”. I hadn’t heard of the book before and one I’m surely going to give a read!
Unfortunately, this dataset doesn’t have a genre category (horror/crime/funny/...) and a fiction/non-fiction category, which would’ve made the list far more curated and personalised (thinking about creating such a list makes me squeal with excitement)!
→ Dealing with “corrupted rows”→ Filtering out data as much as possible to get desired results and increase performance→ A major lesson is to reduce the number of for loops as much as possible, since they are slower than other methods:
defining a function and then applying it on the dataframe
filtering data within dataframe brackets
calculating function values directly
Hope you enjoyed took away some valuable insights!