Using Python to find new books to read | Towards Data Science

There are tons of books out there, but not all are worth reading. To quote the old Faber in Fahrenheit 451, the best books are the ones that “show the pores in the face of life.”
So, with an overwhelming number of books out there, how can you find the best ones?
One way is to go to a dedicated book website (such as Amazon or Goodreads). They provide filtering tools to refine search results, and display users’ reviews and average ratings to help narrow down the choices.
Alternatively, one could ask their friends or acquaintances for recommendations.
These two methods are undeniably effective.
But there is a third way to get great results: Look at the recommendations of well-known people who say the books they read played a significant role in their achievements.
And this is where this amazing website, Mostrecommendedbooks.com, comes into play.
This website shows pretty much all the books recommended by “world-class people”. Compared to its peers, it is really way ahead of the curve.
So, I decided to retrieve all its book titles1, with two purposes in mind:
Bring all these books together in one DataFrame for my fellow book lovers, andGet some insights into this valuable treasure using Python visualization libraries.
Bring all these books together in one DataFrame for my fellow book lovers, and
Get some insights into this valuable treasure using Python visualization libraries.
I also used Google Books API to get the number of pages, the publication dates, and the categories. I chose Google Books API because:
It offers a wide variety of information about a large pool of books.
The documentation is concise and easy to follow.
There are multiple query parameters (title, author, ISBN, etc.).
Public data, like the one we’d like to access, doesn’t require authorization. You are only required to provide the API key2.
(The code for scraping data and building the dataset is available on my Github. The dataset can also be found in Kaggle)
Before we get started, let’s import the Python libraries we’ll be using.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sbimport textwrap%matplotlib inline
books_df = pd.read_csv('books_clean.csv')books_df.head(10)
And this is how our dataset looks like. Let’s see how many books we got here:
books_df.shape
(3003, 8)
As you can see, we have 3003 book titles.
We also have other features such as number of recommendations, category, publication date, and number of pages.
For anyone who might be interested, I also made sure to include Google books IDs to facilitate data scraping of the same books.
Let’s start by plotting the distribution of the number of recommendations/ book:
#Histogram:plt.figure(figsize=[10,5])plt.hist(data=books_df, x='num_recommendations')plt.title('Distribution of the number of recommendations per book', size=18, fontweight='bold')plt.xlabel('Recommendations', size=15)plt.ylabel('Frequency', size=15)
The distribution is right-skewed and many books have less than 2 recommendations.
So, what are the most recommended books within this list?
To answer this question, we cannot plot all 3003 books in descending order, as this will produce an unaesthetic chart that will not convey information concisely.
So, I will (arbitrarily) take the median of the ‘num_recommendations’ unique values as a minimum threshold for the most recommended books:
#Fixing the threshold:threshold=np.median(books_df['num_recommendations'].value_counts().index)#Subset of the dataset based on the threshold:data=books_df.loc[books_df.num_recommendations >= threshold]#Combining titles and authors: data['title_y']=[','.join(i) for i in list(zip(data['title'],data['author']))]#Barplot: plt.figure(figsize = [5,10])color_base = sb.color_palette()[0]sb.barplot(data = data.sort_values(by = 'num_recommendations', ascending = False), x = 'recommender_count', y = 'title_y', color = color_base)plt.title('The Most Recommended Books', size = 18)plt.xlabel('Recommendations', fontweight = 'bold', size = 15)plt.ylabel('Titles', size = 15);
Here, we have 22 books at the absolute pinnacle of our DataFrame.
As you can see, Man’s Search for Meaning (Victor Frankl) tops the list with more than 20 recommendations, followed by Sapiens (Yuval Harari) and then Principles by Ray Dalio.
However, it appears that this plot contains many business and self-help books. We can hardly see a fiction book, let alone philosophy or another specific category.
Digging deeper into the categories within the DataFrame will certainly refine our findings and bring forth meaningful information that would help choose the best books for us.
But first, we need to know the most frequently occurring categories in our dataset.
For the sake of simplicity, I will only plot the top 20:
#Plotting the 20 most frequently occurring categories:plt.figure(figsize = [10,5])color_base = sb.color_palette()[0]sb.barplot(books_df.category.value_counts().head(20).index, books_df.category.value_counts().head(20).values, color=color_base)plt.xticks(rotation = 'vertical')plt.title('The Most Frequently Occurring Categories', fontweight = 'bold', size = 18)plt.xlabel('Categories', size = 15)plt.ylabel('Count', size = 15)
Unsurprisingly, fiction and business books are the most popular with over 400 books in each of them. Biography books are the third in the list, followed by history and science.
Having many books in a given category doesn’t mean this category is the most recommended, nor that the books in said category are the most recommended of all. There’s no accounting for taste. Some people prefer fiction, while others are more interested in economics.
For this reason, it would be reasonable to see the distribution of recommendations in each category.
This will give us an idea about the outliers and will help us spot the most recommended books within each category:
cat = books_df.category.value_counts().head(20).indexfig, ax = plt.subplots(figsize = [10,5])sb.swarmplot(data = books_df[books_df.category.isin(cat)], x = 'category', y = 'num_recommendations', palette = 'tab20')ax.set_xticklabels(ax.get_xticklabels(), rotation = 90)ax.set_title('Distribution of Recommendations by Category', fontweight = 'bold', size = 18)ax.set_xlabel('Categories',  size = 15)ax.set_ylabel('Recommendations',  fontsize = 15)
We see that most categories have books within the 1–5 recommendations range.
Meanwhile, some categories have books recommended by more than 5 people, while others have books recommended by more than 10 people. Biography, business and economics, fiction, psychology, and self-help all have numerous books with many recommendations.
Now, let’s find out the must-read books in each category.
I will only consider 9 categories and plot the top 5 based on the number of recommendations. But please note it is possible to add more categories or widen the list of books using the code below:
#Defining the categories:categories = books_df.category.value_counts().head(9).index.tolist()#Defining a function to select a subset of data:def selection(cat):    data = books_df[books_df['category'] == cat].sort_values(by =  'num_recommendations', ascending = False).head()    return datafig, ax = plt.subplots(nrows = 3, ncols = 3,figsize = [20,15])plt.suptitle('Top 5 Recommended Books in each Category', size = 30, y = 1.05, fontweight = 'bold')fig.tight_layout(h_pad = 8)fig.text(-0.02, 0.5, 'Recommendations', va = 'center', rotation='vertical', fontweight='bold', fontsize=25)fig.text(0.5, -0.06, 'Titles', va = 'center', ha = 'center', fontweight = 'bold', fontsize=25)ax = ax.flatten()color_base = sb.color_palette()[0]y = sorted(books_df['num_recommendations'].unique())i=0for cat in categories:    sb.barplot(data = selection(cat), x='title', y = 'num_recommendations', ax = ax[i], color = color_base)    ax[i].set_title(cat, fontweight = 'bold', fontsize = 18, color = 'crimson')    ax[i].set_ylabel('')    ax[i].set_xlabel('')    ax[i].set_xticklabels((textwrap.fill(x.get_text(), width=11) for x in ax[i].get_xticklabels()), fontsize = 14)    ax[i].set_yticks(np.arange(0, max(y)+1, 5))    ax[i].set_yticklabels(labels = ax[i].get_yticks(), fontsize =14)    i += 1
Now we have a variety of books we can choose from depending on our preferred genre!
Of course, as mentioned earlier, it is possible to plot more than five books. This is just an illustration to show how it can be presented.
Finally, we can go further and look at our dataset from another angle.
An author can have multiple books, and if some books have racked up many likes, some authors may likewise have more fans than others.
In other words, let’s see “who are the most recommended authors”:
#Grouping by 'recommender' and 'author' to get the unique values of #authors per each recommender: data = recommended_books.groupby(['recommender', 'author']).count().reset_index(level=1)#Plotting the 20 most recommended authors: plt.figure(figsize=[10,5])color_base = sb.color_palette()[0]sb.barplot(data['author'].value_counts().head(20).index, data['author'].value_counts().head(20).values, color=color_base)plt.xticks(rotation = 'vertical')plt.title('The Most Recommended Authors', fontweight = 'bold', size = 18)plt.xlabel('Authors', size = 15)plt.ylabel('Count', size = 15)
This is a groovy bar plot with many names that appeared earlier in the first plot.
However, the order is not the same, and some names are new.
In other words, even if some books are highly recommended, let’s not ignore the authors, as they are equally important.
Saying the recommendations from famous personalities are “the best ones ever” is not what I aimed for. There are certainly many other astonishing books that are unnoticed (or haven’t been translated into English).
Recommendations simply create the social proof that drives our choices. The greater the number of reviews or recommendations a book has, the greater the number of people who will want to read it. And when these reviews or recommendations come from the most successful among us, social proof tends to increase. Recommendations are like a compass that shows us the right direction when we are lost.
1 I’d like to thank Richard Reis and Anurag Ramdasan, co-founders of mostrecommendedbooks.com for allowing me to use their website as a main source of information.
2 Accessing public data through Google books API can be done even without the API key. I’ve tried with and without the key and it did work in both cases.