Exploring Best-Selling Books of the Last Five Years | by Scott Jones | Towards Data Science

Every week, the New York Times (NYT) publishes a list of the top ten highest-selling novels in each of the fiction and non-fiction categories. As an avid reader, I wanted to determine whether these lists might offer suggestions for future reads (i.e. whether they might cater more strongly to a particular sub-genre) and to finally answer one of the most nagging questions for me of the past one plus years: is it just me or do a large percentage of recent best-sellers contain the word “girl”?!
What follows is a quick exploration of a NYT bestseller data set that I’ve self-curated using the newspaper’s own API. This work was partially inspired by work done by others, particularly that done by Michael Tauberg (https://medium.com/@michaeltauberg/how-to-name-a-bestseller-6f7313fbb9e, https://towardsdatascience.com/book-titles-are-getting-longer-1c341fbd4829). I’ve chosen to examine two time periods: the last year, and the last five years, so as to unearth any underlying trends.
As you’d expect with a weekly publication rate, there is not much variation in the titles that appear on the list from one week to the next. As my first step I therefore removed any and all duplicates. The word cloud below shows the prevalence of each word in the remaining corpus for the preceding five years. Sure enough, we see right away that the word “Girl” dominates the diagram.
In the last year exclusively, this trend does not disappear. Instead we are left with higher frequencies of words with darker themes, such as “murder”, “war”, “lost”, “devil” and indeed “dark” itself.
In this era of sequels and prequels, we might also expect to see lengthier titles most recently, as series append modifying descriptions (think “Harry Potter and ...”). Indeed this is marginally true, as the five year period has a mean title length of 2.74 words, while the last year increases to 2.79 (a year-by-year analysis would prove most useful here).
Alongside the book titles, the API also provides a modestly thorough description of the novel. With this data, we should be able to make a first guess at classifying the book’s genre. However, as the best selling books are those that appeal to the broadest number of people, we might not expect outstanding results in this exercise.
The information content of the titles themselves is probably not dense enough to make any accurate judgments. Therefore, we’ll employ the raw text of the descriptions. As a matter of practice, anytime you are dealing with raw text, the standard preprocessing steps of tokenization, removing common (stop) words, and reducing each word to its basest form (stemming/lemmatization) must be performed. Here we do so prior to submitting the tokens to TfidfVectorizer, which converts them into numeric features based on the frequency of each token within the learned vocabulary. In this way we have built our own “book-related” corpus without the use of any previously created embeddings.
There are many widely-used and available clustering techniques. With any of these unsupervised learning methods, the toughest decision is the number of clusters to use as a starting parameter. In our case, there is certainly a limit to the number of genres that we can imagine, at least on a first cut. Again, it is unlikely that a NYT best-seller list will travel too deeply down the nested tree of possible sub-genres. (And most of the fun involved with unsupervised learning is seeing what the algorithm can come up with).
In the most common type of clustering, k-means, there are smart ways to justify this choice, such as the “elbow method”. This method attempts to select ‘k’ such that the marginal gain in variance explained by the next highest choice decreases for the first time. In other words, it selects the turning point, or “elbow” in a plot of variance explained versus number of clusters. With our dataset, up to 50 clusters, we obtain the plot below:
Already, we can see that K-means is having difficulty sorting out the clusters, as there is no discernible elbow present in the plot. For this reason and others, it is more common to use the silhouette method, which uses a distance measure (often euclidean) to assess the separation between points of a cluster. A silhouette coefficient near 1 implies good separation. However here as well, we obtain nearly linearly increasing silhouette scores with the number of clusters, suggesting that there is no clear separation being achieved.
In our five-year dataset, we have roughly 1400 unique titles. The most obvious choice left is to assume an equal split among the clusters, and select sqrt(1400)=37 as a first guess.
Having done so, we can confirm our suspicion of the poorly separated clusters by using the TSNEVisualizer, which renders vectorized corpus in two dimensions, coloured by the predetermined clusters.
Although at first glance this looks like a mess, there do seem to be some denser pockets that might truly be capturing similarities.
In contract to K-means clustering, spectral clustering is a graph-clustering technique which uses a similarity matrix to identify clusters, rather than simply a distance measure. Practically, spectral clustering is often better at separating overlapping data points; it cares less about where they lie in space, and more about their similarity in content. Such sounds quite relevant to us; we’ll use spectral clustering from here on.
Doing so, we get a distribution quite common with clustering, where the largest cluster dwarfs those that are smaller than it.
Cluster_number Count0     26133     8813     8736     6518     6417     642      6310     591      5819     4930     4832     4724     42...
How best do we visualize text clusters to learn what they contain? One of the best ways is with the Termite d3 package out of the University of Washington, which charts the prevalence, or saliency, of common words by topic. Work for this project has been primarily in Python, so we’ll constrain ourselves to that language for now. Perhaps we can rank order the words in each topic and present the median one or two words? Often this still leads to words lacking meaning, especially if the size of a cluster is small.
The nltk package has a nice method called freqDist that determines the frequency distribution of words within a string. For cluster 0, shown above to be the most populous, if we sample only those words longer than 5 characters and occurring more than 3 times, we get a list that looks like:
Cluster 0:['america', 'american', 'ancient', 'artist', 'becomes', 'between', 'british', 'criminal', 'dangerous', 'during', 'encounters', 'father', 'follows', 'forensic', 'former', 'friendship', 'herself', 'investigation', 'investigator', 'involving', 'island', 'missing', 'mysterious', 'nantucket', 'parents', 'people', 'private', 'pursues', 'search', 'sequel', 'series', 'summer', 'together', 'treasure', 'vampire', 'wedding']
There are certainly a lot of words to go from here, but they really are quite diverse and lacking consistency. Contrast that with a tighter cluster, 2, with 63 data points, and we start to see a pattern emerge:
Cluster 2:['detective', 'detectives', 'disappearance', 'husband', 'investigate', 'investigates', 'lieutenant', 'murder', 'partner', 'president', 'series']
Indeed if we access the entire descriptions for the first 5 element of cluster 2, we get:
Sally Grissom investigates the disappearance of President Harrison Tucker’s wife.D.D. Warren and Flora Dane investigate whether a pregnant woman shot and killed her husband.Detective Dave Robicheaux and his new partner Bailey Ribbons investigate the death of a young woman by crucifixion.Nick Fourcade and Annie Broussard, a husband-and-wife detective team, investigate a boy’s murder and the disappearance of his babysitter.A television producer investigates the murder of a physician and whether it was his wife who killed him.
All clearly investigative mysteries!
Finally, we can also further sub-cluster the largest cluster found above using only its description as our new corpora. Now we have sqrt(261)=16 smaller clusters. For the largest of these, we now have the following common words and a sample of some of the descriptions:
['american', 'encounters', 'finds', 'found', 'widow']A Star Wars saga. Grand Admiral Thrawn must choose between his sense of duty to the Chiss Ascendancy and loyalty to the Empire.A photographer embarks on a road trip to reconnect with three men she might have married.Maxim Trevelyan inherits several estates and beds his cleaner Alessia Demachi, an Albanian piano prodigy who has been trafficked into England.Hans-Peter Schneider pauses his ghastly deeds to seek a dead man’s gold hidden under a Miami mansion, but its caretaker’s surprising skills prove daunting.Investigations by Leaphorn, Chee and Manuelito overlap in the desert Southwest.
We still have the words “american” and “encounters”, but many less than in our initial cluster 0. However, the descriptions themselves still lack much consistency.
We have explored our New York Times best sellers dataset, unveiling a trend toward darker themes in more recent years. There also seems to be a suggestion that book title lengths are increasing. We then took a look at clustering the books based on their descriptions using both K-means and spectral clustering, and brainstormed a few ideas for ways in which to visualize the results. Although there were some common themes that emerged, it appears there is not enough information in order to make conclusive statements on genre.
Stay tuned for a follow up post where we’ll use transfer learning in order to predict the polarity (positive or negative) of Kindle book reviews!
Please find the code at https://github.com/scjones5/nyt-bestsellers.git.