Draw insights from fiction books with Text Mining
By: Rafael Belokurows
Tags: R Programming, Artificial Intelligence, R, Data Science, Books

Getting Started

What you’ll need: book in PDF or TXT format

Programming language and IDE: R and the IDE of your choice

Packages we’re gonna use: tm, stopwords, tidytext, tidyverse, wordcloud2, ggplot2, and others.

The book I have chosen to analyze this time is Carrie, by Stephen King. While it’s actually not my favorite book, it was written by my favorite author of all time, Mr. Stephen King. The story of the young girl Carrie is known by many people, perhaps more than any other of his books — and that’s saying something, coming from someone who wrote It, The Shining, The Dark Tower, Under the Dome, and many other masterpieces.

Carrie, by Stephen King

In addition to that, it has the added benefit of being very short, especially for King’s standards, with less than 200 pages and around 60K words.

Loading the book and preparing the data

Basically, 80% of the time you spend doing Text Mining and NLP will be actually dedicated to obtaining, transforming, and preparing the data so it can be used with the methods you’re gonna use — and that’s basically the way doesn’t matter the method and function you choose.

First, we use readLines to import our book in TXT format and load it into a vector of 4500+ strings of characters, each string containing a paragraph.

carrie <- readLines("carrie.txt",skipNul = T,encoding = "UTF-8")

If your book is in PDF file format — way more common than TXT— you can use package pdftools, which would help you load the text fairly easily into your R environment. The command is pretty simple and it would look like this:

carrie <- pdf_text("carrie.pdf")

Next, we create an object called corpus — a collection of text documents. Some additional steps take as input that corpus we just created and that are necessary to transform and prepare the data. They are explained in more detail in the source code for this article, which you can find on my Github page. Some of those include:

filtering out the cover, dedicatory, preface, footnotes, etc. if your file has those sections.

removing undesired characters /, “, ”, — and others.

transforming the text to lowercase.

removing numbers, punctuation, etc.

For those transformations and others, I’ve used the package tm, which provides methods for data import, corpus handling, preprocessing, metadata management, and creation of term-document matrices. Another step you can’t miss while preparing your book for analysis is removing any stopwords.

Stopwords are mainly articles, pronouns, and prepositions such as “the”, “my”, “he”, “she”, “for”, etc. that usually don’t add any value themselves to the text, they’re there to connect other types of words. We use a well-established database of stopwords for the English language obtained from package stopwords, which has 5 different sources of stopwords for more than 40 languages.

stopwords::stopwords("en", source = "stopwords-iso")

Some of the words considered “stopwords” for the English language

When crossing the dataframe of 1298 stopwords with our corpus of 4500+ paragraphs, we end up with only words important for a thorough but meaningful analysis of the book.

Last but not least, we perform a procedure called tokenization, which breaks the text into words so then we can analyze them individually. There are other types of tokenization: by sentence, by paragraph, and others, check more about it here.

carrie %>% unnest_tokens(input = text, output = word)

text tokenized by word

Now we got something that looks promising to really begin our analysis.

Part 2: The fun stuff

The most basic step when analyzing a book, in my opinion, is looking into the most frequent words contained in the text. Let’s plot the top 10.

A few observations:

Carrie : it’s common and obvious that the main character’s name is the most mentioned word of the book. If you feel that word doesn’t add any value to your analysis, it’s possible (and easy) to remove it so it doesn’t appear in your plots.

: it’s common and obvious that the main character’s name is the most mentioned word of the book. If you feel that word doesn’t add any value to your analysis, it’s possible (and easy) to remove it so it doesn’t appear in your plots. Momma : pretty indicative of how messed up the relationship between Carrie and her mom was and how violent were their interactions.

: pretty indicative of how messed up the relationship between Carrie and her mom was and how violent were their interactions. White : “of course, it’s their last name” you might say since the last name of the two main characters is bound to appear a lot. But one of the most memorable and impressive things about this book is the contrast between white (purity) and red (love, anger, danger) and I have no question at all that the surname and the number of times the word appear are no coincidence at all.

: “of course, it’s their last name” you might say since the last name of the two main characters is bound to appear a lot. But one of the most memorable and impressive things about this book is the contrast between white (purity) and red (love, anger, danger) and I have no question at all that the surname and the number of times the word appear are no coincidence at all. Blood: I personally thought this word was going to be mentioned a bit more, but it makes sense since there are only two main events in which blood is involved (the very beginning and the very ending of the book). But don’t get me wrong, this is a bloody/gory story.

Analyzing only single words makes it easy to overlook important relations, so another way of aiding your analysis is breaking the text down into pairs of words, what we’ll call bigrams. In doing that it’s possible to see better how words are related and identify common pairs, amongst other things.

Looking past the obvious “Carrie White” and “Margaret White” combinations, the third bigram shows a very interesting phenomenon common in books written by Stephen King:

Creepy writing style (but that’s one of the reasons we love him)

If you ever read a book by Stephen King, especially the ones belonging to the horror genre, you probably remember a part like this where he pauses the story for a bit to give us an often creepy look into what the character’s thinking or feeling. Stephen King at his best!

Word Cloud

Another way of visualizing text is a Word Cloud. Although there are several authors now advocating against it — and for good reasons, it is consensus that a Word Cloud is still very useful. I usually like to construct a Word Cloud when working with text since 1.it is easy 😅 and 2. it shows unmistakenly the most frequent and important words in the text.

Using package wordcloud2 and our dataframe of frequent words, we plot 200 words of those. The intensity of the red color and the size of the font are greater for more frequent words.

wordcloud::wordcloud2(data = words[1:200,], size = 1.6,

shape = "oval",

rotateRatio = 0.5,

color= rev(cartography::carto.pal("red.pal",n1=20))

)

Word Cloud of the most frequent words of the book

Correlation between words

Now we want to determine the most relevant correlations between our 10 most frequent words and any other words of the book. For that, we take a few steps.

First, we find associations of words with correlation greater than 0.15 for our 10 words. Then we select the top 5 most correlated words to our 10.

wordassociation=findAssocs(x=dtm,terms= head(words$word,10),corlimit = 0.15) association = as.data.frame(unlist(wordassociation)) %>%

tibble::rownames_to_column(var = "word") %>%

rename(corr = `unlist(wordassociation)`) %>%

tidyr::separate(col=word,sep="([.])",into=c("word1","word2")) %>%

mutate(word1 = factor(word1,levels=head(words$word,10)),wordno = as.numeric(word1)) %>%

group_by(word1) %>%

slice(seq_len(5)) %>%

arrange(word1, desc(corr)) %>%

mutate(row = row_number()) %>%

ungroup() ggplot(association, aes(corr, reorder(word2,corr),fill=word1)) +

geom_col(show.legend = FALSE) +

facet_wrap(~word1,scales="free_y")+

theme(panel.grid.major.x = element_blank(),plot.title = element_text(hjust = 0.5)) +

ggtitle("Relationship of top words") + xlab("Correlation") + ylab("Words")+ scale_fill_manual(values = cores)

Words more correlated with our top 10 most frequent words

For instance, the word “blood”, relates more strongly with “expiate”, “pig”, “pour”, “coppery” and “awful”. All of those make sense based on the story of the book and the properties of the blood itself, so we’re doing well. 😀

Network of bigrams

Another intuitive and eye-popping way of visualizing bigrams is a network plot, which you can make with ggraph package.

bigram_graph <- carrie_bigrams %>%

filter(n > 5) %>%

graph_from_data_frame()

ggraph(bigram_graph,layout = "fr") +

geom_edge_link(color="red") +

geom_node_point(color="red") +

geom_node_text(aes(label = name), vjust = 1, hjust = 1)

Network graph of the relationships between pairs of words

This method takes as input our dataframe of tokenized bigrams (pairs) and links those pairs of words that have a high correlation. For instance, “carrie” is directly linked with screamed, white, looked, and “tommy ross”, all of those words that appear many times in the book next to it.

Other relationships are pretty apparent and make a lot of sense, such as “police x station”, “root x beer”, “dance x floor”, since those are pairs that appear several times together in the text.

Final remarks

That concludes our first part of the analysis with methods and comments on how to prepare your data, visualize the text in a more general way, and look for relationships within the data.

There are certainly many other ways of analyzing and visualizing text but those were some of the most interesting I wanted to try for this book. Stay tuned for the next articles and analysis of more books.

People don’t get better, they just get smarter. When you get smarter you don’t stop pulling the wings off flies, you just think of better reasons for doing it. Stephen King.

Shoutout to the great people that wrote these articles. I definitely couldn’t have done it without those amazing pieces of information.