Lovecraft with NLP: Tokenisation and Word Counts | Towards Data Science

This is the second blog post in my series in which I analyse the works of H. P. Lovecraft through the lens of Natural Language Processing. In the first post, we kicked things off by analysing the overall sentiment of the stories. We can’t postpone this any longer, we have to talk about the basis of all NLP analysis: tokenisation. I thought it would be fun to pair this with a particular question, so in this post, we are going to find which words Lovecraft used the most in each story, and in his literary work combined.
One of the things you remember the most about reading Lovecraft is the odd language. He tends to use lots of negative words, horror, madness, depression, especially with adjectives: unnamable, indescribable, unspeakable horrors everywhere, right? There is something about this section from The Outsider (1921):
God knows it was not of this world — or no longer of this world — yet to my horror I saw in its eaten-away and bone-revealing outlines a leering, abhorrent travesty on the human shape; and in its mouldy, disintegrating apparel an unspeakable quality that chilled me even more.
that screams Lovecraft. On Arkham Archivist (the website from where I downloaded the text), there is a separate page dedicated to word counts, people submitted suggestions and the person who collected the stories counted them. I was curious to see if this notion can be observed “scientifically”. Are the stories as negative as we thought? What are the most used adjectives, are they “horrible” and “unknown” and “ancient”? Are verbs about knowledge and/or going crazy? Does he ever use the word “woman”? Well, let’s find out!
We are going to start with a quick theory background, apply some practical prep-work on the text, have a look at a couple of examples in spaCy, and then finally get on with word counting.
First, let’s have a look at a few concepts that we are going to use in this project.
tokenisation: a kind of document segmentation technique that breaks unstructured (text) data in small pieces of data that can be counted as discrete elements. In our analysis, individual tokens are going to be words, but that’s not necessarily the case, a token can be a paragraph, a sentence, a part of the word, or even characters.
bag-of-words: an unordered aggregated representation of a larger volume of text, which can be a document, chapter, paragraph, sentence, etc.... The grammar, punctuation, and word order from the original text are ignored, the only thing that’s kept is the unique words and a number attached to them. That number can be the frequency with which the word occurs in the text, or it can be a binary 0 or 1, simply measuring whether the word is in the text or not.
normalisation: in the NLP token context, normalisation is a group of techniques that take the original tokens as they appear in the text, and convert them to another form. This form can be the root of the word, for example, when we count the word “house” in a text, we probably want to also include “houses” and “House”, if it’s capitalised because it is at the beginning of a sentence. There are many different approaches, the one that we are going to use is lemmatisation, for example stemming and even taking the lower-case version of the text are also normalisations.
lemmatisation: the process of finding the lemma of the words. Lemmas are basically the dictionary forms of words. For example, the lemma of “houses” is “house”, “better” becomes “good”, “thought” becomes “think”, etc.
stop words: words that are so frequent in a language that their presence in a text does not contain much meaning regarding the topic of the document. For example, “a”, “why”, “do”, etc. For certain NLP tasks (like ours), it makes sense to ignore these words.
When I first did the text pre-work for this project, I suspected that I will have to circle back and do some additional tweaking, which turned out to b true — as soon as I started playing with the tokenisation, I ran into problems. For example, would you guess that these two sentences:
“Why wouldn’t we?”
“Why wouldn‘t we?”
will be lemmatised differently? The reason is the apostrophes. Generally, you want the “n’t” part of a word to be separated from the base. So “isn’t” becomes “is” and “not”. But it does not work with all apostrophes. Similar issues with hyphens, dashes, quotation marks — you wouldn’t imagine how many different characters exist for the same exact purpose!
(Note: Lovecraft almost never used contractions because he saw them as a sure sign of lack of proper education. However, he sprinkles them in sometimes in dialogues, to show — and also tell — how uneducated that person is. He did use a lot of dashes though.)
So before we do the investigation, we need to hunt down these weird characters. I think it’s important to talk a bit about these steps, even if it might seem like a bit boring, because (in my personal experience) a lot of NLP guides skip it entirely and just start with a clean text. There might be methods for this in large NLP libraries like NLTK, it might be a Unicode decoding issue, but to me, manually adjusting the text seemed like the most logical solution.
I started with analysing the characters in one story, The Dunwich Horror (1928), assumed to be saved in the text variable. Excluded whitespace and made it lower-case:
text = text.lower()text = text.replace('\n','')text = text.replace('\x0c','')text = text.replace(' ','')
Then you can create a string of unique characters like so:
unique_characters = set(text)
If you want to print them as a string, you can simply join the set
unique_characters_in_string = ''.join(set(text))
In the case of Dunwich, these were the unique characters:
'-2wdj0i”m8uüosczävèr‖l)q,4b76e’n—a1(:é.ftpk?x‘hg!;93―y5'
Then, I created a string called regular_characters, which included letters from the English alphabet, numbers, punctuation, brackets, regular quotation marks and apostrophes. We loop through them and remove the regular characters from the unique characters, so what we are left with are going to be the irregular characters.
for character in regular_characters:    try:        unique_characters.remove(character)    except:        pass
These are the characters that are left:
'ä', 'è', 'é', 'ü', '—', '―', '‖', '‘', '’', '”'
and yes,
'—' == '―'
is FALSE for some reason, and at that point, we already excluded what I thought to be the regular hyphen and dash! Similar issue with quotation marks, ‘‖’ being one of them. (That’s not a capital I, those are two vertical lines that were quotation marks in the text.) The non-English Latin letters are OK, their usage is legit in the text, in case you are wondering:
ä: “Iä! Shub-Niggurath!”
è: “Bibliothèque Nationale in Paris”
é: ”De Vigenère‘s Traité des Chiffres”
ü: “Klüber‘s Kryptographik”
I repeated this process for all the stories and collected all the weird characters that I wanted to tweak. I ended up with this replace dictionary:
replace_dict = {}replace_dict['*'] = ''replace_dict['—'] = '–'replace_dict['―'] = ''replace_dict['‖'] = '\"'replace_dict[' ̳'] = ''replace_dict['”'] = '"'replace_dict['‘'] = '\''replace_dict['’'] = '\''
And looped through all the text files, applied the replacements:
for replace_char in replace_dict:     text = raw_text.replace(\        replace_char,  replace_dict[replace_char])
and saved them as new TXT files. They should work properly. All right, on to the real thing now.
We start by installing the NLP library we are going to work with: spaCy.
pip install spacy
We also need to separately install a language model that we are going to use. There are models available for a number of languages, three different ones for English. Most of the guides I found recommend starting with en_core_web_sm, and I figured that would do for a simple project, but after some trial and error, I decided to useen_core_web_lg. (We will see an example of why later on.)
To install the model:
python -m spacy download en_core_web_lg
Please note that it might take some time to download and build the large model.
Now, we can set up an nlp object that is going to be our NLP model, and create a doc object that is the processed version of the string input:
nlp = spacy.load(“en_core_web_lg”)doc = nlp(“At midnight the doorbell rang, startling him fearfully.”)
doc is a Doc object, it’s basically a container of tokens. So at this point, we already have our text tokenised! Doc keeps all the information from the original text like whitespace, so you can re-build the text from it.
The tokens in a Doc object are Token objects, with a long list of attributes you can request, see the full list here. For now, we are going to use three attributes:
text: simply the text of the token as it appears in the text
lemma_: lemma of the token
pos_: the part-of-speech tag of the token (for example noun, adjective, verb, etc...)
To analyse our sentence, we can list the token attributes:
for token in doc:     print(token.text, token.lemma_, token.pos_)
Which will print out this:
At at ADPmidnight midnight NOUNthe the DETdoorbell doorbell PROPNrang ring VERB, , PUNCTstartling startle VERBhim -PRON- PRONfearfully fearfully ADV. . PUNCT
It seems almost correct, falsely categorised “doorbell” as a proper noun, as opposed to a noun. This was the large model. The small model performed worse, categorised “rang” as a proper noun, and did not find the present tense of the verb. For some reason, this turned out to be a tricky sentence.
You can also visualise a sentence with the display method, although I think this is mainly useful in an educational environment. If you are working in a Jupyter Notebook, you already have a browser window open, and you can use the render method:
spacy.displacy.render(doc, style=”dep”)
You can also save these visualisations as SVG files, you can read more about how to do that in the documentation.
We saw how well spaCy works out of the box with a random sentence. What about really weird sentences with intentional misspellings like this one from Old Bugs (1919):
Some swell guy, too–’s father’s a big corp’ration lawyer up in his burg, ’n’ ‘s mother’s some lit’ry genius.
Well, the library struggles, as expected. It more or less gets the part of speech indexes of the tokens, but does not recognise that “ ’n’ ” is supposed to be “and”, “ ‘s “ is supposed to be “his”, etc...
One more thing I want to talk about in the spaCy library is how it recognises named entities in a text. To my surprise, it even works on fictional ones.
Take the opening sentence from Herbert West — Reanimator (1922).
Of Herbert West, who was my friend in college and in after life, I can speak only with extreme terror.
We can loop through the ents in the Doc object for the named entities:
doc = nlp(text)for ent in doc.ents:    print(ent.text, ent.label_)
Will return:
Herbert West PERSON
Which I found to be a quite impressive feature. Let’s see another sentence from the same story:
We were in the third year of our course at the Miskatonic University Medical School in Arkham.
After we list the entities in the same way, we get:
the third year DATEUniversity Medical School ORGArkham GPE
Almost perfect, although it missed the “Miskatonic” part for some reason.
This can also be visualised, this time we should set the style to “ent”:
spacy.displacy.render(doc, style=”ent”)
returns:
That’s all we need from spaCy for now.
As a first step, we are concentrating on one story. There are many built-in methods in libraries that do bag-of-words for you automatically, I did not find one that did exactly what I needed, and fortunately, it’s really simple to code it ourselves! Let’s create a number of helper functions first for this project.
First, we build a create_word_counts_by_pos function that takes in a raw text file, a list of part-of-speech tags, and optionally a word count dictionary in case we want to aggregate multiple stories this way. It will return one dictionary, where the keys are part-of-speech tags, and the values are smaller dictionaries with the words of the corresponding tag as values and their counts in the text as values... So the result dictionary of word_count_dict will have a word_count_dict[‘NOUN’] dictionary inside, which will consist of key-value pairs like ‘man’: 4, ‘house’: 28, ..., similar structure for word_count_dict[‘ADJ’], word_count_dict[‘VERB’], etc.
An important part of this process is that we want to exclude the stop words, which is really simple to do with spaCy, the tokens have an is_stop method that returns a boolean, we only need to consider tokens where this attribute is False.
Assuming we have a word_count_dict in this format, we might want to filter down to words above a certain threshold. This is what filter_word_count_dict_to_frequent will do, keep the structure of the input dictionary, but delete all elements where the count was below the threshold.
And finally, we will need some way to order the words so we can know which ones were the most frequent. This is what collect_most_frequent_words will be responsible for, you enter a word count dictionary with a number, and you get that many of the highest frequency words from each part-of-speech category.
This is all the code we are going to need to accomplish what we just discussed:
Now that we have these functions, getting the word counts in different structures is very easy. For now, we are only curious about the nouns, proper nouns, adjectives, and verbs:
list_of_pos = ['NOUN', 'PROPN', 'ADJ', 'VERB']
Assuming we have all the text in the text variable, to get the word_count_dict:
word_count_dict = \    word_count_func.create_word_counts_by_pos(text, list_of_pos)
word_count_dict will be a long collection of words, any noun, adjective, or verb that appears in the text even once will be represented there. We can filter it down to only words that appear more than 10 times:
frequent_word_count_dict = \    word_count_func.filter_word_count_dict_to_frequent(        word_count_dict, 10)
frequent_word_count_dict is concise enough that it does not seem excessive to simply copy it here:
{'NOUN': {'friend': 13,  'college': 21,  'life': 39,  'work': 18,  'year': 19,  'experiment': 24,  'fear': 13,  'death': 16,  'solution': 22,  'time': 20,  'case': 11,  'reanimation': 13,  'corpse': 14,  'specimen': 34,  'body': 49,  'eye': 17,  'field': 15,  'thing': 53,  'room': 12,  'laboratory': 18,  'house': 16,  'horror': 15,  'man': 28,  'result': 15,  'night': 17,  'sound': 14,  'police': 13}, 'PROPN': {'Herbert': 20,  'West': 132,  'Arkham': 18,  'Dr.': 15,  'Bolton': 12}, 'ADJ': {'great': 20,  'hideous': 17,  'dead': 23,  'human': 21,  'fresh': 28,  'small': 12,  'new': 14,  'large': 12}, 'VERB': {'see': 17,  'come': 24,  'hold': 11,  'know': 18,  'find': 14,  'restore': 11,  'inject': 15,  'think': 18,  'look': 14,  'tell': 11}}
If you read the story / saw the film, you will probably find a lot of these satisfying. For those of you who did not: the story is basically about a mad scientist who creates zombies, and is on a never-ending quest to find fresh corpses.
Let’s take the adjectives, the top three adjectives are: “fresh dead human” could be the working title of the story. “specimen” is used more often than “man”, apparently the narrator who is friends with Herbert West mostly refers to him as “West”, not as “Herbert”, and the verb “inject” is used suspiciously often.
Now let’s do this for all the stories!
I decided to omit the proper nouns from the analysis, and concentrate on the top 5 words per part-of-speech for each story. First, we need to set up some variables:
number_of_words_to_collect = 5list_of_pos = ['NOUN', 'ADJ', 'VERB']words = {}word_counts = {}for part_of_speech in list_of_pos:    words[part_of_speech] = {}    word_counts[part_of_speech] = {}    for number in range(number_of_words_to_collect):        words[part_of_speech][number] = []        word_counts[part_of_speech][number] = []
Then, we can do a similar process to what we did with the individual story in the previous chapter. We are not using the frequent dictionaries, instead, request a list of the most frequent words. Assuming filenames contains the names of files we want to import, and they are saved in the txt_folder:
for filename in filenames:     my_file = open(txt_adjusted_folder + filename + '.txt')    print("Currently processing: " + filename)    raw_text = my_file.read()    word_count_dict = \         word_count_func.create_word_counts_by_pos(            raw_text, list_of_pos)    most_frequent_list_dict = \        word_count_func.collect_most_frequent_words(            word_count_dict, number_of_words_to_collect)    for part_of_speech in list_of_pos:        for number in range(number_of_words_to_collect):            words[part_of_speech][number].append(                most_frequent_list_dict[part_of_speech][number][0])            word_counts[part_of_speech][number].append(                most_frequent_list_dict[part_of_speech][number][1])
After that’s done, we should have everything we need in the two dictionaries, words and word_counts. I combined them with pandas, for the ‘NOUN’ group, we can do it like so:
noun_counts = pd.DataFrame()noun_counts['filename'] = filenamesfor i in range(number_of_words_to_collect):    noun_counts['word_' + str(i+1)] = \        words['NOUN'][i]    noun_counts['word_' + str(i+1) + '_count'] = \         word_counts['NOUN'][i]
Repeat the same for adjectives and verbs, and now we have the top 5 words with their counts in the 3 part-of-speech groups for the 63 stories in the corpus. Copying them all here would be a bit much, you can find the data in CSV files here for nouns, adjectives, and verbs.
Let me just grab a couple of examples that I found to be really interesting.
Nouns:
One of my personal favourites, The Music of Erich Zann (1921) is represented by these nouns: street, music, night, window, room.
The Lurking Fear (1922): mansion, fear, night, thing, eye.
The Whisperer in Darkness (1930): thing, man, letter, time, voice
Adjectives:
Celephaïs (1920): strange, white, great, golden, old
The Call of Cthulhu (1925): great, old, strange, young, dead
The verbs are usually not that closely matching with the story, there are a lot of general verbs out there after all. Plus Lovecraft’s style is very descriptive and distant, and there is hardly any action.
All right, I’m pretty satisfied so far, I think it’s impressive how such a basic analysis managed to capture the essence of so many of the stories.
And finally, back to the big question, which words did he use the most often ever? Following a similar process as above, but in every turn, we keep the word_count_dict, so it calculates a combined score. (Combining all the text does not work, it’s roughly 2.5 times more than the memory limit in spaCy.)
So, drumroll, these are the 20 nouns, adjectives, and verbs that Lovecraft used the most often:
{'NOUN': [('thing', 1152),  ('man', 1075),  ('time', 830),  ('night', 666),  ('place', 569),  ('house', 490),  ('day', 476),  ('city', 453),  ('year', 432),  ('stone', 407),  ('dream', 392),  ('room', 383),  ('world', 379),  ('door', 370),  ('way', 367),  ('horror', 352),  ('light', 350),  ('life', 340),  ('wall', 335),  ('eye', 333)], 'ADJ': [('old', 917),  ('great', 797),  ('strange', 488),  ('certain', 446),  ('black', 379),  ('little', 336),  ('ancient', 308),  ('high', 297),  ('small', 282),  ('dark', 277),  ('new', 268),  ('human', 267),  ('unknown', 259),  ('terrible', 256),  ('long', 244),  ('curious', 227),  ('low', 216),  ('hideous', 213),  ('young', 210),  ('good', 204)], 'VERB': [('come', 1100),  ('know', 1095),  ('see', 1079),  ('find', 784),  ('tell', 649),  ('think', 571),  ('hear', 528),  ('look', 474),  ('go', 439),  ('say', 425),  ('leave', 401),  ('begin', 400),  ('feel', 397),  ('take', 368),  ('grow', 310),  ('give', 306),  ('shew', 275),  ('bring', 267),  ('speak', 266),  ('try', 262)]}
Once again, I think the verbs are the least exciting, although you can see how most of the verbs are about knowledge gathering or sharing. In case you are wondering, “shew” is the old-fashioned spelling of show.
Nouns are better, “time”, “night”, “stone”, “dream”, “horror” — I would guess the writer is Lovecraft based on just these five words. I mean, horror is there in the top 20 nouns ever used, that is certainly reassuring!
And finally, the adjectives, at least 2/3rd of them would generally be weird to have so high on the list. The most often used adjective is “old”, I mean, how awesome is that? And the list goes on to “great”, “strange”, “ancient”, “unknown”, “terrible”, etc.
(You might notice that we have “hideous” 213 times, but on the Arkham Archivist page that I mentioned in the intro, there are 260 counts of the word. The difference is due to the fact that we separate word usage by part-of-speech. He used the word 213 times as an adjective, but that does not include “hideousness” which is a noun, or “hideously” which is an adverb.)
I think there are many more interesting analysis you could do with these simple NLP tools. We limited our scope to nouns, adjectives, and verbs, it would be interesting to see adverbs or proper nouns too. Analysing the relative portion of let’s say verbs in different years would also be interesting. That way, we could see if the stories became more action-packed as time went by, or was it the other way around? There is also a very important concept we did not consider: the n-grams. A very similar analysis could be done to find the most frequently used expressions in his writings.
We started with a very simple question: I wonder what is Lovecraft's most frequently used word? And for such a simple question this turned out to be more complicated than I imagined. It was really cool to see such a simple NLP representation to work surprisingly well.
In the next post, I am going to write about how one can take word counts a step further and calculate the differences between documents using TF-IDF vectors.
Hobson, L. & Cole, H. & Hannes, H. (2019). Natural Language Processing in Action: Understanding, Analyzing, and Generating Text with Python. Manning Publications, 2019.
spacy.io
The Complete Works of H. P. Lovecraft: