What to Read Next? Analyzing the Digital Fragmenta Historicorum Graecorum with Python and NLP
Tags: Classics, Python, Digital Humanities, Ancient History, NLP

Getting the data

The DFHG’s API relies on queries of authors: this makes sense because of the fragmentary nature of the corpus — we don’t have specific “works” to search through, like The Republic, so author is the most atomic we can get. We thus need to extract a list of all the authors included in the corpus. We can then build out API calls and feed our data into an NLP engine.

This code outlines the steps I took to send a request to the DFHG’s API documentation, clean up the HTML, and get a (clean) list of authors.

Here is a quick check of the word counts from the fragments in our corpus (ignore the warning):

Latium est, non legitur

This is not a clean dataset — at least, not in the sense that we’re ready to do any actual analytics yet. The problem here is the nature of the material: these are fragmentary pieces of writing, some of which survive only through quotations in other sources. What makes life even more difficult is disentangling the “actual” authorial content from whatever may have accrued to the text at the hands of scribes across the centuries. As a more particular matter, what we have here are actually several fragments that are in Latin.

The DFHG does provide a Latin translation for most (if not all) of the fragments in the corpus, but when Latin comments turn up in the ‘text’ field of the data we scraped earlier, that suggests the influence of a commentator, not necessarily the author we want to look at. To proceed, we need to figure out a way to separate the content we want from the commentary we don’t. Luckily for us, Latin and Greek are different languages written in different alphabets. (It would be a different question entirely if we were trying to segregate, say, Spanish and English words when they’re bunched up together, since they use the same alphabet.)

Since Latin uses, naturally, the Roman alphabet, we can use regex (regular expressions — basically word searches) to find patterns of text in the data that are made up of Latin characters. I’m going to be pretty broad here: we want to capture as much of the Latin as possible while leaving Greek. If a passage is all “original,” we should let it stand. Otherwise, we should flag it. This line will pick up any row in our data that has a Latin character in the “text” field:

latin_index = author_df[‘text’].str.contains(‘([a-zA-Z])\)’, regex = True)

Let’s do some further cleaning to get the number of Latin words, their positions within the text, and the relative frequency of Latin/non-Latin words in each fragment:

We now have counts and frequencies. At this point we have to make some decisions about how to handle these data points: we can just drop these rows and move on, or we can try and keep the texts that have a minimal amount of Latin in them.

Try the second route. There are options here as well: we can either drop rows entirely, or try and filter out the Latin words from each text. I’m going to try a combination of both, going through the data first and trying to drop rows that are entirely or almost entirely Latin, then scrubbing the remaining text.

First, let’s look at the distribution of Latin in the corpus we have.

What these charts tell me is that our intuition was right — we have a few authors whose work preserved in the DFHG is almost entirely in Latin (i.e. those fragments that fall on the left hand side of the lower figure), and many others where a few stray Latin words have made their way into the mainly Greek text.

To make a final pass at cleaning the corpus, I defined a few functions to take a list of words and return the indices of all Latin words in the list. I then filter the list of words by deleting elements at those indices.

for index, row in final_df.iterrows():

indicies = row['latin_word_index']

for index in sorted(indicies, reverse = True):

del row['words'][index]

I then drop fragments with remaining Latin words.

final_df = df.drop(df[df['latin_word_count'] != 0])

NLP Time

Now we can move on to the NLP part of this project. I’m taking my cues from some of the example notebooks posted on the CLTK Github page. I’m going to calculate a few summary metrics, including lexical density, which measures the number of “lexical” words as a percentage of total words in a passage. A “lexical” word is a word that is not extremely common. These are called “stop” words (think something like “and,” or “or” in English) and we don’t want to include them in our analysis. Counting stop words doesn’t tell us much about a text, beyond an author’s penchant for conjunctions.

So, when we look at a text like the quote from Jefferson, above, we expect the lexical density to be somewhere in the middle, since there are a decent number of stop words, but few words are repeated. According to Analyze My Writing, the lexical density quote 48.24%, which is about what we’d expect.

We can take a first look at lexical density by fragment after some cleanup. We’ll then look at the lexical density by author. First, we need to regularize the texts and strip out punctuation and so-called stop words. (A special note on punctuation — Greek uses the semicolon ; to indicate a question, so we won’t include that character in the punctuation list to filer.) A stop word in this case is a common word that doesn’t add much to our understanding of the meaning of the text. In English these are words like “the,” “is,” “and,” and so on. The CLTK comes with a built-in list of stop words for both Greek and Latin, so I’ll use that functionality to clean up the fragments. This code will clean up the list of words we’ve extracted from each fragment:

def filter_punct(words):

out = [word for word in words if word not in punct]

out = [word for word in out if word not in stops] final_df['words'].apply(clean_punct)

final_df['lemmata'].apply(clean_punct) ## Note that this function actually cleans both punctuation and stop

## words final_df[['author', 'clean_lemmata']] # prints the following

Now a quick check of the length of each fragment we have:

And finally we can now calculate the fragment level lexical density for our corpus:

def lemmatize(text):

lemmata = lemmatizer.lemmatize(text) # from cltk

return lemmata df['lemmata'] = df['clean_words'].apply(lemmatize) df['lexical_density'] = df.apply(lambda x: len(set(x['lemmata'])) / len(x['lemmata']), 1) # python set return set of unique elements

Let’s remove the fragments whose lexical density is 1 and plot to see what kind of distribution we have. (I’m omitting fragments where density is 1 for two reasons. First, these fragments tend to be much shorter — the average word count for a 100% density fragment is ~10 words, while those with lower densities average ~82 words. Since these are fragments, it’s likely that passages with 100% density are not representative of an author’s voice. Second, shorter fragments will almost by definition have higher lexical densities than longer fragments. Because shorter fragments are made up of fewer words, there are fewer opportunities for an author to repeat herself, and fewer repetitions leads to a “denser” text.)

Looks like the distribution is approximately normal, but with lots of left skew. We can check the descriptive statistics for more: