Codex to corpus: Exploring annotation and processing for an open and extensible machine-readable edition of the Florentine Codex
Year: 2023

Abstract:
This paper describes an ongoing effort to create, from the original hand-written text, a machine-readable, linguistically-annotated, and easily-searchable corpus of the Nahuatl portion of the Florentine Codex, a 16th century Mesoamerican manuscript written in Nahuatl and Spanish. The Codex consists of 12 books and over 300,000 tokens. We describe the process of annotating 3 of these books, the steps of text preprocessing undertaken, our approach to efficient manual processing and annotation, and some of the challenges faced along the way. We also report on a set of experiments evaluating our ability to automate the text processing tasks to aid in the remaining annotation effort, and find the results promising despite the relatively low volume of training data. Finally, we briefly present a real use case from the humanities that would benefit from the searchable, linguistically annotated corpus we describe.

heading: 1 Introduction
text: The Nahuatl language, an agglutinating and polysynthetic member of the Uto-Aztecan family spoken throughout Mexico by about 1.5 million people today, has a rich literary tradition (Gingerich, 1975; León-Portilla, 1985). With a strong preconquest oral tradition and a hieroglyphic writing system, Nahuatl speakers quickly adopted the Latin alphabet for writing their language after its introduction almost immediately after the Spanish invasion. As a result, the volume of the colonialera Nahuatl literary canon is unrivalled in Latin America (Olko and Sullivan, 2013). These texts are invaluable resources to scholars interested in the history, culture, and language of colonial and pre-invasion Nahua communities.
Perhaps the most notable Nahuatl text of the early colonial period, the Historia General de las Cosas de Nueva España “General History of the Things of New Spain” (Florentine Codex, FC) is an encyclopaedic work in Nahuatl and Spanish
compiled by Indigenous scholars from the Colegio de Santa Cruz de Tlatelolco and Franciscan friar Bernardino de Sahagún.
The FC is undoubtedly one of the most valuable manuscripts of the early modern period. However, it was forgotten for centuries until Angelo Maria Bandini described it in 1793. He named it “Codice Fiorentino” after the Biblioteca Medicea Laurenziana in Florence, where it is still kept. But only at the beginning of the 20th century did Francisco del Paso y Troncoso bring it to a wider audience (Martínez, 1982). Charles Dibble and Arthur Anderson published a translation of the books into English throughout the second half of the 20th century. The original manuscript became available in the World Digital Library only ten years ago, thanks to the Library of Congress.
The impetus for the present project was the need of the third author, a humanities scholar, to search the text of the FC for specific linguistic constructions and terminology. This proposition is complicated by a number of factors:
First, there are few fully digitised versions of the FC, and those that do exist are under copyright, constraining the ability of a scholar to reproduce, annotate, and/or re-release any part of the text that results from a given research endeavour.
Second, the FC, having multiple authors and being written in the early years of Nahuatl alphabetic writing, contains numerous orthographic inconsistencies throughout the 12 books, with many words written in multiple distinct ways and decisions about word tokenisation not being standardised. Furthermore, due to constraints on column width in the original manuscript, words are frequently split by line breaks with no indication of whether the following line continues the word from the end of the previous one. Keyword searching this text is a seemingly-futile process involving determining all possible spellings for a given word and all possible tokenisations of a single syntactic
19
word into multiple orthographic words. Finally, Nahuatl is a morphologically complex language with large amounts of inflection and derivation, making querying the surface/inflected form, instead of e.g., a lemma, particularly difficult.
The present project attempts to address these issues by creating an open-source, retokenised, and normalised corpus of the FC with queryable linguistic annotations following the Universal Dependencies framework (Nivre et al., 2020a). In the following sections, we describe the corpus, each component involved in its creation, and an investigation into automating the processing. We conclude by outlining a road map for the project’s completion and a vision of future applications.

heading: 2 Related work
text: The FC has been the subject of a great deal of research in the humanities by scholars interested in the cultural beliefs and practices of the Nahua people during the early colonial period (Sullivan et al., 1966; Gingerich, 1988; Sigal, 2007; McDonough, 2020; Olivier, 2021). It has also served
as a foundational component for work studying so-called “Classical Nahuatl,” or Nahuatl spoken during the period (Launey, 1986; Lockhart, 1992, 2001). Both Olko et al. (2015) and Olko (2018) leverage corpus-based approaches using a multitude of historical Nahuatl documents, but it is unclear how much linguistic information was available in the corpus, and to our knowledge, this corpus has not been released to the public.
Gutierrez-Vasques et al. (2016) released Axolotl, a large, Spanish-Nahuatl parallel corpus with a focus on machine translation. It includes Nahuatl from multiple variants and time periods, including the early colonial period, but does not include text from the FC. Furthermore, the text in Axolotl is unprocessed and unannotated.
Other corpora that include Nahuatl texts include the Johns Hopkins University Bible Corpus (McCarthy et al., 2020), a parallel multilingual corpus that includes numerous contemporary Nahuatl variants. This corpus has been used to produce morphosyntactically-annotated resources for a large number of languages (Nicolai and Yarowsky, 2019; Nicolai et al., 2020).
The first open morphosyntactically-annotated corpus of Nahuatl was recently released by Pugh et al. (2022) and includes 10,000 tokens of the Western Sierra Puebla variety. Following this work, we also select UD as our annotation schema.
Marc Eisinger was the first to publish a computerised version of the FC, which is not freely available (Eisinger, 1977). The Universidad Autónoma de México (UNAM) hosts a website, Temoa, containing a large volume of digitised colonial-era Nahuatl texts, with minimal processing (at the very least, tokenisation problems in the FC appear to be corrected (Universidad Nacional Autónoma de México, 2023). However, the copyright and rights to use for annotation and re-release are retained by UNAM,1 making it not possible to create derivative works, such as the annotated corpus described in this paper. Furthermore, the original text (before fixing tokenisation) is not available.
Related to the computational processing of colonial Mexican texts, The “Digging into colonial Mexico” project (Murrieta-Flores et al., 2022) involves the creation of a number of processed and machine-readable resources based on colonial Mexican documents, mostlywritten in colonial-era Mexican Spanish. As for colonial texts written in Mexican languages, the Ticha project (Broadwell et al., 2020), a collaboration between members of Zapotec-speaking communities and academics from universities in the United States of America, offers an “online digital text explorer” for colonial Zapotec texts and includes morphological analyses and translations.

heading: 3 Corpus
text: Our corpus comes from a typed transcription upholding the original layout, published in the openaccess repository Zenodo2 to allow the semantic and computational study of the text from the primary source (de Sahagún, 2022). In Figure 1 we present a folio from the manuscript where the text in Spanish (left) and Nahuatl (right) is seen in two columns, and an example of the transcription output in our corpus.

heading: 3.1 Orthography
text: There is a great deal of orthographic variation in the FC, in both the Nahuatl and Spanish sections, with multiple characters used inconsistently
1https://temoa.iib.unam.mx/creditos 2https://zenodo.org/
throughout. For example, the letter [v] can represent either /w/, e.g. veue /wewe/ ‘big’ (norm. huehue), or a long /o:/, e.g. vmpa /o:mpa/ ‘there’ (norm. ompa). [j] is used both for the vowel /i/ e.g., jnpilhoan /inpilwa:n/ ‘their (pl) children’ (norm. inpilhuan) and the glide /j/, e.g. jollochicaoac /jol:otSika:wak/ ‘brave’ (norm. yollochicahuac). The letter [i] is also observed in both of these contexts.
There are also instances where a single sound, e.g. /S/ can be represented by multiple letters, in this case [x] or [s]. For example, the word axcan /a:Ska:n/ ‘now, today’ can appear as ascan or axcan. But [s] can also be the voiceless alveolar sibilant /s/ in loan words from Spanish visorrej /bisorei/ ‘viceroy’ (norm. visorrey).

heading: 4 Processing
text: A major theme of the processing of the FC is the use of initial detailed hand-annotation in order to bootstrap automated approaches for the remaining text. Crucially, the resulting corpus should be usable for academic research and, as such, must maintain the utmost quality. In this context, then, we consider automation a strategy to assist in human annotation, but still require manual auditing of the entirety of the annotated corpus.

heading: 4.1 Sentence segmentation
text: Full stops (or in dialogue, exclamation marks, and question marks) are used as sentence boundaries throughout the corpus, with the colon symbol often used to separate clauses, making sentence segmentation fairly straightforward. There are a number of abbreviations, such as xpo. for Christ and p. for Pedro. Table 5 presents the size of each book in terms of sentences, space-separated tokens, and words. Words are only given for the three books we have processed so far.

heading: 4.2 Retokenisation
text: There are a number of tokenisation inconsistencies in the original manuscript, resulting from (1) physical constraints, namely the author running out of room on one line and splitting a word across a line boundary (see Figure 1), (2) inconsistent tokenisation practices by the authors, such as sometimes writing the article subordinator in and an adjacent verb together as a single orthographic word, and (3) possible mistakes introduced during the process of manually typing up the manuscript.
Our first step in processing the codex, after obtaining text files transcribed from the original manuscript, involves “retokenisation”: altering the word boundaries in the text to align them with canonical Nahuatl words.3 An example of the input and output of this process is shown in Table 1, wherein a space is represented by the mid-dot character, ·, and newline is represented by the pilcrow character, ¶.
As with the rest of the processing steps, retokenisation starts as a manual process. For each identified case where retokenisation is necessary, we use the left and right contexts to write a rule for handling that case, ensuring that the contexts are large enough to avoid potential ambiguities (for instance, a minimal-context rule such as “n·c →nc” will likely produce many false positive matches). In the event that a rule produces false positives, we expand its contexts (e.g., “qujn·caoa →qujncaoa”). We use a left-to-right longest-match (LRLM) algorithm to apply the approximately 4,000 retokenisation rules.

heading: 4.3 Normalisation
text: Once the text is correctly tokenised, the next processing step is orthographic normalisation. We use the ACK (Andrews, Campbell, Karttunen) orthographic standard for the target orthography, since it is designed to reflect colonial-era Nahuatl writing (Campbell and Karttunen, 1989; Andrews, 2003; Karttunen, 1992).
For Spanish words we use contemporary orthography, so for example, gouernadores is normalised to gobernadores ‘governors.’
For proper nouns, we also use modern orthographic conventions where available. For example, tlatilulco is normalised to Tlatelolco, and motecu-
3Following authoritative resources like Andrews (2003) and Campbell and Karttunen (1989) in identifying “canonical words”, which should include subject, object, and aspectual affixes.
coma is normalised to Moctezuma. The process uses a hand-curated dictionary mapping original word forms to their normalised counterparts (e.g. the normalised form yaoyotl ‘war’ is written variably as iaoiotl, iauiotl, iaviotl, iaujutl and iaujotl. Thus, our dictionary has an entry for each of these forms mapping to the normalised form). To build the dictionary, we start with a naïve finite-state transducer (FST) model designed using general patterns of colonial-era Nahuatl writing. We then post-edit the output of the FST, adding all correct word pairs to the dictionary. We update the FST weights as we add forms to the dictionary to improve its performance. After processing three books, the dictionary contains 6,515 entries.
The main motivation for performing the normalisation manually is to ensure a high-quality data set with which to train a model for automating the process. We discuss the evaluation of such an approach in §6.2.

heading: 4.4 Part-of-speech tagging
text: The part-of-speech tags are based on the Universal Part-of-Speech categories (UPOS) defined and used in the Universal Dependencies framework (Nivre et al., 2020b).
We accomplish part-of-speech tagging in three steps. We use a lexicon, a morphological analyser (see §4.5) and a set of ordered, regular-expressionbased guessing rules applied to the normalised form, in sequence. We refer to this last component as ‘the guesser.’
The lexicon is simply a list of normalised surface forms and their part of speech. Of the 10,959 types presently annotated for part-of-speech, 1,478 (6,916 tokens) received their POS from the lexicon.
In the event that a given surface form is not observed in the lexicon, we next run the word through the morphological analyser. This method accounts for 13,762 of the tokens thus far annotated (1,705 types).
Finally, any word not identified in the previous two steps is passed to the guesser. The guesser consists of 36 rules which use regular expressions to look for particular prefixes and suffixes and assign part-of-speech tags with high precision. For example, words beginning with nimitz-, a combination of the first person subject marker and second person object marker are categorised as verbs, and words ending in -tzitzin, which is the plural reverential marker, are categorised as nouns. These rules are high precision, but low recall: a total of 986 forms out of 10,959 forms (1,471 tokens) in the three processed books receive guessed analyses.
We randomly sampled and manually checked 200 of these guesses and found that 198 were correct. In one case the mistake was due to a mistaken normalisation (iehoatin → *yehuatin instead of yehhuantin ‘they, them’), which resulted in the word being tagged as a noun due to the -tin ‘PL’ ending (plural). The second casewas to dowith the same plural rule, which resulted in the word xixitin ‘it crumbled’ (from the verb xixintini ‘to crumble’) being tagged as a noun.

heading: 4.5 Morphological analysis
text: Morphological analysis is the task of producing, for a given surface form, a lemma and a set of morphosyntactic tags describing that form. For example, given the form tictlamacazque /ti-c-tlamaca-z-que/ ‘We will give something to him’ (or ‘We will make offerings to him’) it would produce,
<s_pl1><i_sg3><o_nn3>maca<v><dv><fut>
Where <s_pl1> stands for 1st person plural subject, <i_sg3> stands for 3rd person singular secondary object, <o_nn3> stands for 3rd person inanimate indefinite object, <v> stands for verb, <dv> stands for ditransitive and <fut> stands for future. Note that there is a long distance dependency between the prefix ti-, which can be 2nd person singular or 1st person plural and the suffix -que which marks a plural subject.
A given token can produce more than one analysis, so for example, quinchihua ‘They made them’ or ‘He made them’ produces,
<s_pl3><o_pl3>chihua<v><tv><pres>
<s_sg3><o_pl3>chihua<v><tv><pres>
In this case, because of underspecification in the orthography, the plural subject-marking suffix -h
is not written, resulting in an ambiguous analysis. The omission of this suffix is quite common in Nahuatl texts.
For implementing the morphological analyser we used the Helsinki Finite-State Toolkit (HFST) (Lindén et al., 2009). The analyser was implemented over the normalised forms. Morphotactics and the lexicon were implemented using lexc, while any morphographemic constraints were implemented with twol. A given surface form, for example, omoyollochichili ‘He strove strongly’ (lit. “he waited for himself on behalf of the heart”), consists of three parts, the surface form (1), the morphotactic form (2) and the lexical form/analysis (3).
1. omoyollochichili 2. o>mo>«yollo»chichi>lia 3. <aug><s_sg3><o_ref>«yollotl<n>»
chichilia<v><tv><past>
Themorphotactic form is the combination of the morphs beforemorphographemic rules are applied, it includes symbols to mark segment boundaries, such as ‘>’ for an inflectional boundary, ‘«...»’ for incorporated elements (in this case, the second object), ~ for reduplication and ‘·’ for clitic boundaries. The symbols around the incorporated element allow that part of the surface form to be extracted for use in the representation of incorporation (see §5.1).

heading: 5 Representations
text: In this section we discuss a number of features of Nahuatl that require special attention in the Universal Dependencies framework.

heading: 5.1 Incorporation
text: Incorporation is the process by which a verb can incorporate, that is, be syntactically incorporated with one or more of its arguments or adjuncts. Incorporation has been understudied in the field of natural language processing, and there are few articles that describe annotation projects for languages exhibiting this feature.
In this project, we follow the proposal laid out by Tyers and Mishchenkova (2020) in which incorporated items are exposed in the enhanced dependency graph annotated with the relation of the slot that they fulfill in the argument structure.
Table 2 demonstrates this with the verb moyollochichili, where the verb chichilia ‘enbitter’ takes the incorporated object yollo- ‘heart.’

heading: 5.2 Relational nouns
text: Relational nouns are nouns which express spatial and temporal relations when used with other noun phrases. These may be used as independent words in a possessive structure (1) or compounded to other words (2).
1. inepantla in ilhuicatl ‘in the midst of the heavens’ (lit. its-midst the heaven)
2. ilhuicayollotitech ‘in the heart of the heavens’ (lit. heavens-heart-on)
The first case is straightforward, each noun is analysed as a separate word, with the relational noun receiving a lexical feature NounType=Relat in addition to the necessary possessive morphology.
In the second, we take advantage of the multitoken word encoding in the CoNLL-U format and analyse the compound as consisting of two parts, the head and the compounded relative noun.

heading: 5.3 Lemmas
text: We also include the lemmas, or the stems, for each word. Lemmas ignore any of the inflectional morphology on the surface form of the word. Lemmatisation is performed first by looking up a surface
form in the lexicon and, if the word is not in the lexicon, by the morphological analyser.

heading: 6 Automated processing
text: We experiment with the existing processed FC data to see to what extent we might be able to automate the retokenisation and normalisation steps. Following previous work showing that historical text normalisation can be modelled effectively as a character-based machine translation problem (Bollmann, 2019), we train an encoder-decoder Seq2Seq model with Attention on character sequences for both tasks. While a natural inclination would be to train both retokenisation and spelling normalisation jointly, we are interested in storing each intermediate step for potential future research, and so train a separate model for each task.
For the orthography normalisation model, we treat each word as a training instance, and map the unnormalised word (e.g. qujchioa) to its corresponding normalised form (e.g. quichihua).
For the retokenisation model, training on each word would not work since the phenomenon we are modelling spans word boundaries. Instead, we split the text on unambiguous punctuation (‘.,:;?!’), creating numerous subsequences from each sentence.
Since the objective is to evaluate how well we could automate the text processing for future books, we used two of the three already-complete books (Books 1 and 8) for training, and held out
Book 5 for evaluation. The models used a bidirectional LSTM encoder, and training was done using OpenNMT (Klein et al., 2020). We trained both for 100 epochs.
Results of the experiments are listed in Table 3. They are generally favourable, though perhaps not quite to the point of being able to completely automate the low-level processing of the remaining books.

heading: 6.1 Retokenisation
text: A number of the mistakes we see from the retokenisation model involve a type of ‘hallucinations,’ where the output contains characters not in the input. This is an effect of treating this problem as one of translation with a relatively low volume of training data. To remedy this problem, we may try adding an additional auto-encoding or “copying” auxiliary task as discussed in Mager et al. (2019), wherein we add training examples that are already correctly tokenised in order to provide more examples of correct outputs.
Alternatively, the task of retokenisation can be straightforwardly modelled as a one-to-one sequence tagging problem, where for each input character the model must assign one of three “retokenisation actions”: (1) merge, or remove a token boundary that follows the current character, (2) split, or add a token boundary after the current character, or (3) do nothing. For comparison, we also evaluate this approach, using a bidirectional LSTM also trained for 100 epochs.4 This approach has a slightly worse word error rate compared to the MTbased approach, but has a lower character error rate. The advantage to this approach is that we don’t risk transforming characters or inserting substrings during the tokenisation step.

heading: 6.2 Orthographic normalisation
text: The orthographic normalisation model correctly normalises 87% of the words in the held out book. The errors suggest a similar issue seen in the retokenisation model, namely the insertion of multiple additional characters not corresponding to the input (e.g. converting input ie, to *yeyecye instead of ye). This issue, as mentioned above, would likely be alleviated with some data aug-
4Given our limited data volume and the interest to simulate testing on an unseen book, the results we report here do not include a hyper-parameter tuning step using a heldout development set. With an additional held out book we could tune these models’ hyperparameters and improve performance.
mentation and/or multi-task training to ensure the model sees enough examples of properly formed output strings. We plan to leverage this model as a backup in the case where we are not able to identify a normalisation via our dictionary-lookup approach. For example, by first checking if we have seen a given word in the training data and, if so, using the corresponding output from training and using the model’s prediction on unseen words only, the word error rate drops to 8.3.

heading: 7 Use cases
text: In this section, we provide descriptions of a few research questions that could be informed by our corpus. The use cases are based on information that is available in the corpus and is not found in other editions of the manuscript.
The first use case concerns the status of the tlamatinimeh ‘sages, wise men’ (lit. those who know things). It is widely claimed that there is no philosophy outsideWestern philosophy (Maffie, 2014), but this claim has been contested by scholars, starting from Ángel María Garibay and his student Miguel León-Portilla who identify the tlamatinimeh with philosophers and argue that the precontactMexicans had long philosophical traditions (León-Portilla, 1956). Analysing individual words has since this work been the basis of understanding Nahua thought. However, to date this process is difficult and error-prone as it involves carefully reading through unannotated concordances of surface forms and it is easy to miss examples that appear in forms that are unknown or unfamiliar to the researcher.
Our corpus will be able to help by allowing scholars to extract examples that are morphologically and syntactically related. For example, by allowing queries based on lemmas (encompassing for example tlamatini ‘sage’, tlamatinimeh ‘sages’, etc.) It will also allow for searching for specific syntactic constructions, such as those where a tlamatini is the subject of a speech verb.
The second use case concerns concepts of time. For instance, consider Maffie (2014)’s statement about the Mexica conceiving time and space as a single unit. He argues that the Mexica did not separate time and space but had a perception of a “time-place” . This argument is based on the word cahuitl, which means ‘time’, and the intransitive verb cahui, which means “to stay or end”. However, Maffie argues that cahuitl is also related to
the transitive verb cahua, which means “to leave or abandon”, among other senses.
The morphological annotations (including lemmas) in our corpus will allow for searching on lemma (to be able to distinguish forms of cahui from forms of cahua). And the syntactic annotations will allow for the extraction of time and place obliques that are dependents of those two verbs.

heading: 8 Concluding remarks
text: We have outlined the strategies and approaches involved in creating a free and open, linguisticallyannotated corpus of the FC. Having nearly completed retokenisation, orthographic normalisation, lemmatisation, part-of-speech tagging, and morphological analysis for 3 of the 12 books, we have established the key linguistic information to include in the corpus, and have engineered the foundations of the annotation process. Results of our preliminary experiments into automatic annotation suggest that some tasks, like orthographic normalisation, can largely be automated with the existing data, whereas others, e.g., retokenisation, likely still require more labelled data and/or a more powerful architecture.

heading: 8.1 Future work
text: Our first priority for the future is to continue the annotation process, automating some of the text normalisation, expanding the lexica, and enhancing
the morphological analyser. We are optimistic that with each subsequent book, the additional amount of available annotated data will enable faster future annotation via automation. Finally, adding dependency syntax annotations will enable quantitative analysis of colonial Nahuatl syntax, a field with relatively little prior work.
The study described in §7 is one of many potential uses of an annotated corpus as described here. We expect that the release of this corpus with complete morphosyntactic annotations and an unambiguous free licence will promote future research from scholars in a variety of fields.
Additionally, the tools for automatic processing of the FC will likely be applicable to the numerous additional texts written in Nahuatl during the colonial period, contributing to the advancement of language technology development for Nahuatl.
Finally, another important project related to the development of this corpus involves the translation of the FC into contemporary Nahuatl variants, making the rich cultural heritage of theNahuatl language more accessible to Nahuatl-speaking communities. It is our hope that the production of this corpus can aid in the translation process.

heading: Acknowledgements
text: We would like to thank Maira Cayetano Nemecio and Stephanie Berthoud Frías for their valuable contributions. We are grateful to Mitsuya Sasaki and Joe Campbell for fielding numerous questions about language use in the Florentine Codex, and to the anonymous reviewers for their helpful feedback. Finally, a special thanks to Daniel Swanson, AndrewDavis, Zack Leech, andMaria Lucero Guillen Puon, for stimulating discussions about Nahuatl and the Florentine Codex.

heading: A Books of the Florentine Codex
text: Table 5 presents some statistics about the books of the Florentine Codex.

heading: Future Work
text: Our first priority for the future is to continue the annotation process, automating some of the text normalisation, expanding the lexica, and enhancing
the morphological analyser. We are optimistic that with each subsequent book, the additional amount of available annotated data will enable faster future annotation via automation. Finally, adding dependency syntax annotations will enable quantitative analysis of colonial Nahuatl syntax, a field with relatively little prior work. The study described in §7 is one of many potential uses of an annotated corpus as described here. We expect that the release of this corpus with complete morphosyntactic annotations and an unambiguous free licence will promote future research from scholars in a variety of fields. Additionally, the tools for automatic processing of the FC will likely be applicable to the numerous additional texts written in Nahuatl during the colonial period, contributing to the advancement of language technology development for Nahuatl. Finally, another important project related to the development of this corpus involves the translation of the FC into contemporary Nahuatl variants, making the rich cultural heritage of theNahuatl language more accessible to Nahuatl-speaking communities. It is our hope that the production of this corpus can aid in the translation process