Venue: NAACL 2016

# An Unsupervised Model of Orthographic Variation

for Historical Document Transcription

 Dan Garrette

Computer Science & Engineering

University of Washington

dhg@cs.washington.edu

Hannah Alpert-Abrams

Comparative Literature Program

University of Texas at Austin

halperta@gmail.com

###### Abstract

Historical documents frequently exhibit extensive orthographic variation, including archaic spellings and obsolete shorthand. OCR tools typically seek to produce so-called _diplomatic_ transcriptions that preserve these variants, but many end tasks require transcriptions with _normalized_ orthography. In this paper, we present a novel joint transcription model that learns, unsupervised, a probabilistic mapping between modern orthography and that used in the document. Our system thus produces dual diplomatic and normalized transcritions simultaneously, and achieves a 35% relative error reduction over a state-of-the-art OCR model on diplomatic transcription, and a 46% reduction on normalized transcription.

## 1 Introduction

Optical Character Recognition (OCR) for historical texts, a challenging problem due to unknown fonts and deteriorating documents, is made even more difficult by the fact that orthographic conventions including spelling, accent usage, and shorthands have not been consistent across the history of printing. For this reason, modern language models (LMs) yield poor performance when trying to recognize characters on the pages of these documents. Furthermore, transcription of the actual printed characters may not always be the most desirable output.

Greg (1950) describes two types of transcription: one that preserves variants and typographical errors, and another that records the _substantive_ content, with this noise removed. Though in 1950 the substantive version was the norm, today these have become two distinct but equally valid tasks. _Diplomatic_ transcription, the standard in contemporary OCR, preserves the variants of the document valuable to book historians and linguists. _Normalized_ or modernized transcription recovers the substantive content, producing a text that adheres to modern standards. Normalized transcriptions are easier for users to read, and make large collections of historical texts indexable and searchable Driscoll (2006).

The current ideal for digital editions of historical texts has been described as a combination of diplomatic and normalized transcription Pierazzo (2014). This is generally achieved with a pipeline: first OCR is used to transcribe the document, then an (often manual) post-hoc normalization is performed. However, such a pipeline will result in cascading errors from OCR mistakes, and fails to make use of knowledge about modern language during the initial transcription. Additionally, post-processing tools are typically cumbersome language-specific, handbuilt systems Baron and Rayson (2008); Burns (2013); Hendrickx and Marquilhas (2011).

In this work, we introduce a novel OCR model designed to jointly produce both diplomatic and normalized transcriptions. The model is an extension of Berg-Kirkpatrick et al.'s (2013)_Ocular_, the state of the art in historical OCR. Ocular's innovative ability to handle the material challenges of OCR (unknown fonts, uneven sinking, etc.) depends on its use of a character \(n\)-gram LM. Our model improves the quality of Ocular's transcriptions by automatically learning a probabilistic mapping between the LM, which is trained on modern text, and the unique orthography of the document. This results in both an

improved orthographically-correct diplomatic transcription and a modern-style normalized transcription. To our knowledge, this represents the first OCR system that jointly produces both diplomatic and normalized transcriptions.

We evaluate our model on a multilingual collection of books exemplifying a high degree of orthographic variation. For diplomatic transcription, our unsupervised joint model achieves an error reduction of 35% over the baseline Ocular system without support for orthographic variation, and nearly matches the error rate of an approach proposed by earlier work that uses a hand-constructed ruleset of orthographic rewrites. However, for the new task of normalized transcription, we achieve a 46% error reduction over the baseline, as well as a 28% reduction over the hand-built ruleset approach.

## 2 Data

The _Primeros Libros_ corpus dataset introduced in our previous work consists of multilingual (Spanish/Latin/Nahuatl) books printed in Mexico in the 1500s Garrette et al. (2015). The original dataset includes gold diplomatic transcriptions of pages from five books of different time periods, regions, languages, and fonts. We additionally include two new monolingual Spanish _PrimerosLibros_ books annotated with both diplomatic and normalized transcriptions. Spanish-only texts were needed in order to find language-competent annotators skilled enough to create the more challenging normalized transcriptions. We used each of the seven books in isolation since they each have a different font. For each book, we used 20 pages for training and 10 for testing. For two of the books, an additional 10 pages were held out for tuning hyperparameters with grid search.

To produce the Spanish and Latin LMs, we used texts from Project Gutenberg; these documents were written during the target historical period, but all follow modernization standards including substitution for obsolete characters and expansion of shorthand. These texts were chosen because they are a realistic sample set that is freely and publicly available. In the Nahuatl case, scarce online resources made it necessary to supplement Project Gutenberg text with that from a private collection.

## 3 Model

We extend Ocular, the generative model of Bergkirkpatrick et al. (2013), and its EM training procedure, to support our unsupervised approach to jointly modeling both diplomatic and normalized transcription. Ocular works by modeling the operation of a hand press in order to learn unknown fonts in the presence of the visual noise of the printing process: uneven linking and spacing in particular. Briefly, Ocular's generative story is as follows. First, a sequence of language states \(\ell_{i}\) is generated according to \(P^{\text{Lang}}(\ell_{i}\mid\ell_{i-1})\), where \(\ell_{i-1}\) and \(\ell_{i}\) may only differ on the start of a word. For each state, a character \(c_{i}\) is generated according to its language-specific \(n\)-gram LM: \(P^{\text{Charr}}_{\ell_{i}}(c_{i}\mid c_{i-n+1}\leavevmode\nobreak\ ... \leavevmode\nobreak\ e_{i-1})\). Next, a state's _typesetting_\(t_{i}\), consisting of a character's bounding box, vertical offset, and between-character padding, is generated according to \(P^{\text{TYPE}}(t_{i}\mid c_{i})\). Finally, the character image is rendered as a pixel-matrix inside the bounding box: \(P^{\text{RED}}(x_{i}\mid c_{i},t_{i})\).1

Footnote 1: See previous work for fuller detail including how a typesetting is composed of its parts, and how pixels are generated.

A major downside to the Oc

Figure 1: Our generative OCR model with the new _glyph_ layer (bolded). The Spanish (_sp._) \(n\)-gram language model (LM) generates a sequence of characters according to standard Spanish spellings, _uran_ in this case, from the word _procurando_ which may be written _procvrato_. Language-specific characterization probabilities are used to generate a _glyph char_ from each LM char, producing _vra_ and a zero-width (elided) \(n\). Finally, the model generates a bounding box and right-side padding (the typesetting) and a pixel-rendering of the glyph character.

 

dered image \(x_{i}\) is always generated directly from LM character \(c_{i}\), resulting in transcription errors when printed characters don't follow the spellings in the LM. Our model (Figure 1) adds an additional layer to the generative model that de-couplets the LM from the rendering by allowing the LM-generated character \(c_{i}\) to be replaced by a possibly different _glyph character_\(g_{i}\) which is rendered instead.

For the generative story of our new model, we again begin by generating pairs \((\ell_{i},c_{i})\). However, instead of typesetting \(c_{i}\), we generate a distinct glyph character \(g_{i}\) as its replacement, according to \(P^{\textsc{GLYPH}}_{\ell_{i}}(g_{i}\mid c_{i})\). Orthographic substitution patterns are language-specific, and thus \(P^{\textsc{GLYPH}}\) is as well. Finally, we typeset and render \(g_{i}\) (instead of \(c_{i}\)) using \(P^{\textsc{TYPE}}(t_{i}\mid g_{i})\) and \(P^{\textsc{Rend}}(x_{i}\mid g_{i},t_{i})\).

We follow the previous Ocular work for the definitions of \(P^{\textsc{LANG}}\), \(P^{\textsc{CHAR}}_{\ell_{i}}\), \(P^{\textsc{TYPE}}\), and \(P^{\textsc{Rend}}\). We define the new conditional distribution \(P^{\textsc{GLYPH}}_{\ell_{i}}\), specifying the probability of rendering \(g\) when the LM generated \(c\), given that the language is \(\ell\), as follows:

Constant \(\kappa\) defines a Bernoulli parameter specifying the fixed probability of deterministically choosing to render \(c_{i}\) directly (i.e., \(g_{i}=c_{i}\)). We set \(\kappa=0.9\) to bias the model away from substitutions in general. The remaining \((1-\kappa)\) probability mass is then divided among all potential output glyph characters.

Table 1 shows some of the common substitution patterns that our model addresses. For a direct rendering of \(c_{i}\), a letter substitution, or the dropping of an accent, \(g_{i}\) will be a simple character drawn from the language's set of valid characters (each language may have a different set of permitted characters, e.g. accented letters). In order to support the tilde-elision shorthand, we permit \(g_{i}\) to be a tilde-annotated version of \(c_{i}\), and for doubled letters, we permit \(g_{i}\) to be \(c_{i}c_{i}\), for which we typeset and render \(c_{i}\) twice. Finally, to allow for elided letters, including the dropping of a line-break hyphen, we allow \(g_{i}\) to be a special elision glyph that renders only as a zero-width space character.

The parameters of the glyph substitution model are learned in an unsupervised fashion as part of Ocular's EM procedure via a hard parameter update:

\[p(g\mid c,\ell)=\frac{\textit{freq}(\ell,c,g)+1}{\sum_{g^{\prime}}(\textit{ freq}(\ell,c,g^{\prime})+1)}\]

where \(\textit{freq}(\ell,c,g)\) is the number of times in the training iteration that the model chose to replace \(c\) with \(g\) in a word (automatically) determined to be of language \(\ell\). The \(+1\) term is Laplace smoothing.

To guide the model and improve efficiency, we employ a number of constraints governing which kinds of substitutions are valid. Among these, we stipulate that substitutions must be letter-to-letter, diacritics may only be added to lowercase letters, only \(s\) can replace long-\(s\), and elision-tilde-marked letters must be followed by one or more elisions.

## 4 Experiments

As a first baseline, we compare against Ocular with no orthographic variation handling, in which characters generated by the LM are rendered directly.

As a second baseline, we compare to our previous work, which improved Ocular's diplomatic transcription accuracy by introducing orthographic variation directly into the LM with hand-constructed language-specific orthographic rules to rewrite the LM training data prior to \(n\)-gram estimation Garrette et al. (2015). However, this rule-based preprocessing approach is inadequate in many ways. First, annotators do not know the full range of orthographic variations, or their frequencies, in each document, and it is impossible to write rules to handle typos. Furthermore, a highly language-proficient

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline  & char sub. & char sub. & elision & accent drop & doubled & typo \\  & (c\(\rightarrow\) q) & (s\(\rightarrow\) long s) & (que \(\rightarrow\) @) & (6 \(\rightarrow\) o) & (c\(\rightarrow\) cc) & (e \(\rightarrow\) r) \\ \hline Original image & quil & **efta** & **a\(\bar{\mathfrak{d}}\)l** & confideración & peccados & Primerarmnete \\ \hline Baseline trans. & **qual** & efña & à ol & confidercasion & peecados & Primeraminte \\ \hline Our diplomatic trans. & **qual** & efta & **a\(\bar{\mathfrak{q}}\)l** & confideración & peecados & Primerarmnte \\ Our normalized trans. & **cual** & esta & **aquel** & consideración & pecados & Primeramente \\ \hline \end{tabular}
\end{table}
Table 1: Examples of automatic diplomatic and normalized transcriptions taken from actual system output.

 

annotator is required, which is not always feasible with, e.g, rare indigenous languages.

Each baseline model has a single output, evaluated against both diplomatic and normalized gold.

Our source code and evaluation data are freely available at [https://github.com/tberg12/ocular](https://github.com/tberg12/ocular) and [https://github.com/dhgarrette/ocr-evaluation-data](https://github.com/dhgarrette/ocr-evaluation-data).

## 5 Results and Discussion

Our results are shown in Table 3 and some correct example system outputs can be seen in Table 1.

Our model's accuracy in producing diplomatic transcriptions is substantially better than baseline Ocular performance, yielding a 35% relative character error rate reduction. Further, our model's diplomatic transcription accuracy is comparable to the LM replacement-rule approach of our previous work, but achieves this result with only unsupervised learning, as opposed to expert-produced rules. We can see in Table 4 that our model is able to discover and apply appropriate probabilities to relevant substitution rules. For the new normalized-transcription task, even larger gains were achieved: a 46% relative error reduction over Ocular, and a 28% reduction over the rule-based approach.

### Error Analysis

Table 2 displays a sample of errors in our system output. (1) The word _das_ is printed without an accent though it has one in modern Spanish. Our system is unable to distinguish between a dot and an accent above the \(i\), and thus it opts to output the accented version since it is preferred by the LM. (2) The word _suplicar_ does not have any accents in modern Spanish, but the model is over-eager in this case and attempts to revive an accent where there should not be one. (3) The word _salvo_ is printed here as the variant _saluo_, but its under-inking leaves the \(u\) in disconnected pieces, resembling a pair of \(i\) characters. The LM model believes this to be the (valid) word _salio_ with the accent dropped and the \(i\) doubled. (4) The model guesses incorrectly that the elision at the end of _alab\(\bar{a}\)_ is an \(r\) since _alabar_ is a valid word. (5) The model correctly recognizes that the letters \(ti\) are printed, but the LM believes the normalized form is \(te\) even though \(te\)_pus_ is not valid Spanish, perhaps because \(te\)_puedo_ is a very common phrase and the six-gram context isn't enough to make that distinction. (6) Finally, there are some special idiomatic shorthands that our model is simply unable to understand because they have no clear

\begin{table}
\begin{tabular}{|l c r|c c|} \hline _Orthographic variation strategy_ & Diplomatic & Normalized & \multicolumn{2}{c|}{} \\ cer & wer & cer & wer \\ \hline No handling & 13.2 & 45.7 & 17.4 & 47.6 \\ Hand-written rules & **8.5** & **30.8** & 13.1 & 37.9 \\ Unsupv. joint model & 8.6 & 32.7 & **9.5** & **27.6** \\ \hline \end{tabular}
\end{table}
Table 3: Experimental results for both Diplomatic (preserving variation) and Normalized (modern orthography) transcription tasks. Results given as both character error rate (cer, including punctuation) and word error rate (wer, without punctuation).

 

connection to what they are replacing. Here, _x\(\bar{p}o\)_ (from Greek letters _chi rho_) is shorthand for _Cristo_.

## 6 Conclusion

In this paper we presented a novel unsupervised OCR model for the joint production of diplomatic (variant-preserving) and normalized transcriptions of historical documents. The model is able to automatically learn a probabilistic mapping from a LM trained on modern text to the orthographic variants present in the document. This has the dual result of both considerably improving diplomatic transcription accuracy, while also enabling the model to simultaneously produce a normalized transcription.

Our model has the potential to significantly impact the work of scholars and librarians who wish to make digital texts easier to read, index, search, and study. Our approach also has the fortunate side-effect of producing metadata about orthographic variation in printed documents that may be valuable to scholars of book history. With our model, we are able to automatically induce sets of variation patterns used by printers, and the locations in the texts where those variants appear, without the need for labor-intensive page-by-page reading. Further, these induced mappings have probabilities attached, and are not simple rulebanks like those used in existing normalization work Garrette et al. (2015); Baron and Rayson (2008). Table 4, above, shows a sample of rules and their frequencies that resulted from training our model on Spanish documents.

Finally, our work helps to bridge the gap between historical text and mainline NLP. Orthographic variation lowers the accuracy of NLP tools due to high out-of-vocabulary rates and mismatched morphological features Piotrowski (2012). This is especially true when these tools are trained on the modern texts of the standard corpora used in NLP Yang and Eisenstein (2016). Normalization of historical texts have been shown to improve the quality of, for example, taggers Rayson et al. (2007); Yang and Eisenstein (2016) and parsers Rayson et al. (2007). These techniques mirror those applied to the processing of text in social media, such as Twitter, where there is a high degree of slang and shorthand Gimpel et al. (2011); Eisenstein (2013); Yang and Eisenstein (2013). Most approaches train off-the-shelf NLP tools on modern text and then apply normalization techniques to historical texts to transform them into something resembling the modern training input Scheible et al. (2011). A joint model such as ours that automatically learns orthographic variations while training the NLP model might overcome some of the limitations of using such a pipeline approach.

We would like to thank Stephanie Wood, Kelly McDonough, Albert Palacios, Adam Coon, Sergio Romero, and Kent Norsworthy for their input, advice, and assistance on this project. We would also like to thank Taylor Berg-Kirkpatrick, Dan Klein, and Luke Zettlemoyer for their valuable feedback on earlier drafts of this paper. This work is supported in part by a Digital Humanities Implementation Grant from the National Endowment for the Humanities for the project Reading the First Books: Multilingual, Early-Modern OCR for Primeros Libros.

\begin{table}
\begin{tabular}{c c} _Original Image_ & \\ dc las dos que fe figuen en las quales apro & \\ necha mucho acoftubrar el anima à fe lei & \\ pantar hazia arriba ponefe aquí vn modo & \\ el qual parece mas coforme al o & fé figure \\ \end{tabular}
\end{table}
Table 5: A document excerpt along with actual system outputs for both diplomatic and normalized transcriptions. Note that the normalization recovers the first line’s missing line-break hyphen, allowing the full word _aprovecha_ to be reassembled.

 



## References

* [Baron and Rayson2008] Alistair Baron and Paul Rayson. 2008. VARD2: A tool for dealing with spelling variation in historical corpora. In _Proceedings of The Postgraduate Conference in Corpus Linguistics_.
* Berg-Kirkpatrick and Klein2014 Taylor Berg-Kirkpatrick and Dan Klein. 2014. Improved typesetting models for historical OCR. In _ACL_.
* Berg-Kirkpatrick et al.2013 Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013 Unsupervised transcription of historical documents. In _Proceedings of ACL_.
* Burns2013 Philip R. Burns. 2013 MorphAdorner v2: A Java library for the morphological adornment of English language texts. [https://morphadorner.northwestern.edu/morphadorner/download/morphadorner.pdf](https://morphadorner.northwestern.edu/morphadorner/download/morphadorner.pdf).
* Dempster et al.197 Arthur P. Dempster, Nan M. Laird, and Donald. B. Rubin. 1977 Maximum likelihood from incomplete data via the EM algorithm. _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_, 39.
* Dolan2012 Thomas G. Dolan. 2012 The Primeros Libros Project. _The Hispanic Outlook in Higher Education_, 22:20-22, March.
* Driscoll2006 M. J. Driscoll. 2006 Electronic textual editing: Levels of transcription. In Lou Burnard, Katherine O'Brien O'Keeffe, and John Unsworth, editors, _Electronic Textual Editing_. Modern Language Association.
* Eisenstein2013 Jacob Eisenstein. 2013 What to do about bad language on the internet. In _Proceedings of NAACL_.
* Garrette et al.2015 Dan Garrette, Hannah Alpert-Abrams, Taylor Berg-Kirkpatrick, and Dan Klein. 2015 Unsupervised code-switching for multilingual historical document transcription. In _Proceedings of NAACL_.
* Gimpel et al.2011 Kevin Gimpel, Nathan Schneider, Brendan OConnor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, , and Noah A. Smith. 2011. Part-of-speech tagging for twitter: annotation, features, and experiments. In _Proceedings of ACL_.
* Greg1950 W. W. Greg. 1950 The rationale of copy-text. _Studies in Bibliography_, 3:19-36.
* Hendrickx and Marquilhas2011 Iris Hendrickx and Rita Marquilhas. 2011 From old texts to modern spellings: An experiment in automatic normalisation. _Journal of Language Technology and Computational Linguistics_, 26(2):65-76.
* Kneser and Ney1995 Reinhard Kneser and Hermann Ney. 1995 Improved backing-off for m-gram language modeling. In _Proceedings of the International Conference on Acoustics, Speech, and Signal Processing_.
* Pierazzo2014 Elena Pierazzo. 2014 Digital documentary editions and the Others. _Scholarly Editing_, 35:1-23.
* Piotrowski2012 Michael Piotrowski. 2012 Natural language processing for historical texts. _Synthesis Lectures on Human Language Technologies_, 5:1-157.
* Libros2010 Primeros Libros. 2010 Los Primeros Libros de las Americas: Impresos Americanos del siglo XVI en las bibliottecas del mudo. _[http://primeroslibros.org/_](http://primeroslibros.org/_).
* Rayson et al.2007 Paul Rayson, Dawn Archer, Alistair Baron, Jonathan Culpeper, and Nicholas Smith. 2007 Tagging the bard: Evaluating the accuracy of a modern POS tagger on early modern English corpora. In _Corpus Linguistics Conference_.
* Scheible et al.2011 Silke Scheible, Richard J Whitt, Martin Durrell, and Paul Bennett. 2011 Evaluating an 'off-the-shelf' POStagger on early modern German text. In _Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities_.
* Yang and Eisenstein2013 Yi Yang and Jacob Eisenstein. 2013 Ai log-linear model for unsupervised text normalization. In _Proceedings of EMNLP_.
* Yang and Eisenstein2016 Yi Yang and Jacob Eisenstein. 2016 Part-of-speech tagging for historical English. In _Proceedings of NAACL_.