SlowGuess/ACL_2020_OCR huggingface dataset:
Kartik Goyal<sup>1</sup> Chris Dyer<sup>2</sup> Christopher Warren<sup>3</sup> Max G'Sell<sup>4</sup> Taylor Berg-Kirkpatrick<sup>5</sup>
$^{1}$ Language Technologies Institute, Carnegie Mellon University  $^{2}$ Deepmind
$^{3}$ Department of English, Carnegie Mellon University
$^{4}$ Department of Statistics, Carnegie Mellon University
<sup>5</sup>Computer Science and Engineering, University of California, San Diego
{kartikgo,cnwarren,mgsell}@andrew.cmu.edu
cdyer@google.com tberg@eng.ucsd.edu
We propose a deep and interpretable probabilistic generative model to analyze glyph shapes in printed Early Modern documents. We focus on clustering extracted glyph images into underlying templates in the presence of multiple confounding sources of variance. Our approach introduces a neural editor model that first generates well-understood printing phenomena like spatial perturbations from template parameters via interpretable latent variables, and then modifies the result by generating a non-interpretable latent vector responsible for inkling variations, jitter, noise from the archiving process, and other unforeseen phenomena associated with Early Modern printing. Critically, by introducing an inference network whose input is restricted to the visual residual between the observation and the interpretably-modified template, we are able to control and isolate what the vector-valued latent variable captures. We show that our approach outperforms rigid interpretable clustering baselines (Ocular) and overly-flexible deep generative models (VAE) alike on the task of completely unsupervised discovery of typefaces in mixed-font documents.
Scholars interested in understanding details related to production and provenance of historical documents rely on methods of analysis ranging from the study of orthographic differences and stylometrics, to visual analysis of layout, font, and printed characters. Recently developed tools like Ocular (Berg-Kirkpatrick et al., 2013) for OCR of historical documents have helped automate and scale some textual analysis methods for tasks like compositor attribution (Ryskina et al., 2017) and digitization of historical documents (Garrette et al., 2015). However, researchers often find the need to go beyond
Figure 1: We desire a generative model that can be biased to cluster according to typeface characteristics (e.g. the length of the middle arm) rather than other more visually salient sources of variation like inking.
textual analysis for establishing provenance of historical documents. For example, Hinman (1963)'s study of typesetting in Shakespeare's First Folio relied on the discovery of pieces of damaged or distinctive type through manual inspection of every glyph in the document. More recently, Warren et al. (2020) examine pieces of distinctive types across several printers of the early modern period to posit the identity of clandestine printers of John Milton's Areopagitica (1644). In such work, researchers frequently aim to determine whether a book was produced by a single or multiple printers (Weiss (1992); Malcolm (2014); Takano (2016)). Hence, in order to aid these visual methods of analyses, we propose here a novel probabilistic generative model for analyzing extracted images of individual printed characters in historical documents. We draw from work on both deep generative modeling and interpretable models of the printing press to develop an approach that is both flexible and controllable – the later being a critical requirement for such analysis tools.
As depicted in Figure 1, we are interested in identifying clusters of subtly distinctive glyph shapes as these correspond to distinct metal stamps in the type-cases used by printers. However, other
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2954-2960 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sources of variation (inking, for example, as depicted in Figure 1) are likely to dominate conventional clustering methods. For example, powerful models like the variational autoencoder (VAE) (Kingma and Welling, 2014) capture the more visually salient variance in inking rather than typeface, while more rigid models (e.g. the emission model of Ocular (Berg-Kirkpatrick et al., 2013)), fail to fit the data. The goal of our approach is to account for these confounding sources of variance, while isolating the variables pertinent to clustering.
Hence, we propose a generative clustering model that introduces a neural editing process to add expressivity, but includes interpretable latent variables that model well-understood variance in the printing process: bi-axial translation, shear, and rotation of canonical type shapes. In order to make our model controllable and prevent deep latent variables from explaining all variance in the data, we introduce a restricted inference network. By only allowing the inference network to observe the visual residual of the observation after interpretable modifications have been applied, we bias the posterior approximation on the neural editor (and thus the model itself) to capture residual sources of variance in the editor – for example, inkling levels, ink bleeds, and imaging noise. This approach is related to recently introduced neural editor models for text generation (Guu et al., 2018).
In experiments, we compare our model with rigid interpretable models (Ocular) and powerful generative models (VAE) at the task of unsupervised clustering subtly distinct typeface in scanned images early modern documents sourced from Early English Books Online (EEBO).
Our model reasons about the printed appearances of a symbol (say majuscule F) in a document via a mixture model whose  $K$  components correspond to different metal stamps used by a printer for the document. During various stages of printing, random transformations result in varying printed manifestations of a metal cast on the paper. Figure 2 depicts our model. We denote an observed image of the extracted character by  $X$ . We denote choice of typeface by latent variable  $c$  (the mixture component) with prior  $\pi$ . We represent the shape of the  $k$ -th stamp by template  $T_{k}$ , a square matrix of parameters. We denote the interpretable latent variables corresponding to spatial adjustment of
Figure 2: Proposed generative model for clustering images of a symbol by typeface. Each mixture component  $c$  corresponds to a learnable template  $T_{k}$ . The  $\lambda$  variables warp (spatially adjust) the original template  $T$  to  $\tilde{T}$ . This warped template is then further transformed via the  $z$  variables to  $\hat{T}$  via an expressive neural filter function parametrized by  $\theta$ .
the metal stamp by  $\lambda$ , and the editor latent variable responsible for residual sources of variation by  $z$ . As illustrated in Fig. 2, after a cluster component  $c = k$  is selected, the corresponding template  $T_{k}$  undergoes a transformation to yield  $\hat{T}_k$ . This transformation occurs in two stages: first, the interpretable spatial adjustment variables  $(\lambda)$  produce an adjusted template ( $\S 2.1$ ),  $\hat{T}_k = \mathrm{warp}(T_k,\lambda)$ , and then the neural latent variable transforms the adjusted template ( $\S 2.2$ ),  $\hat{T}_k = \mathrm{filter}(\tilde{T}_k,z)$ . The marginal probability under our model is
$$
p (X) = \sum_ {k} \pi_ {k} \int p (X | \lambda , z; T _ {k}) p (\lambda) p (z) d z d \lambda ,
$$
where  $p(X|\lambda, z; T_k)$  refers to the distribution over the binary pixels of  $X$  where each pixel has a bernoulli distribution parametrized by the value of the corresponding pixel-entry in  $\hat{T}_k$ .
2.1 Interpretable spatial adjustment
Early typesetting was noisy, and the metal pieces were often arranged with slight variations which resulted in the printed characters being positioned with small amounts of offset, rotation and shear. These real-valued spatial adjustment variables are denoted by  $\lambda = (r,o,s,a)$ , where  $r$  represents the rotation variable,  $o = (o_h,o_v)$  represents offsets along the horizontal and vertical axes,  $s = (s_h,s_v)$
2955
denotes shear along the two axes. A scale factor,  $\tilde{a} = 1.0 + a$ , accounts for minor scale variations arising due to the archiving and extraction processes. All variables in  $\lambda$  are generated from a Gaussian prior with zero mean and fixed variance as the transformations due to these variables tend to be subtle.
In order to incorporate these deterministic transformations in a differentiable manner, we map  $\lambda$  to a template sized attention map  $H_{ij}$  for each output pixel position  $(i,j)$  in  $\tilde{T}$  as depicted in Figure 3. The attention map for each output pixel is formed in order to attend to the corresponding shifted (or scaled or sheared) portion of the input template and is shaped according to a Gaussian distribution with mean determined by an affine transform. This approach allows for strong inductive bias which contrasts with related work on spatial-VAE (Bepler et al., 2019) that learns arbitrary transformations.
Figure 3: Translation operation: The mode of the attention map is shifted by the offset values for every output pixel in  $\tilde{T}$ . Similar operations account for shear, rotation, and scale.
2.2 Residual sources of variations
Apart from spatial perturbations, other major sources of deviation in early printing include random inking perturbations caused by inconsistent application of the stamps, unpredictable ink bleeds, and noise associated with digital archiving of the documents. Unlike in the case of spatial perturbations which could be handled by deterministic affine transformation operators, it is not possible to analytically define a transformation operator due to these variables. Hence we propose to introduce a non-interpretable real-valued latent vector  $z$ , with a Gaussian prior  $\mathcal{N}(\mathbf{0},\mathbf{I})$ , that transforms  $\tilde{T}$  into a final template  $\hat{T}$  via neurally-parametrized function filter  $(\tilde{T},z;\theta)$  with neural network parameters  $\theta$ . This function is a convolution over  $\tilde{T}$  whose kernel is parametrized by  $z$ , followed by non-linear operations. Intuitively, parametrizing the filter by  $z$  results in the latent variable accounting for variations like ink ing appropriately because convolution filters capture local variations in appearance. Srivatsan et al. (2019) also observed the effectiveness of using  $z$  to define a deconvolutional kernel for
Figure 4: Inference network for  $z$  conditions on the mixture component and only the residual image left after subtracting the  $\lambda$ -transformed template from the image. This encourages  $z$  to model variance due to sources other than spatial adjustments.
font generation.
2.3 Learning and Inference
Our aim is to maximize the log likelihood of the observed data  $\left(\{X_{d} \mid d \in \mathbb{N}, d < n\}\right)$  of  $n$  images wrt. model parameters:
$$
\operatorname {L L} (T _ {1, \dots , k}, \theta) = \max _ {T, \theta} \sum_ {d} \log \left[ \sum_ {k} \pi_ {k} \right.
$$
$$
\int p (X _ {d} | \lambda_ {d}, z _ {d}; T _ {k}, \theta) p (\lambda_ {d}) p (z _ {d}) d z _ {d} d \lambda_ {d} \Big ]
$$
During training, we maximize the likelihood wrt.  $\lambda$  instead of marginalizing, which is an approximation inspired by iterated conditional modes (Besag, 1986):
$$
\max  _ {T, \theta} \sum_ {d} \log \sum_ {k} \max  _ {\gamma_ {k, d}} \pi_ {k} \int p (X _ {d} | \lambda_ {d} = \gamma_ {k, d}, z _ {d};
$$
$$
T _ {k}, \theta) p (\lambda_ {d} = \gamma_ {k, d}) p (z _ {d}) d z _ {d}
$$
However, marginalizing over  $z$  remains intractable. Therefore we perform amortized variational inference to define and maximize a lower bound on the above objective (Kingma and Welling, 2014). We use a convolutional inference neural network parametrized by  $\phi$  (Fig. 4), that takes as input, the mixture component  $k$ , the residual image  $R_{k} = X - \tilde{T}_{k}$ , and produces mean and variance parameters for an isotropic gaussian proposal distribution  $q(z \mid R_{k}, k; \phi)$ . This results in the final training objective:
$$
\begin{array}{l} \max  _ {T, \theta , \phi} \sum_ {d} \log \sum_ {k} \mathrm {E} _ {q (z _ {d} | R _ {d, k}, k; \phi)} \left[ \max  _ {\gamma_ {k, d}} \left(\pi_ {k} \right. \right. \\ \left. p (X _ {d} | \lambda = \gamma_ {k, d}, z _ {d}; T _ {k}, \theta) p (\lambda = \gamma_ {k, d}) \right] \\ - \mathrm {K L} \left(q \left(z _ {d} \mid R _ {d, k}, k; \phi\right) \| p (z)\right) \\ \end{array}
$$
We use stochastic gradient ascent to maximize this objective with respect to  $T, \gamma, \theta$  and  $\phi$ .
We train our models on printed occurrences of 10 different uppercase character classes that scholars have found useful for bibliographic analysis (Warren et al., 2020) because of their distinctiveness. As a preprocessing step, we ran Ocular (Berg-Kirkpatrick et al., 2013) on the grayscale scanned images of historical books in EEBO dataset and extracted the estimated image segments for the letters of interest.
3.1 Quantitative analysis
We show that our model is superior to strong baselines at clustering subtly distinct typefaces (using realistic synthetic data), as well as in terms of fitting the real data from historical books.
3.1.1 Baselines for comparison
Ocular: Based on the emission model of Ocular that uses discrete latent variables for the vertical/horizontal offset and inking variables, and hence has limited expressivity.
$\lambda$ -only: This model only has the interpretable continuous latent variables pertaining to spatial adjustment.
VAE-only: This model is expressive but doesn't have any interpretable latent variables for explicit control. It is an extension of Kingma et al. (2014)'s model for semi-supervised learning with a continuous latent variable vector in which we obtain tighter bounds by marginalizing over the cluster identities explicitly. For fair comparison, the encoder and decoder convolutional architectures are the same as the ones in our full model. The corresponding training objective for this baseline is:
$$
\begin{array}{l} \max  _ {T, \theta , \phi} \sum_ {d} \log \sum_ {k} \operatorname {E} _ {q (z _ {d} | X _ {d}, k; \phi)} \left[ \pi_ {k} p (X _ {d} | z _ {d}; T _ {k}, \theta) \right] \\ - \mathrm {K L} \left(q \left(z _ {d} \mid X _ {d}, k; \phi\right) \| p (z)\right) \\ \end{array}
$$
No-residual: The only difference from the full model is that the encoder for the inference network conditions the variational distribution  $q(z)$  on the entire input image  $X$  instead of just the residual image  $X - \tilde{T}$ .
3.1.2 Font discovery in Synthetic Data
Early modern books were frequently composed from two or more type cases, resulting in documents with mixed fonts. We aim to learn the dif
V-measure	Mutual Info	F&M	NLL
Ocular	0.42	0.45	0.61	379.21
λ-only	0.49	0.51	0.70	322.04
VAE-only	0.22	0.29	0.38	263.45
No-residual	0.54	0.58	0.73	264.27
Our Model	0.73	0.74	0.85	257.92
Table 1: (a) Clustering results on synthetic data (V-measure, Mutual Info, F&M). (b) Test negative log likelihood (NLL) on real data from historical documents, or negative ELBO bound for intractable models (NLL).
ferent shapes of metal stamps that were used as templates for each cluster component in our model.
Data: In order to quantitatively evaluate our model's performance, we experiment with synthetically generated realistic dataset for which we know the ground truth cluster identities in the following manner: For each character of interest, we pick three distinct images from scanned segmented EEBO images, corresponding to three different metal casts. Then we randomly add spatial perturbations related to scale, offset, rotation and shear. To incorporate varying inking levels and other distortions, we randomly either perform erosion, dilation, or a combination of these warpings using OpenCV (Bradski, 2000) with randomly selected kernel sizes. Finally, we add a small Gaussian noise to the pixel intensities and generate 300 perturbed examples per character class.
Results: We report macro-averaged results across all the character classes on three different clustering measures, V-measure (Rosenberg and Hirschberg, 2007), Mutual Information and Fowlkes and Mallows Index (Fowlkes and Mallows, 1983). In Table 1, we see that our model significantly outperforms all other baselines on every metric. Ocular and  $\lambda$ -only models fail because they lack expressiveness to explain the variations due to random jitters, erosions and dilations. The VAE-only model, while very expressive, performs poorly because it lacks the inductive bias needed for successful clustering. The No-residual model performs decently but our model's superior performance emphasizes the importance of designing a restrictive inference network such that  $z$  only focuses on extraneous sources of variation.
3.1.3 Fitting Real Data from Historical Books
For the analysis of real books, we selected three books from the EEBO dataset printed by different printers. We modeled each character class for each book separately and report the macro-aggregated
upper bounds on the negative log likelihood (NLL) in Table 1. We observe that adding a small amount of expressiveness makes our  $\lambda$ -only model better than Ocular. The upper bounds of other inference network based models are much better than the tight<sup>1</sup> bounds of both the interpretable models. Our model has the lowest upper bound of all the models while retaining interpretability and control.
3.2 Qualitative analysis
We provide visual evidence of desirable behavior of our model on collections of character extractions from historical books with mixed fonts. Specifically, we discuss the performance of our model on the mysterious edition of Thomas Hobbes' Leviathan known as "the 25 Ornaments" edition. (Hobbes, 1651 [really 1700?]). The 25 Ornaments Leviathan is an interesting test case for several reasons. While its title page indicates a publisher and year of publication, both are fabricated (Malcolm, 2014). The identities of its printer(s) remain speculative, and the actual year of publication is uncertain. Further, the 25 Ornaments exhibits two distinct fonts.
3.2.1 Quality of learned templates
Figure 5: The learned templates for  $\mathbf{F}$  and  $\mathbf{R}$  and the transformed templates  $\hat{T}$  for four examples of  $\mathbf{F}$  are shown. Our model is able to learn desirable templates based on underlying glyph structure.
Our model is successful in discovering distinctly shaped typefaces in the 25 Ornaments Leviathan. We focus on the case study of majuscule letters  $\mathbf{F}$  and  $\mathbf{R}$ , each of which have two different typefaces mixed in throughout. The two typefaces for  $\mathbf{F}$  differ in the length of the middle arm (Fig. 1), and the two typefaces for  $\mathbf{R}$  have differently shaped legs. In Fig. 5, we show that our model successfully learns the two desired templates  $T_{1}$  and  $T_{2}$  for both the characters which indicates that the clusters in our
model mainly focus on subtle differences in underlying glyph shapes. We also illustrate how the latent variables transform the model templates  $T$  to  $\hat{T}$  for four example  $\mathbf{F}$  images. The model learns complex functions to transform the templates which go beyond simple affine and morphological transformations in order to account for inking differences, random jitter, contrast variations etc.
3.2.2 Interpretable variables  $(\lambda)$  and Control
Unaligned raw Images
Aligned Images
Figure 6: Result of alignment on Leviathan extractions using the interpretable  $\lambda$  variables along with their pixelwise average images. Aligned average image is much sharper than the unaligned average image.
Finally, we visualize the ability of our model to separate responsibility of modelling variation among the interpretable and non-interpretable variables appropriately. We use the inferred values of the interpretable  $(\lambda)$  variable for each image in the dataset to adjust the corresponding image. Since the templates represent the canonical shape of the letters, the  $\lambda$  variables which shift the templates to explain the images can be reverse applied to the input images themselves in order to align them by accounting for offset, rotation, shear and minor size variations. In Fig. 6, we see that the input images (top row) are uneven and vary by size and orientation. By reverse applying the inferred  $\lambda$  values, we are able to project the images to a fixed size such that they are aligned and any remaining variations in the data are caused by other sources of variation. Moreover, this alignment method would be crucial for automating certain aspects of bibliographic studies that focus on comparing specific imprints.
Beyond applications to typeface clustering, the general approach we take might apply more broadly to other clustering problems, and the model we developed might be incorporated into OCR models for historical text.
This project is funded in part by the NSF under grants 1618044 and 1936155, and by the NEH under grant HAA256044-17.
For Ocular and  $\lambda$ -only models, we report the upper bound obtained via maximization over the interpretable latent variables. Intuitively, these latent variables are likely to have unimodal posterior distributions with low variance, hence this approximation is likely tight.
Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. 2019. Explicitly disentangling image content from translation and rotation with spatial-vae. In Advances in Neural Information Processing Systems, pages 15409-15419.
Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. Unsupervised transcription of historical documents. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 207-217, Sofia, Bulgaria. Association for Computational Linguistics.
Julian Besag. 1986. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society: Series B (Methodological), 48(3):259-279.
G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools.
Edward B Fowlkes and Colin L Mallows. 1983. A method for comparing two hierarchical clusterings. Journal of the American statistical association, 78(383):553-569.
Dan Garrette, Hannah Alpert-Abrams, Taylor Berg-Kirkpatrick, and Dan Klein. 2015. Unsupervised code-switching for multilingual historical document transcription. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1036-1041, Denver, Colorado. Association for Computational Linguistics.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Charlton Hinman. 1963. The printing and proof-reading of the first folio of Shakespeare, volume 1. Oxford: Clarendon Press.
Thomas Hobbes. 1651 [really 1700?]. Leviathan, or, the matter, form, and power of a common-wealth ecclesiastical and civil. By Thomas Hobbes of Malmesbury. Number R13935 in ESTC. [false imprint] printed for Andrew Crooke, at the Green Dragon in St. Pauls Church-yard, London.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581-3589.
Noel Malcolm. 2014. Editorial Introduction. In Leviathan, volume 1. Clarendon Press, Oxford.