Venue: ACL 2020

# A Probabilistic Generative Model for

Typographical Analysis of Early Modern Printing

Kartik Goyal\({}^{1}\) Chris Dyer\({}^{2}\) Christopher Warren\({}^{3}\) Max G'Sell\({}^{4}\) Taylor Berg-Kirkpatrick\({}^{5}\)

\({}^{1}\)Language Technologies Institute, Carnegie Mellon University \({}^{2}\)Deepmind

\({}^{3}\)Department of English, Carnegie Mellon University

\({}^{4}\)Department of Statistics, Carnegie Mellon University

\({}^{5}\)Computer Science and Engineering, University of California, San Diego

{kartikgo,cnwarren,mgsell}@andrew.cmu.edu

cdyer@google.com tberg@eng.ucsd.edu

###### Abstract

We propose a deep and interpretable probabilistic generative model to analyze glyph shapes in printed Early Modern documents. We focus on clustering extracted glyph images into underlying templates in the presence of multiple confounding sources of variance. Our approach introduces a neural editor model that first generates well-understood printing phenomena like spatial perturbations from template parameters via interpertable latent variables, and then modifies the result by generating a non-interpretable latent vector responsible for mixing variations, jitter, noise from the archiving process, and other unforeseen phenomena associated with Early Modern printing. Critically, by introducing an inference network whose input is restricted to the visual residual between the observation and the interpretably-modified template, we are able to control and isolate what the vector-valued latent variable captures. We show that our approach outperforms rigid interpretable clustering baselines (Ocular) and overly-flexible deep generative models (VAE) alike on the task of _completely unsupervised_ discovery of typefaces in mixed-font documents.

+
Footnote †: July 5 - 10, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

Scholars interested in understanding details related to production and provenance of historical documents rely on methods of analysis ranging from the study of orthographic differences and stylometrics, to visual analysis of layout, font, and printed characters. Recently developed tools like Ocular [8] for OCR of historical documents have helped automate and scale some textual analysis methods for tasks like compositor attribution [6] and digitization of historical documents [7]. However, researchers often find the need to go beyond textual analysis for establishing provenance of historical documents. For example, Hinman (1963)'s study of typesetting in Shakespeare's First Folio relied on the discovery of pieces of damaged or distinctive type through manual inspection of every glyph in the document. More recently, Warren et al. (2020) examine pieces of distinctive types across several printers of the early modern period to posit the identity of clandestine printers of John Milton's _Areopagitica_ (1644). In such work, researchers frequently aim to determine whether a book was produced by a single or multiple printers [6, 10]. Hence, in order to aid these visual methods of analyses, we propose here a novel probabilistic generative model for analyzing extracted images of individual printed characters in historical documents. We draw from work on both deep generative modeling and interpretable models of the printing press to develop an approach that is both flexible and controllable - the later being a critical requirement for such analysis tools.

As depicted in Figure 1, we are interested in identifying clusters of subtly distinctive glyph shapes as these correspond to distinct metal stamps in the type-cases used by printers. However, other

Figure 1: We desire a generative model that can be biased to cluster according to typeface characteristics (e.g. the length of the middle arm) rather than other more visually salient sources of variation like linking.



sources of variation (inking, for example, as depicted in Figure 1) are likely to dominate conventional clustering methods. For example, powerful models like the variational autoencoder (VAE) Kingma and Welling (2014) capture the more visually salient variance in sinking rather than typeface, while more rigid models (e.g. the emission model of Ocular Berg-Kirkpatrick et al. (2013)), fail to fit the data. The goal of our approach is to account for these confounding sources of variance, while isolating the variables pertinent to clustering.

Hence, we propose a generative clustering model that introduces a neural editing process to add _expressivity_, but includes _interpretable_ latent variables that model well-understood variance in the printing process: bi-axial translation, shear, and rotation of canonical type shapes. In order to make our model _controllable_ and prevent deep latent variables from explaining all variance in the data, we introduce a restricted inference network. By only allowing the inference network to observe the visual residual of the observation after interpretable modifications have been applied, we bias the posterior approximation on the neural editor (and thus the model itself) to capture residual sources of variance in the editor - for example, linking levels, ink bleeds, and imaging noise. This approach is related to recently introduced neural editor models for text generation Guu et al. (2018).

In experiments, we compare our model with rigid interpretable models (Ocular) and powerful generative models (VAE) at the task of unsupervised clustering subtly distinct typeface in scanned images early modern documents sourced from Early English Books Online (EEBO).

## 2 Model

Our model reasons about the printed appearances of a symbol (say majuscule F) in a document via a mixture model whose \(K\) components correspond to different metal stamps used by a printer for the document. During various stages of printing, random transformations result in varying printed manifestations of a metal cast on the paper. Figure 2 depicts our model. We denote an observed image of the extracted character by \(X\). We denote choice of typeface by latent variable \(c\) (the mixture component) with prior \(\pi\). We represent the shape of the \(k\)-th stamp by template \(T_{k}\), a square matrix of parameters. We denote the interpretable latent variables corresponding to spatial adjustment of the metal stamp by \(\lambda\), and the editor latent variable responsible for residual sources of variation by \(z\). As illustrated in Fig. 2, after a cluster component \(c=k\) is selected, the corresponding template \(T_{k}\) undergoes a transformation to yield \(\hat{T}_{k}\). This transformation occurs in two stages: first, the interpretable spatial adjustment variables (\(\lambda\)) produce an adjusted template (SS2.1), \(\hat{T}_{k}=\text{warp}(T_{k},\lambda)\), and then the neural latent variable transforms the adjusted template (SS2.2), \(\hat{T}_{k}=\text{filter}(\bar{T}_{k},z)\). The marginal probability under our model is

\[p(X)=\sum_{k}\pi_{k}\int p(X|\lambda,z;T_{k})p(\lambda)p(z)dzd\lambda,\]

where \(p(X|\lambda,z;T_{k})\) refers to the distribution over the binary pixels of \(X\) where each pixel has a bernoulli distribution parametrized by the value of the corresponding pixel-entry in \(\hat{T}_{k}\).

### Interpretable spatial adjustment

Early typesetting was noisy, and the metal pieces were often arranged with slight variations which resulted in the printed characters being positioned with small amounts of offset, rotation and shear. These real-valued spatial adjustment variables are denoted by \(\lambda=(r,o,s,a)\), where \(r\) represents the rotation variable, \(o=(o_{h},o_{v})\) represents offsets along the horizontal and vertical axes, \(s=(s_{h},s_{v})\)

Figure 2: Proposed generative model for clustering images of a symbol by typeface. Each mixture component \(c\) corresponds to a _learnable_ template \(T_{k}\). The \(\lambda\) variables warp (spatially adjust) the original template \(T\) to \(\hat{T}\). This warped template is then further transformed via the \(z\) variables to \(\hat{T}\) via an expressive neural filter function parametrized by \(\theta\).

 

denotes shear along the two axes. A scale factor, \(\tilde{a}=1.0+a\), accounts for minor scale variations arising due to the archiving and extraction processes. All variables in \(\lambda\) are generated from a Gaussian prior with zero mean and fixed variance as the transformations due to these variables tend to be subtle.

In order to incorporate these deterministic transformations in a differentiable manner, we map \(\lambda\) to a template sized attention map \(H_{ij}\) for each output pixel position \((i,j)\) in \(\tilde{T}\) as depicted in Figure 3. The attention map for each output pixel is formed in order to attend to the corresponding shifted (or scaled or sheared) portion of the input template and is shaped according to a Gaussian distribution with mean determined by an affine transform. This approach allows for strong inductive bias which contrasts with related work on _spatial-VAE_(Bepler et al., 2019) that learns arbitrary transformations.

### Residual sources of variations

Apart from spatial perturbations, other major sources of deviation in early printing include random linking perturbations caused by inconsistent application of the stamps, unpredictable ink bleeds, and noise associated with digital archiving of the documents. Unlike in the case of spatial perturbations which could be handled by deterministic affine transformation operators, it is not possible to analytically define a transformation operator due to these variables. Hence we propose to introduce a non-interpretable real-valued latent vector \(z\), with a Gaussian prior \(\mathcal{N}(\mathbf{0},\mathbf{I})\) , that transforms \(\tilde{T}\) into a final template \(\hat{T}\) via neurally-parametrized function filter\((\tilde{T},z;\theta)\) with neural network parameters \(\theta\). This function is a convolution over \(\tilde{T}\) whose kernel is parametrized by \(z\), followed by non-linear operations. Intuitively, parametrizing the filter by \(z\) results in the latent variable accounting for variations likeinking appropriately because convolution filters capture local variations in appearance. Srivatsan et al. (2019) also observed the effectiveness of using \(z\) to define a deconvolutional kernel for font generation.

### Learning and Inference

Our aim is to maximize the log likelihood of the observed data (\(\{X_{d}\mid d\in\mathbb{N},d<n\}\)) of \(n\) images wrt. model parameters:

\[\text{LL}(T_{1,...,k},\theta)=\max_{T,\theta}\sum_{d}\log\Big{[} \sum_{k}\pi_{k}\] \[\int p(X_{d}|\lambda_{d},z_{d};T_{k},\theta)p(\lambda_{d})p(z_{d} )dz_{d}d\lambda_{d}\Big{]}\]

During training, we maximize the likelihood wrt. \(\lambda\) instead of marginalizing, which is an approximation inspired by iterated conditional modes (Besag, 1986):

\[\max_{T,\theta}\sum_{d}\log\sum_{k}\max_{\gamma_{k,d}}\pi_{k} \int p(X_{d}|\lambda_{d}=\gamma_{k,d},z_{d};\] \[T_{k},\theta)p(\lambda_{d}=\gamma_{k,d})p(z_{d})dz_{d}\]

However, marginalizing over \(z\) remains intractable. Therefore we perform amortized variational inference to define and maximize a lower bound on the above objective (Kingma and Welling, 2014). We use a convolutional inference neural network parametrized by \(\phi\) (Fig. 4), that takes as input, the mixture component \(k\), the residual image \(R_{k}=X-\tilde{T}_{k}\), and produces mean and variance parameters for an isotropic gaussian proposal distribution \(q(z\mid R_{k},k;\phi)\). This results in the final training objective:

\[\max_{T,\theta,\phi}\sum_{d}\log\sum_{k}\mathrm{E}_{q(z_{d}|R_{d ,k},k;\phi)}\big{[}\max_{\gamma_{k,d}}\big{(}\pi_{k}\] \[p(X_{d}|\lambda=\gamma_{k,d},z_{d};T_{k},\theta)p(\lambda=\gamma_ {k,d})\big{)}\big{]}\] \[-\text{KL}\big{(}q(z_{d}|R_{d,k},k;\phi)||p(z)\big{)}\]

Figure 4: Inference network for \(z\) conditions on the mixture component and only the residual image left after subtracting the \(\lambda\)-transformed template from the image. This encourages \(z\) to model variance due to sources other than spatial adjustments.

Figure 3: Translation operation: The mode of the attention map is shifted by the offset values for every output pixel in \(\tilde{T}\). Similar operations account for shear, rotation, and scale.

 

We use stochastic gradient ascent to maximize this objective with respect to \(T,\gamma,\theta\) and \(\phi\).

## 3 Experiments

We train our models on printed occurrences of 10 different uppercase character classes that scholars have found useful for bibliographic analysis Warren et al. (2020) because of their distinctiveness. As a preprocessing step, we ran Ocular (Bergkirkpatrick et al., 2013) on the grayscale scanned images of historical books in EEBO dataset and extracted the estimated image segments for the letters of interest.

### Quantitative analysis

We show that our model is superior to strong baselines at clustering subtly distinct typefaces (using realistic synthetic data), as well as in terms of fitting the real data from historical books.

#### 3.1.1 Baselines for comparison

**Ocular:** Based on the emission model of Ocular that uses discrete latent variables for the vertical/horizontal offset and sinking variables, and hence has limited expressivity.

\(\lambda\)**-only:** This model only has the interpretable continuous latent variables pertaining to spatial adjustment.

**VAE-only:** This model is expressive but doesn't have any interpretable latent variables for explicit control. It is an extension of Kingma et al. (2014)'s model for semi-supervised learning with a continuous latent variable vector in which we obtain tighter bounds by marginalizing over the cluster identities explicitly. For fair comparison, the encoder and decoder convolutional architectures are the same as the ones in our full model. The corresponding training objective for this baseline is:

\[\max_{T,\theta,\phi}\sum_{d}\log\sum_{k}\mathrm{E}_{q(z_{d}|X_{ d},k;\phi)}\big{[}\pi_{k}p(X_{d}|z_{d};T_{k},\theta)\big{]}\] \[-\text{KL}\big{(}q(z_{d}|X_{d},k;\phi)||p(z)\big{)}\]

**No-residual:** The only difference from the full model is that the encoder for the inference network conditions the variational distribution \(q(z)\) on the entire input image \(X\) instead of just the residual image \(X-\bar{T}\).

#### 3.1.2 Font discovery in Synthetic Data

Early modern books were frequently composed from two or more type cases, resulting in documents with mixed fonts. We aim to learn the different shapes of metal stamps that were used as templates for each cluster component in our model.

**Data:** In order to quantitatively evaluate our model's performance, we experiment with synthetically generated realistic dataset for which we know the ground truth cluster identities in the following manner: For each character of interest, we pick three distinct images from scanned segmented EEBO images, corresponding to three different metal casts. Then we randomly add spatial perurbations related to scale, offset, rotation and shear. To incorporate varying inking levels and other distortions, we randomly either perform _erosion_, _diolation_, or a combination of these warpings using OpenCV Bradski (2000) with randomly selected kernel sizes. Finally, we add a small Gaussian noise to the pixel intensities and generate 300 perturbed examples per character class.

**Results:** We report macro-averaged results across all the character classes on three different clustering measures, V-measure Rosenberg and Hirschberg (2007), Mutual Information and Fowlkes and Mallows IndexFowlkes and Mallows (1983). In Table 1, we see that our model significantly outperforms all other baselines on every metric. _Ocular_ and \(\lambda\)_-only_ models fail because they lack expressiveness to explain the variations due to random jitters, erosions and dilations. The _VAE-only_ model, while very expressive, performs poorly because it lacks the inductive bias needed for successful clustering. The _No-residual_ model performs decently but our model's superior performance emphasizes the importance of designing a restrictive inference network such that \(z\) only focuses on extraneous sources of variation.

#### 3.1.3 Fitting Real Data from Historical Books

For the analysis of real books, we selected three books from the EEBO dataset printed by different printers. We modeled each character class for each book separately and report the macro-aggregated

\begin{table}
\begin{tabular}{l l l l|c} \hline  & **V-measure** & **Mutual Info** & **F\&M** & **NLL** \\ \hline
**Ocular** & 0.42 & 0.45 & 0.61 & 379.21 \\ \(\lambda\)**-only** & 0.49 & 0.51 & 0.70 & 322.04 \\
**VAE-only** & 0.22 & 0.29 & 0.38 & 263.45 \\
**No-residual** & 0.54 & 0.58 & 0.73 & 264.27 \\
**Our Model** & **0.73** & **0.74** & **0.85** & **257.92** \\ \hline \end{tabular}
\end{table}
Table 1: (a) Clustering results on synthetic data (**V-measure**, **Mutual Info, F&M**). (b) Test negative log likelihood (NLL) on real data from historical documents, or negative ELBO bound for intractable models (**NLL**).

 

upper bounds on the negative log likelihood (NLL) in Table 1. We observe that adding a small amount of expressiveness makes our _\(\lambda\)-only_ model better than _Ocular_. The upper bounds of other inference network based models are much better than the tight1 bounds of both the interpretable models. Our model has the lowest upper bound of all the models while retaining interpretability and control.

Footnote 1: For **Ocular** and \(\lambda\)**-only** models, we report the upper bound obtained via maximization over the interpretable latent variables. Intuitively, these latent variables are likely to have unimodal posterior distributions with low variance, hence this approximation is likely tight.

### Qualitative analysis

We provide visual evidence of desirable behavior of our model on collections of character extractions from historical books with mixed fonts. Specifically, we discus the performance of our model on the mysterious edition of Thomas Hobbes' _Leviathan_ known as "the 25 Ornaments" edition. [7] The 25 Ornaments _Leviathan_ is an interesting test case for several reasons. While its title page indicates a publisher and year of publication, both are fabricated [38]. The identities of its printer(s) remain speculative, and the actual year of publication is uncertain. Further, the 25 Ornaments exhibits two distinct fonts.

#### 3.2.1 Quality of learned templates

Our model is successful in discovering distinctly shaped typefaces in the 25 Ornaments _Leviathan_. We focus on the case study of majuscule letters **F** and **R**, each of which have two different typefaces mixed in throughout. The two typefaces for **F** differ in the _length of the middle arm_ (Fig. 1), and the two typefaces for **R** have _differently shaped legs_. In Fig. 5, we show that our model successfully learns the two desired templates \(T_{1}\) and \(T_{2}\) for both the characters which indicates that the clusters in our model mainly focus on subtle differences in underlying glyph shapes. We also illustrate how the latent variables transform the model templates \(T\) to \(\hat{T}\) for four example **F** images. The model learns complex functions to transform the templates which go beyond simple affine and morphological transformations in order to account for mixing differences, random jitter, contrast variations etc.

#### 3.2.2 Interpretable variables (\(\lambda\)) and Control

Finally, we visualize the ability of our model to separate responsibility of modelling variation among the interpretable and non-interpretable variables appropriately. We use the inferred values of the interpretable (\(\lambda\)) variable for each image in the dataset to adjust the corresponding image. Since the templates represent the canonical shape of the letters, the \(\lambda\) variables which shift the templates to explain the images can be _reverse applied_ to the input images themselves in order to _align_ them by accounting for offset, rotation, shear and minor size variations. In Fig. 6, we see that the input images (top row) are uneven and vary by size and orientation. By reverse applying the inferred \(\lambda\) values, we are able to project the images to a fixed size such that they are aligned and any remaining variations in the data are caused by other sources of variation. Moreover, this _alignment_ method would be crucial for automating certain aspects of bibliographic studies that focus on comparing specific imprints.

## 4 Conclusion

Beyond applications to typeface clustering, the general approach we take might apply more broadly to other clustering problems, and the model we developed might be incorporated into OCR models for historical text.

## 5 Acknowledgements

This project is funded in part by the NSF under grants 1618044 and 1936155, and by the NEH under grant HAA256044-17.

Figure 5: The learned templates for **F** and **R** and the transformed templates \(\hat{T}\) for four examples of **F** are shown. Our model is able to learn desirable templates based on underlying glyph structure.

Figure 6: Result of alignment on _Leviathan_ extractions using the interpretable \(\lambda\) variables along with their pixelwise average images. Aligned average image is much sharper than the unaligned average image.

 



## References

* Bepler et al. (2019) Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. 2019. Explicitly disentangling image content from translation and rotation with spatial-vae. In Advances in Neural Information Processing Systems, pp. 15409-15419. Cited by: SS3.
* Berg-Kirkpatrick et al. (2013) Unsupervised transcription of historical documents. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Sofia, Bulgaria, pp. 207-217. External Links: Link Cited by: SS1.
* Besag (1986) On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society: Series B (Methodological)48 (3), pp. 259-279. Cited by: SS1.
* G. Bradski (2000) The OpenCV Library. Dr. Dobb's Journal of Software Tools. Cited by: SS1.
* E. B. Fowlkes and C. L. Mallows (1983) A method for comparing two hierarchical clusterings. Journal of the American statistical association78 (383), pp. 553-569. Cited by: SS1.
* D. Garrette, H. Alpert-Abrams, T. Berg-Kirkpatrick, and D. Klein (2015)Unsupervised code-switching for multilingual historical document transcription. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, pp. 1036-1041. External Links: ISBN 978-1-450-3213-3 Cited by: SS1.
* K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang (2018) Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics6, pp. 437-450. Cited by: SS1.
* C. Hinman (1963)The printing and proofreading of the first folio of Shakespeare. Vol. 1, Oxford: Clarendon Press.. Cited by: SS1.
* T. Hobbes (1651) [really 1700?]. Leviathan, or, the matter, form, and power of a common-wealth ecological and civil. By Thomas Hobbes of Malmesbury. Number R13935 in ESTC. [false imprint] printed for Andrew Crooke, at the Green Dragon in St. Pauls Church-yard, London. Cited by: SS1.
* D. P. Kingma and M. Welling (2014) Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Cited by: SS1.
* D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling (2014) Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589. Cited by: SS1.
* N. Malcolm (2014) Editorial introduction. In Leviathan, Vol. 1. Clarendon Press, Oxford. Cited by: SS1.
* A. Rosenberg and J. Hirschberg (2007) V-measure: a conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pp. 410-420. Cited by: SS1.
* M. Ryskina, H. Alpert-Abrams, D. Garrette, and T. Berg-Kirkpatrick (2017) Automatic compositort attribution in the first folio of shakespeare. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, pp. 411-416. External Links: Document, Link Cited by: SS1.
* N. Srivastava, J. Barron, D. Klein, and T. Berg-Kirkpatrick (2019) A deep factorization of style and structure in fonts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 2195-2205. External Links: Link Cited by: SS1.
* A. Takano (2016) Thomas Warren: a printer of leviathan (head edition). Annals of Nagoya University Library Studies13, pp. 1-17. Cited by: SS1.
* C. N. Warren, P. Williams, S. Rijhwani, and M. G'Sell (2020) Damaged type and areopasitic's clandestine printers. Milton Studies62, pp. 1. Cited by: SS1.
* A. Weiss (1992) Shared Printing, Printer's Copy, and the Text(s) of gascoigne s"A hundreth Sundrie Flowres". Studies in Bibliography45, pp. 71-104. Cited by: SS1.

## Appendix A Character wise quantitative analysis

The quantitative experiments were performed on the following character classes: A, B, E, F, G, H, M, N, R, W.

\begin{table}
\begin{tabular}{l l l l|l} \hline  & **V-measure** & **Mutual Info** & **F\&M** & **NLL** \\ \hline \(\lambda\)**only** & 0.77 & 0.82 & 0.89 & 264.90 \\
**VAE-only** & 0.33 & 0.38 & 0.5 & 230.45 \\
**No-residual** & 0.79 & 0.85 & 0.90 & 231.45 \\
**Our Model** & **0.78** & **0.86** & **0.89** & **226.25** \\ \hline \end{tabular}
\end{table}
Table 2: Results for character A

\begin{table}
\begin{tabular}{l l l l|l} \hline  & **V-measure** & **Mutual Info** & **F\&M** & **NLL** \\ \hline \(\lambda\)**-only** & 0.37 & 0.39 & 0.59 & 261.1 \\
**VAE-only** & 0.15 & 0.2 & 0.32 & 229.1 \\
**No-residual** & 0.37 & 0.39 & 0.58 & 228.1 \\
**Our Model** & **0.68** & **0.73** & **0.81** & **226.25** \\ \hline \end{tabular}
\end{table}
Table 3: Results for character B



\begin{table}
\begin{tabular}{l c c c|c} \hline  & V-measure & Mutual Info & F\&M & NLL \\ \hline \(\lambda\)**only** & 0.62 & 0.64 & 0.78 & 392.06 \\
**VAE-only** & 0.29 & 0.38 & 0.40 & 323.5 \\
**No-residual** & 0.70 & 0.83 & 0.74 & 329.25 \\
**Our Model** & **0.75** & **0.84** & **0.87** & **323.04** \\ \hline \end{tabular}
\end{table}
Table 8: Results for character M

\begin{table}
\begin{tabular}{l l c c|c} \hline  & V-measure & Mutual Info & F\&M & NLL \\ \hline \(\lambda\)**-only** & 0.33 & 0.36 & 0.55 & 282.4 \\
**VAE-only** & 0.17 & 0.19 & 0.30 & 253.2 \\
**No-residual** & 0.33 & 0.35 & 0.56 & 251.45 \\
**Our Model** & **0.65** & **0.70** & **0.76** & **234.05** \\ \hline \end{tabular}
\end{table}
Table 4: Results for character E

\begin{table}
\begin{tabular}{l l l l|l} \hline  & V-measure & Mutual Info & F\&M & NLL \\ \hline \(\lambda\)**-only** & 0.65 & 0.71 & 0.79 & 418.01 \\
**VAE-only** & 0.31 & 0.45 & 0.42 & 364.2 \\
**No-residual** & 0.72 & 0.78 & 0.82 & 369.5 \\
**Our Model** & **0.72** & **0.79** & **0.84** & **364.21** \\ \hline \end{tabular}
\end{table}
Table 11: Results for character W