Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning

1 INTRODUCTION . A key factor for progress in machine learning has been the availability of well curated , easy-to-use , standardised and sufficiently large annotated datasets for benchmarking different algorithms and models . This has led to major advances in speech recognition , computer vision , and natural language processing . A commonality between these tasks is their natural formulation as supervised learning tasks , wherein performance can be measured in terms of accuracy on a test set . The general problem of representation learning ( i.e . to reveal latent structure in data ) is more difficult to assess due the lack of suitable benchmarks . Although the field is very active , with many recently proposed techniques such as probabilistic autoencoders and adversarial learning , it is less clear where the field stands in terms of progress or which approaches are more expressive for specific tasks . The lack of reproducible ways to quantify performance has led to subjective means of evaluation : visualisation techniques have been used to show low-dimensional projections of the latent space and visual inspection of generated or reconstructed samples are popular to provide subjective measures of descriptiveness . On the other hand , the quality of sampled images generally tells us little about how well the learned representations capture known factors of variation in the training distribution . In order to advance progress , the availability of tools for objective assessment of representation learning methods seems essential yet lacking . This paper introduces Morpho-MNIST , a collection of shape metrics and perturbations , in a step towards quantitative assessment of representation learning . We build upon one of the most popular machine learning benchmarks , MNIST , which despite its shortcomings remains widely used . While MNIST was originally constructed to facilitate research in image classification , in the form of recognising handwritten digits ( LeCun et al. , 1998 ) , it has found its use in representation learning , for example , to demonstrate that the learned latent space yields clusters consistent with digit labels . Methods aiming to disentangle the latent space claim success if individual latent variables capture specific style variations ( e.g . stroke thickness , sidewards leaning digits and other visual characteristics ) . The main appeal of selecting MNIST as a benchmark for representation learning is that , while manifesting complex interactions between pixel intensities and underlying shapes , it has well understood and easily measurable factors of variation . More generally , MNIST remains popular in practice due to several factors : it allows reproducible comparisons with previous results reported in the literature ; the dataset is sufficiently large for its complexity and consists of small , two-dimensional greyscale images defining a tractable ten-class classification problem ; computation and memory requirements are low ; most popular deep learning frameworks and libraries offer tutorials using MNIST , which makes it straightforward for new researchers to enter the field and to experiment with new ideas and explore latest developments . We take advantage of these qualities and extend MNIST in multiple ways , as summarised in the following . 1.1 CONTRIBUTIONS . Our aim is to bridge the gap between methodology-focused research and critical real-world applications that could benefit from latest machine learning methods . As we preserve the general properties of MNIST—such as image size , file format , numbers of training and test images , and the original ten-class classification problem—we believe this new quantitative framework for assessing representation learning will experience widespread use in the community and may inspire further extensions facilitated by a publicly available Morpho-MNIST code base . Morphometrics : We propose to describe true and generated digit images in terms of measurable shape attributes . These include stroke thickness and length , and the width , height , and slant of digits ( cf . Fig . 1 , left ) . Whereas some of these properties have been analysed qualitatively in previous work , we demonstrate that objectively quantifying each of them allows to identify the role of inferred representations . Moreover , these tools can be used to measure model samples , enabling assessment of generative performance with respect to sample diversity ( Section 4.1 ) and disentanglement of latent variables ( Section 4.2 ) . These measurements can be directly employed to re-evaluate existing models and may be added retrospectively to previous experiments involving the original MNIST dataset . Adoption of our morphometric analysis may provide new insights into the effectiveness of representation learning methods in terms of revealing meaningful latent structures . Furthermore , for other datasets it suffices to design the relevant scalar metrics and include them in the very same evaluation framework . Perturbations : We introduce a set of parametrisable global and local perturbations , inspired by natural and pathological variability in medical images . Global changes involve overall thinning and thickening of digits , while local changes include both swelling and fractures ( see examples on the right in Fig . 1 and many more in Appendix B ) . Injecting these perturbations into the dataset adds a new type of complexity to the data manifold and opens up a variety of interesting applications . The proposed perturbations are designed to enable a wide range of new studies and applications for both supervised and unsupervised tasks . Detection of ‘ abnormalities ’ ( i.e . local perturbations ) is an evident application , although more challenging tasks can also be defined , such as classification from noisy/corrupted data , domain adaptation , localisation of perturbations , characterising semantics of learned latent representations , and more . We explore a few supplementary examples of supervised tasks in Appendix D . 1.2 RELATED WORK : DATASETS . In this section , we provide an overview of some datasets that are related to MNIST , by either sharing its original source content , containing transformations of the original MNIST images or being distributed in the same format for easy replacement . We also mention a few prevalent datasets of images with generative factor annotations , similarly to the morphometrics proposed in this paper . NIST datasets : The MNIST ( modified NIST ) dataset ( LeCun et al. , 1998 ) was constructed from handwritten digits in NIST Special Databases 1 and 3 , now released as Special Database 19 ( Grother and Hanaoka , 2016 ) . Cohen et al . ( 2017 ) generated a much larger dataset based on the same NIST database , containing additional upper- and lower-case letters , called EMNIST ( extended MNIST ) . MNIST perturbations : The seminal paper by LeCun et al . ( 1998 ) employed data augmentation using planar affine transformations including translation , scaling , squeezing , and shearing . Loosli et al . ( 2007 ) employed random elastic deformations to construct the Infinite MNIST dataset . Other MNIST variations include rotations and insertion of random and structured background ( Larochelle et al. , 2007 ) , and Tieleman ( 2013 ) applied spatial affine transformations and provided ground-truth transformation parameters . MNIST format : Due to the ubiquity of the MNIST dataset in machine learning research and the resulting multitude of compatible model architectures available , it is appealing to release new datasets in the same format ( 28×28 , 8-bit grayscale images ) . One such effort is Fashion-MNIST ( Xiao et al. , 2017 ) , containing images of clothing articles from ten distinct classes , adapted from an online shopping catalogue . Another example is notMNIST ( Bulatov , 2011 ) , a dataset of character glyphs for letters ‘ A ’ – ‘ J ’ ( also ten classes ) , in a challengingly diverse collection of typefaces . Annotated datasets : Computer vision datasets that are popular for evaluating disentanglement of learned latent factors of variation include those from Paysan et al . ( 2009 ) and Aubry et al . ( 2014 ) . They contain 2D renderings of 3D faces and chairs , respectively , with ground-truth pose parameters ( azimuth , elevation ) and lighting conditions ( faces only ) . A further initiative in that direction is the dSprites dataset ( Matthey et al. , 2017 ) , which consists of binary images containing three types of shapes with varying location , orientation and size . The availability of the ground-truth values of such attributes has motivated the accelerated adoption of these datasets in the evaluation of representation learning algorithms . 1.3 RELATED WORK : QUANTITATIVE EVALUATION . Evaluation of representation learning is a notoriously challenging task and remains an open research problem . Numerous solutions have been proposed , with many of the earlier ones focusing on the test log-likelihood under the model ( Kingma and Welling , 2013 ) or , for likelihood-free models , under a kernel density estimate ( KDE ) of generated samples ( Goodfellow et al. , 2014 ; Makhzani et al. , 2015 ) —being shown not to be reliable proxies for the true model likelihood ( Theis et al. , 2016 ) . Another perspective for evaluation of generative models of images is the visual fidelity of its samples to the training data , which would normally require manual inspection . To address this issue , a successful family of metrics have been proposed , based on visual features extracted by the Inception network ( Szegedy et al. , 2016 ) . The original Inception score ( Salimans et al. , 2016 ) relies on the ‘ crispness ’ of class predictions , whereas the Fréchet Inception distance ( FID ) ( Heusel et al. , 2017 ) and the kernel Inception distance ( KID ) ( Bińkowski et al. , 2018 ) statistically compare high-level representations instead of the final network outputs . Although the approaches above can reveal vague signs of mode collapse , it may be useful to diagnose this phenomenon on its own . With this objective , Arora et al . ( 2018 ) proposed to estimate the support of the learned distribution ( assumed discrete ) using the birthday paradox test , by counting pairs of visual duplicates among model samples . Unfortunately , the adoption of this technique is hindered by its reliance on manual visual inspection to flag identical images . There have been several attempts at quantifying representation disentanglement performance . For example , Higgins et al . ( 2017 ) proposed to use the accuracy of a simple classifier trained to predict which factor of variation was held fixed in a simulated dataset . There exist further informationtheoretic approaches , involving the KL divergence contribution from each latent dimension ( Dupont , 2018 ) or their mutual information with each known generative factor ( Chen et al. , 2018 ) . Yet another method , explored in Kumar et al . ( 2018 ) , is based on the predictive accuracy of each latent variable to each generative factor ( continuous or discrete ) . 2 MORPHOMETRY . Meaningful morphometrics are instrumental in characterising distributions of rasterised shapes , such as MNIST digits , and can be useful as additional data for downstream learning tasks . We begin this section by describing the image processing pipeline employed for extracting the metrics and for applying perturbations ( Section 3 ) , followed by details on the computation of each measurement . 2.1 PROCESSING PIPELINE . The original 28×28 resolution of the MNIST images is generally not high enough to enable satisfactory morphological processing : stroke properties ( e.g . length , thickness ) measured directly on the binarised images would likely be inaccurate and heavily quantised . To mitigate this issue and enable sub-pixel accuracy in the measurements , we propose to use the following processing steps : 1. upscale ( e.g . ×4 , to 112×112 ) 1 ; 2. binarise ( e.g . threshold ≥128 ) ; 3. compute Euclidean distance transform ( EDT ) from boundaries ; 4. skeletonise ( medial axis , i.e . ridges of EDT ) ; 5. apply perturbation ( cf . Section 3 ) ; and 6. downscale to original resolution . We illustrate the pipeline in Fig . 2 . The binary high-resolution digits have smooth boundaries and faithfully capture subtle variations in contour shape and stroke thickness that are only vaguely discernible in the low-resolution images . Additionally , note how the final downscaled image is almost indistinguishable from the original . All morphometric attributes described below are calculated for each digit after applying steps 1–4 of this pipeline . The distributions for the plain MNIST training set is plotted in Fig . 3 , and the distributions after applying each type of perturbation can be found in Appendix A . 1Up- and downscaling by a factor of f are done with bicubic interpolation and Gaussian smoothing ( bandwidth σ = 2f/6 ) , following scikit-image defaults ( van der Walt et al. , 2014 ) .

Summary: Authors present a set of criteria to categorize MNISt digists (e.g. slant, stroke length, ..) and a set of interesting perturbations (swelling, fractures, ...) to modify MNIST dataset. They suggest analysing performance of generative models based on these tools. By extracting this kind of features, they effectively decrease the dimmension of  data. Therefore, statistically comparing the distribution of generated vs test data and binning the generated data is now possible. They perform a thorough study regarding MNIST. Their tools are a handy addition to the analytical surveys in several applications (e.g. how classification fails), but not convincingly for generation.