Unsupervised classification into unknown number of classes

1 INTRODUCTION . Image classification has been one of the most important research in machine learning and artificial intelligence . For the predefined number k of classes , the classification models based on deep learning have performed well that the test accuracy is better than humans ( Geirhos et al. , 2017 ; He et al. , 2016 ; Russakovsky et al. , 2015 ; Szegedy et al. , 2015 ) . This result is , however , true only when the number of classes , k is predefined and we use the supervised learning with entire ground truth labels . Humans learn visual information by memorizing the compressed information ( features ) in the brain , reconstruct the object , and recognize the difference between objects . This capability allows humans to classify data where the number of classes is unknown into the reasonable numbers of classes . In this paper , we propose a method to mimic such human capability . The main contribution of the proposed framework consists of two parts . We first show that there exists a approximate linear map between two separated neural networks of Generative Adversarial Networks ( GANs ) . Secondly , we prove that the number of classes in a latent space is the same as the number of classes in the original space in terms of the multiplicity of eigenvalue 0 of a Laplacian matrix in spectral clustering theory . By combining the results , We derive a novel unsupervised classification framework , which estimates the number of classes k and then extract k vectors representing each class . With the framework , the approximate linear map between two networks can induce the information of connected components in a data graph . The latent variable model is commonly used in the generative models . Variational auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ) and GANs ( Goodfellow et al. , 2014 ) are a few of the most famous cases that learn the data distribution successfully . GANs randomly samples the latent variables that are used to train the data distribution so we can not figure out which latent variable generates a result that we are interested in . To avoid the drawback by random sampling , auxiliary conditions on latent variables were introduced ( Chen et al. , 2016 ; Mirza & Osindero , 2014 ) . Introducing a label information into the condition of the estimated data probability is highly effective in classification where a data point x came from , however this method is not a unsupervised learning . Spectral clustering is one of the most popular clustering algorithm that outperforms other wellknown clustering algorithms ( Ng et al. , 2002 ; Shi & Malik , 2000 ) . Despite high performance , spectral clustering is difficult to use because it requires the full dataset that have very largeN data points . In this case , the dimension of the Laplacian matrix is N ×N and it takes at least O ( N3 ) . Therefore , we need a novel dimension reduction technique to use spectral clustering in practical applications ( Elhamifar & Vidal , 2009 ; Ji et al. , 2017 ; Law et al. , 2017 ; Shaham et al. , 2018 ) . Recent image classification methods are based on supervised learning , in which a learning model processes a dataset consisting of pairs of an image and its ground-truth label . This training dataset usually consists of more than 10 K data point , some part of which need to be used for crossvalidation . It also takes a lot of costs to produce a reliable training dataset . In addition , estimating the number of classes is a key aspect of unsupervised learning . There have been a few heuristic method ( Zelnik-Manor & Perona , 2005 ; Von Luxburg , 2007 ) . We propose a method to find the number of classes analytically . The main concepts of spectral clustering , GANs and the self-expressiveness property will be introduced Section 2 . We prove the preliminary propositions that allow us to classify data with unsupervised learning methods in Section 3 . Finally , we propose a unsupervised classification algorithm and test our algorithm in Section 4 . 2 PRELIMINARIES . We first review cycle-consistent adversarial networks and sparse subspace clustering that we will use when extracting indicator vectors . We also overview the basic probabilistic property between a classification model and a generative model , which can be used as tools to generate indicator vectors . 2.1 CYCLE-CONSISTENT ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs ) is a generative model that a generator network learns a data distribution implicitly ( Goodfellow et al. , 2014 ) . GANs is a two player minimax game with two neural networks : a discriminator D and a generator G. The discriminator D ( x ) is a probability measure that an input x follows the data distribution pdata ( x ) rather than the generator ’ s distribution pg . The generator G is a mapping from the latent space z ∼ pz to the data space . This minimax game can be represented as a stochastic optimization problem with value function given by min G max D V ( D , G ) = Ex∼pdata ( x ) [ logD ( x ) ] + Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] . ( 1 ) The cycle-consistent adversarial networks ( cycleGANs ) is a variant of GANs , which learns how to translate an representation from a source domain S to a target domain T without a paired dataset ( Zhu et al. , 2017 ) . The cycle-consistency states that two mappings G : S → T and F : T → S encode the right permutation of images in S and T , so that , for an arbitrary data point s ∈ S , the mapping from S to T and back again , F ( G ( s ) ) , should be the same as s. This property encourages adversarial networks G and D to have a connection between generated images and input data . We adopt two cycle consistency loss for the forward map and the backward map to learn a connection between two high-level representation domain in G and D . 2.2 SPARSE SUBSPACE CLUSTERING . Sparse subspace clustering is a clustering algorithm that is based on spectral clustering . Spectral clustering has many variants depending on the similarity measure , Gaussian kernel similarity function ( Ng et al. , 2002 ) , k-nearest neighbor graphs , -neighborhood graph . A similarity measure defined on the similarity graph G = ( V , E ) is a pairwise metric between two data points ( nodes ) in a dataset and the similarity wij of two data points xi , xj is equal to or greater than 0 . The similarity wij is also same as wji because the above similarity graph is defined as an undirected graph . The similarity matrix W is composed of the elements which correspond to the similarity measure wij respectively and then W is symmetric . See more details about spectral clustering in ( Von Luxburg , 2007 ) . Sparse subspace clustering ( Elhamifar & Vidal , 2009 ) uses a novel similarity matrix that is based on the self-expressiveness property . When we assume that the data space is a union of linear subspaces S = ∪kl=1Sl , we can express an arbitrary data point in the dataset as a linear combination of other data points in the dataset , where the coefficients of the data points in the same subspace are non-zero and others should be 0 ( Elhamifar & Vidal , 2009 ; Ji et al. , 2017 ) . Formally , each data point yi ∈ S can be reconstructed as yi = ∑N j=1 cijyj = Yci , where cii = 0 , N is the size of the dataset , ci : = [ ci1 ci2 · · · ciN ] T is the coefficient vector of yi and Y = [ y1 y2 · · · yN ] is the dictionary matrix . The sparse solution of ci indicates that the node i is in the same subspace as the node j if the element cij is non-zero . In this paper , we expand the notion of the self-expressiveness property to reduce the dimensionality of L in Section 3.3 . 3 GRAPH LAPLACIAN OF THE CONNECTOR NETWORK . We first introduce our proposed architecture of adversarial networks . As an autoencoder point of view , we can consider two adversarial networks as autoencoder . We build a connector network between two networks to improve cycle consistency , and extract the information of connected components ( as shown in Figure 1-b ) . By training with the cycle-consistency loss , we can make the tuples ( Z , X , H ) for each data point X in a given dataset . Then , we show the relation between the spectral clustering and unsupervised classification . In order to show the relationship , we prove the dimension reduction of the mutual expressiveness property in Section 3.3 . 3.1 DISENTANGLING UNFOLDS AND EXPANDING THE VOLUME OF MANIFOLD . Deep neural networks learn a high-level abstract representation of a data manifold and a well trained abstraction can disentangle the underlying factors of variation ( Bengio et al. , 2007 ) . These properties allow interpolation between data samples more naturally and smoothly . ( Bengio et al. , 2013 ) studied the disentangling effect and proposed three hypotheses on the shape of manifolds in different representation domains . These hypotheses are verified by several empirical results ( Goodfellow et al. , 2009 ; Glorot et al. , 2011 ) . The three hypotheses are summerized as follows . • The deeper representations can better disentangle the underlying factors of variation . • Disentangled representations unfold the manifolds near which raw data concentrates ir- regularly , and expand the relative volume occupied by high-probability points near these manifolds . Then these representations are with greater convexity . • The underlying class factors of variations will be better disentangled than other factors , so the deeper layer has better discriminant capability of classes . In our proposed architecture , there are two high-level ( deeper ) representation spaces which are called as feature spaces . These feature spaces fill the space more uniformly than the pixel space X . H and Z also have smoother density . Therefore , H and Z have greater convexity than X and it permits linear combination on these two manifolds in the feature spaces H and Z , are the two manifolds are more convex than the manifold in the pixel space X . By manifold hypothesis , different class manifolds are well-separated by regions of very low density ( Cayton , 2005 ) . The feature spaces are high-level representation spaces so that the density is unfolded and a linear combination of features generates amore natural result in the pixel space Bengio ( 2014 ) . Now , we can define the effective basis of the subspace for each class . In the feature spaces , each class manifold is assumed to have the enough convexity . The class manifolds are well-separated , and show good discriminant capability . Using both properties , we can build the effective basis for each class subspaces and generate the original space by a direct sum of these subspaces , where the subspaces are orthogonal . We empirically tested an effective basis as shown in figure 3 . Randomly sampled features in a subspace of the data space can be an effective basis , and are possible to generate all data points in the same subspace . The features in a different subspace can not generate the given data samples . We can estimate the effective dimension with the number of the effective basis . We then can generate orthogonal subspaces of the entire classes if the effective dimension is small enough than the dimension of feature spaces .

Summary: This paper performs unsupervised classification where the number of classes is unknown. The main idea is to use the CycleGAN framework such that one can reconstruct the original image by first moving to latent space that represents another class (via the connector network), then moving back to the original latent space and going back into image space using a generator. Experiments are conducted on MNIST and CIFAR.