Semi-Supervised Learning via Clustering Representation Space

1 INTRODUCTION . Labeling data is expensive . Thus , it is often hard for us to get enough labeled samples . Thus , semi-supervised learning ( Chapelle et al. , 2009 ) becomes a serious issue . People try to get good performance with limited labeled data and a large amount of unlabeled data . When having a limited amount of labeled samples , extracting information from unlabeled data has played an important role for semi-supervised learning . In general , we often applied unlabeled information as auxiliary tools , such as pre-training ( Hinton and Salakhutdinov , 2006 ) or recursive picking confidence data from unlabeled samples with supervised learning ( Zhu , 2005 ) . However , we notice that when we counter the issue such as Two Half-moon , double circle , or other more complex distribution problem , these methods are lack of considering the spatial distribution information provided by unlabeled samples . In this paper , we aimed to guide our model to extract the spatial distribution information from unlabeled data . We proposed a new approach for semi-supervised learning by adding our loss function term for our target embedding latent space . Within our proposed model , the neural network can now learn correctness and spatial distribution information from labeled and unlabeled samples simultaneously . This provides our feed-forward neural network to have more opportunity passing through the sparse margin between clusters , and elevate the performance of the classifier , see more details in Sec . 3 . Moreover , it is worth noting that our proposed model does not rely on any additional neural networks , which is suitable for any task and is highly compatible with different semi-supervised learning algorithms . In short , the characteristics of our proposed model are as follows : Intuitive The idea of correctness and spatial distribution came up with the characteristics of supervised and unsupervised learning straightly , which is intuitive . Compatibility Our method does not rely on any additional neural networks but only adding new loss term . Our approach is easy to change into any existing feed-forward neural networks . Extensible We designed our approach by the notion of defining an evaluation for spatial distribution , which can replace by any other methods in future researches . 2 RELATED WORK . In recent years , the neural network plays an essential role in various tasks ; more specifically , they are highly applied in the tasks of image classification . Since then , semi-supervised learning ( Weston et al. , 2012 ; Lee , 2013 ) for image classification has become a vital issue . First of all , some works succeeded by proposing regularization methods for NNs ( Bishop , 1995 ; Srivastava et al. , 2014 ) . They regularize the input and hidden layers of their models by applying random permutations . This can smooth the input-output relation and further get improvements in semi-supervised learning . Generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) are popular research for the neural network , several models ( Salimans et al. , 2016 ; Dumoulin et al. , 2016 ; Dai et al. , 2017 ) had gone more in-depth researches with GAN for semi-supervised learning . They showed remarkable results , especially on image classification problems . In 2014 , Kingma et al . proposed Deep generative models ( Kingma et al. , 2014 ) , applying a variational-autoencoder based generative model to semi-supervised learning . They showed good results on image classification for semi-supervised learning and became the benchmark of several datasets . However , in practice , these models require more careful tuning for parameters . Also , they usually require more neural network structures and computation resources . Moreover , Rasmus et al . ( Rasmus et al. , 2015 ) propose semi-supervised with Ladder Networks , a model structure with an autoencoder . This work is similar to denoising autoencoders and applied to every layer . It is impressive that they have got a vast improvement compared with Deep Generative models ( Kingma et al. , 2014 ) . Shortly after , Miyato ey al . ( Miyato et al. , 2018 ) also achieve competitive results on the benchmark data sets with a regularization term . They guide their model to minimize the change of the input and output of the network , which does not require labeled information and able to use unlabeled samples for their regularization term . Labeled propagation proposed by ( Zhu and Ghahramani , 2002 ) is also a family of methods for semi-supervised learning . By smoothing the model around the input data points , the model can extrapolate the labels of unlabeled samples . Similar to this idea , several works ( Laine and Aila , 2016 ; Sajjadi et al. , 2016 ) had succeeded by using the random image augmentation . They try to improve the generation performance of image classification for semi-supervised learning . 3 PROPOSED MODEL . For semi-supervised learning , we assume that in some particular space , samples in the same category should be in the same cluster . Following this assumption , once we can distinguish different clusters properly , and guide our network to find the decision boundary with sparser region between clusters . In this section , we introduce an end-to-end learning method by adding our loss functions . First of all , we tried to guide our network to learn a good mapping from the original input space to the embedding latent space ; see Sec . 3.1 . We next defined loss functions and tried to cluster the samples that should be in the same categories together in the embedding latent space ; see Sec . 3.2 . Moreover , similar to some supervised learning works ( Xu et al. , 2005 ; Rennie and Srebro , 2005 ; Srebro et al. , 2005 ) , we aim to maximize the margin between different clusters to separate them well . Note that since we are lack of labeled samples , we maximize the margin within temporary clustering results for our data , instead of using the ground truth of labeled data , see more in Sec . 3.3 . Overall , we named our proposed model as Maximum Cluster Margin Classifier , referred to as MCMC . 3.1 EMBEDDING LATENT SPACE . In general , measuring a clustering result is very subjective . However , to deal with various kinds of distribution , we avoid evaluating them directly by their original input space . Instead , we pull out a layer from the neural network and set it as the embedding latent space . Next , we evaluate whether the quality of the embedding latent space . In our proposed model , we try to define a measurement for the embedding latent space to satisfy our assumptions . This guides the previous layers to learn about a good mapping from the original input space to a well-distributed embedding latent space . To strengthen the efficiency of the embedding latent space , we add a simple classifier that is fully connected to the embedding latent space . 3.2 DB LOSS . 3.2.1 DAVIES-BOULDIN INDEX . As Davies-Bouldin index ( Davies and Bouldin , 1979 ) proposed , given a dataset with N clusters , for every cluster Ci , we compute Si as a measure of scattering with the cluster , which is defined as follows : Si = ( 1 Ti Ti∑ j=1 |Xji −Ai| p ) 1 p ( 1 ) where Xji is the jth sample of cluster Ci , Ai is the centroid of cluster Ci , Ti is the size of Ci and p is usually set as 2 . To measure the distance between different centroid of clusters Ci and Cj , we compute Mi , j , which is defined as follows : Mi , j = ||Ai −Aj ||p = ( n∑ k=1 |ak , i − ak , j |p ) 1 p ( 2 ) where ak , i is the ith element of Ai , and p is usually set as 2 . Finally , we combine Si with Mi , j , which is defined as follows : Ri , j = Si + Sj Mi , j , Di ≡ max i 6=j Ri , j , DB ≡ 1 N N∑ i=1 Di . ( 3 ) 3.2.2 DB LOSS . Thanks to Davies-Bouldin index Sec . 3.2.1 ( Davies and Bouldin , 1979 ) , we now have a measurement for evaluating the clustering result in embedding latent space . Eq . ( 1 ) provides us the evaluation of whether the samples in a single cluster are close enough to the centroid . Simultaneously , Eq . ( 2 ) separates the centroid of different clusters to become as far as possible . Moreover , Eq . ( 3 ) restricts every single cluster could gather together while different cluster has a long distance between their centroid . We defined our DB loss LDB as follows : Given points X with temporary predictions ŷ , then LDB ≡ DB ( 4 ) where DB can be computed in Sec . 3.2.1 . Till now , DB loss LDB provides us the capability to guide the samples in the same category gather in the same cluster . However , when we look into Davies-Bouldin index ( Davies and Bouldin , 1979 ) , the measurement considers only about the centroid of each cluster . This causes the issue that different clusters may be overlap with each other . That is , the margin between different clusters has vanished , see Fig.3 . For this reason , we came up with Maximum Margin loss , see Sec . 3.3 . 3.3 MAXIMUM MARGIN LOSS . As we mentioned at the start of Sec . 3 , we need to maximize the margin between different clusters to separate them . We hence came up with Maximum Margin Loss LMM . To push each cluster as far as we can , we compute the sum of k pairs single link ( Gower and Ross , 1969 ) between different clusters . When maximizing those pairs of single link , it also implies maximizing the margin between different clusters . We define the Maximum Margin loss LMM as follows : Given clusters C1 , C2 , ... , CN , we let V ` i , j be the ` th pair single link between Ci and Cj . Also , we define ∑k ` =1 1 V ` i , j to be k pairs single link distancei , j . Finally , we have LMM = N∑ i 6=j , i < j ( k pairs single link distancei , j ) , ( 5 ) where k is the number of pairs between different clusters , and N is the number of the clusters . Maximum Margin loss LMM provides the information of k-pair single link distance between different clusters . This loss can guide the model to ameliorate the overlap problem and maximize the margin between different clusters . Fig . 1 ( b ) shows the main idea of Maximum Margin loss LMM , which helps our network avoid the problem illustrated in Fig . 1 ( a ) . Overall , by combining DB loss LDB and Maximum Margin loss LMM , we can find our ideal evaluation for clustering results in embedding latent space . Our network is now able to gather the samples that should be in the same categories and separate different clusters as far as possible . 3.4 TOTAL LOSS . In this section , we took a more in-depth exploration of how we deal with labeled data and unlabeled data for our total loss separately . For labeled data , we applied loss with cross-entropy ( CE ) ( De Boer et al. , 2005 ) , which is widely used for classification tasks in supervised learning . Note that Crossentropy ( CE ) ( De Boer et al. , 2005 ) is available to change into other losses in supervised learning for personal use . Though labeled data might have a small number of samples , they still play an essential role in guiding the network learning the right predictions . Moreover , they can also provide the neural network to learn a rough decision boundary in the early iterations . This leads our network to get a good start and modify the boundary into better results . As we mentioned in Sec . 3.2.2 and Sec . 3.3 , we need DB loss LDB to guide the samples in the same cluster to group together . At the same time , Maximum Margin loss LMM pushes the different clusters as far as possible , i.e. , maximize the margin between different clusters . We define the total loss LTotal as follows : LTotal = LDB + LCE + LMM . ( 6 )

Summary: The authors propose a novel loss function for semi-supervised learning. Arguing that SOTA semi-supervised learning methods neglect spatial information (latent clustering structure) in the data, the authors propose a loss function which combines clustering objectives with classification objectives. The proposed loss function combines the cross-entropy loss, the within-cluster scatter as known from k-means, the distance between centroids and the margin between classes. The proposed loss is notably non-continuous since it makes use of the maximum function. The authors employ  ADAM with a learning rate of 0.001 with exponential decay to optimize the novel loss function. Experiments on MNIST and three comparably small datasets show that the proposed method is able to achieve high accuracy with only few labeled data points.