Semi-Supervised Learning via Clustering Representation Space

1 INTRODUCTION . Labeling data is expensive . Thus , it is often hard for us to get enough labeled samples . Thus , semi-supervised learning ( Chapelle et al. , 2009 ) becomes a serious issue . People try to get good performance with limited labeled data and a large amount of unlabeled data . When having a limited amount of labeled samples , extracting information from unlabeled data has played an important role for semi-supervised learning . In general , we often applied unlabeled information as auxiliary tools , such as pre-training ( Hinton and Salakhutdinov , 2006 ) or recursive picking confidence data from unlabeled samples with supervised learning ( Zhu , 2005 ) . However , we notice that when we counter the issue such as Two Half-moon , double circle , or other more complex distribution problem , these methods are lack of considering the spatial distribution information provided by unlabeled samples . In this paper , we aimed to guide our model to extract the spatial distribution information from unlabeled data . We proposed a new approach for semi-supervised learning by adding our loss function term for our target embedding latent space . Within our proposed model , the neural network can now learn correctness and spatial distribution information from labeled and unlabeled samples simultaneously . This provides our feed-forward neural network to have more opportunity passing through the sparse margin between clusters , and elevate the performance of the classifier , see more details in Sec . 3 . Moreover , it is worth noting that our proposed model does not rely on any additional neural networks , which is suitable for any task and is highly compatible with different semi-supervised learning algorithms . In short , the characteristics of our proposed model are as follows : Intuitive The idea of correctness and spatial distribution came up with the characteristics of supervised and unsupervised learning straightly , which is intuitive . Compatibility Our method does not rely on any additional neural networks but only adding new loss term . Our approach is easy to change into any existing feed-forward neural networks . Extensible We designed our approach by the notion of defining an evaluation for spatial distribution , which can replace by any other methods in future researches . 2 RELATED WORK . In recent years , the neural network plays an essential role in various tasks ; more specifically , they are highly applied in the tasks of image classification . Since then , semi-supervised learning ( Weston et al. , 2012 ; Lee , 2013 ) for image classification has become a vital issue . First of all , some works succeeded by proposing regularization methods for NNs ( Bishop , 1995 ; Srivastava et al. , 2014 ) . They regularize the input and hidden layers of their models by applying random permutations . This can smooth the input-output relation and further get improvements in semi-supervised learning . Generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) are popular research for the neural network , several models ( Salimans et al. , 2016 ; Dumoulin et al. , 2016 ; Dai et al. , 2017 ) had gone more in-depth researches with GAN for semi-supervised learning . They showed remarkable results , especially on image classification problems . In 2014 , Kingma et al . proposed Deep generative models ( Kingma et al. , 2014 ) , applying a variational-autoencoder based generative model to semi-supervised learning . They showed good results on image classification for semi-supervised learning and became the benchmark of several datasets . However , in practice , these models require more careful tuning for parameters . Also , they usually require more neural network structures and computation resources . Moreover , Rasmus et al . ( Rasmus et al. , 2015 ) propose semi-supervised with Ladder Networks , a model structure with an autoencoder . This work is similar to denoising autoencoders and applied to every layer . It is impressive that they have got a vast improvement compared with Deep Generative models ( Kingma et al. , 2014 ) . Shortly after , Miyato ey al . ( Miyato et al. , 2018 ) also achieve competitive results on the benchmark data sets with a regularization term . They guide their model to minimize the change of the input and output of the network , which does not require labeled information and able to use unlabeled samples for their regularization term . Labeled propagation proposed by ( Zhu and Ghahramani , 2002 ) is also a family of methods for semi-supervised learning . By smoothing the model around the input data points , the model can extrapolate the labels of unlabeled samples . Similar to this idea , several works ( Laine and Aila , 2016 ; Sajjadi et al. , 2016 ) had succeeded by using the random image augmentation . They try to improve the generation performance of image classification for semi-supervised learning . 3 PROPOSED MODEL . For semi-supervised learning , we assume that in some particular space , samples in the same category should be in the same cluster . Following this assumption , once we can distinguish different clusters properly , and guide our network to find the decision boundary with sparser region between clusters . In this section , we introduce an end-to-end learning method by adding our loss functions . First of all , we tried to guide our network to learn a good mapping from the original input space to the embedding latent space ; see Sec . 3.1 . We next defined loss functions and tried to cluster the samples that should be in the same categories together in the embedding latent space ; see Sec . 3.2 . Moreover , similar to some supervised learning works ( Xu et al. , 2005 ; Rennie and Srebro , 2005 ; Srebro et al. , 2005 ) , we aim to maximize the margin between different clusters to separate them well . Note that since we are lack of labeled samples , we maximize the margin within temporary clustering results for our data , instead of using the ground truth of labeled data , see more in Sec . 3.3 . Overall , we named our proposed model as Maximum Cluster Margin Classifier , referred to as MCMC . 3.1 EMBEDDING LATENT SPACE . In general , measuring a clustering result is very subjective . However , to deal with various kinds of distribution , we avoid evaluating them directly by their original input space . Instead , we pull out a layer from the neural network and set it as the embedding latent space . Next , we evaluate whether the quality of the embedding latent space . In our proposed model , we try to define a measurement for the embedding latent space to satisfy our assumptions . This guides the previous layers to learn about a good mapping from the original input space to a well-distributed embedding latent space . To strengthen the efficiency of the embedding latent space , we add a simple classifier that is fully connected to the embedding latent space . 3.2 DB LOSS . 3.2.1 DAVIES-BOULDIN INDEX . As Davies-Bouldin index ( Davies and Bouldin , 1979 ) proposed , given a dataset with N clusters , for every cluster Ci , we compute Si as a measure of scattering with the cluster , which is defined as follows : Si = ( 1 Ti Ti∑ j=1 |Xji −Ai| p ) 1 p ( 1 ) where Xji is the jth sample of cluster Ci , Ai is the centroid of cluster Ci , Ti is the size of Ci and p is usually set as 2 . To measure the distance between different centroid of clusters Ci and Cj , we compute Mi , j , which is defined as follows : Mi , j = ||Ai −Aj ||p = ( n∑ k=1 |ak , i − ak , j |p ) 1 p ( 2 ) where ak , i is the ith element of Ai , and p is usually set as 2 . Finally , we combine Si with Mi , j , which is defined as follows : Ri , j = Si + Sj Mi , j , Di ≡ max i 6=j Ri , j , DB ≡ 1 N N∑ i=1 Di . ( 3 ) 3.2.2 DB LOSS . Thanks to Davies-Bouldin index Sec . 3.2.1 ( Davies and Bouldin , 1979 ) , we now have a measurement for evaluating the clustering result in embedding latent space . Eq . ( 1 ) provides us the evaluation of whether the samples in a single cluster are close enough to the centroid . Simultaneously , Eq . ( 2 ) separates the centroid of different clusters to become as far as possible . Moreover , Eq . ( 3 ) restricts every single cluster could gather together while different cluster has a long distance between their centroid . We defined our DB loss LDB as follows : Given points X with temporary predictions ŷ , then LDB ≡ DB ( 4 ) where DB can be computed in Sec . 3.2.1 . Till now , DB loss LDB provides us the capability to guide the samples in the same category gather in the same cluster . However , when we look into Davies-Bouldin index ( Davies and Bouldin , 1979 ) , the measurement considers only about the centroid of each cluster . This causes the issue that different clusters may be overlap with each other . That is , the margin between different clusters has vanished , see Fig.3 . For this reason , we came up with Maximum Margin loss , see Sec . 3.3 . 3.3 MAXIMUM MARGIN LOSS . As we mentioned at the start of Sec . 3 , we need to maximize the margin between different clusters to separate them . We hence came up with Maximum Margin Loss LMM . To push each cluster as far as we can , we compute the sum of k pairs single link ( Gower and Ross , 1969 ) between different clusters . When maximizing those pairs of single link , it also implies maximizing the margin between different clusters . We define the Maximum Margin loss LMM as follows : Given clusters C1 , C2 , ... , CN , we let V ` i , j be the ` th pair single link between Ci and Cj . Also , we define ∑k ` =1 1 V ` i , j to be k pairs single link distancei , j . Finally , we have LMM = N∑ i 6=j , i < j ( k pairs single link distancei , j ) , ( 5 ) where k is the number of pairs between different clusters , and N is the number of the clusters . Maximum Margin loss LMM provides the information of k-pair single link distance between different clusters . This loss can guide the model to ameliorate the overlap problem and maximize the margin between different clusters . Fig . 1 ( b ) shows the main idea of Maximum Margin loss LMM , which helps our network avoid the problem illustrated in Fig . 1 ( a ) . Overall , by combining DB loss LDB and Maximum Margin loss LMM , we can find our ideal evaluation for clustering results in embedding latent space . Our network is now able to gather the samples that should be in the same categories and separate different clusters as far as possible . 3.4 TOTAL LOSS . In this section , we took a more in-depth exploration of how we deal with labeled data and unlabeled data for our total loss separately . For labeled data , we applied loss with cross-entropy ( CE ) ( De Boer et al. , 2005 ) , which is widely used for classification tasks in supervised learning . Note that Crossentropy ( CE ) ( De Boer et al. , 2005 ) is available to change into other losses in supervised learning for personal use . Though labeled data might have a small number of samples , they still play an essential role in guiding the network learning the right predictions . Moreover , they can also provide the neural network to learn a rough decision boundary in the early iterations . This leads our network to get a good start and modify the boundary into better results . As we mentioned in Sec . 3.2.2 and Sec . 3.3 , we need DB loss LDB to guide the samples in the same cluster to group together . At the same time , Maximum Margin loss LMM pushes the different clusters as far as possible , i.e. , maximize the margin between different clusters . We define the total loss LTotal as follows : LTotal = LDB + LCE + LMM . ( 6 )

Summary: This paper attempts to address the semi-supervised learning topic by proposing a method based on an aggregated loss considering both cross-entry and Davies-Bouldin Index. Cross-entropy is used to ensure the maximum margin between classes and Davies-Bouldin Index is applied to the labeled data and to the whole dataset, respectively, to ensure a high quality of clustering. Evaluations in four small and simple datasets are reported to demonstrate the effectiveness of the proposed method.