Semi-Supervised Learning via Clustering Representation Space
Authors: Yen-Chieh Huang
Yuh-Jye Lee
Chih-Chi Wu
Yi-Wei Chiu
Yong-Xiang Lin
CHENG-YING LI
Po-Hung Ko
Venue:  (2021)
Keywords: semi-supervised learning
deep learning
clustering
embedding latent space

Abstract: We proposed a novel loss function that combines supervised learning with clustering in deep neural networks. Taking advantage of the data distribution and the existence of some labeled data, we construct a meaningful latent space.  Our loss function consists of three parts, the quality of the clustering result, the margin between clusters, and the classification error of labeled instances. Our proposed model is trained to minimize our loss function by backpropagation, avoiding the need for pre-training or additional networks. This guides our network to classify labeled samples correctly while able to find good clusters simultaneously. We applied our proposed method on MNIST, USPS, ETH-80, and COIL-100; the comparison results confirm our model's outstanding performance over semi-supervised learning.

Under review as a conference paper at ICLR 2021
SEMI-SUPERVISED LEARNING VIA CLUSTERING REP- RESENTATION SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
We proposed a novel loss function that combines supervised learning with clus- tering deep neural networks. Taking advantage the data distribution and the existence of some labeled data, we construct meaningful latent space. Our loss function consists of three parts, the quality of the clustering result, the margin between clusters, and the classification error of labeled instances. Our proposed model is trained to minimize our loss function, avoiding the need for pre-training or additional networks. This guides network to classify labeled samples correctly while able to find good clusters simultaneously. We applied our proposed method MNIST, USPS, ETH-80, and COIL-100; the comparison results confirm our model's outstanding performance over semi- cemi-supervised learning.
INTRODUCTION
Labeling data is expensive. Thus, it is often hard for us to get enough labeled samples. Thus, semi-supervised learning (Chapelle et al., 2009) becomes a serious issue. People try to get good performance with limited labeled data and large amount of unlabeled data. When having limited amount of labeled samples, extracting information from unlabeled data has played an important role for semi-supervised learning. In general, we often applied unlabeled information auxiliary tools, such as pre-training (Hinton and Salakhutdinov 2006) or recursive picking confidence data from unlabeled samples with supervised learning(Zhu 2005). However, notice that when we counter the issue such as Two Half-moon, double circle, or other more complex distribution problem, these methods are lack of considering the spatial distribution information provided by unlabeled samples.
In this paper, we aimed to guide our model to extract the spatial distribution information from unlabeled data. We proposed a new approach for semi-supervised learning by adding our loss function term for our target embedding latent space. Within our proposed model, the neural network can now learn correctness and spatial distribution information from labeled and unlabeled samples simultancously. This provides our feed-forw forward neural network to have more opportunity passing through the sparse margin between clusters, and elevate the performance of the classifier, see more details in Sec. 3) Moreover, it is worth noting that our proposed model does not rely on any additional neural networks, which is suitable for any task and is highly compatible with different supervisd learning algorithms.
In short, the characteristics of our proposed model are as follows: Intuitive The idea of correctness and spatial distribution came up with the characteristies of super- vised and unsupervised learning straightly, which is intuitive.
Compatibility Our method does not rely any additional neural networks but only adding new loss term. Our approach is easy to change into any existing feed-forward neural networks. Extensible We designed our approach by the notion of defining evaluation for spatial distribution, which can replace by any other methods in future researches.
2 RELATED WORK
In recent years, the neural network plays essential role in various tasks; more specifically, they are highly applied in the tasks of image classification. Since then, semi-suuer learning (Weston
et al. 2012 Lee. 2013) for image classification has become vital issue. First of all, some works

---

Under review as a conference paper at ICLR 2021
succeeded by proposing regularization methods for NNs (Bishop 1995 Srivastava al. 2014). They regularize the input and hidden layers of their models by applying random permutations. This can smooth the input-output relation and further get improvements in semi supervised learning.
Generative adversarial networks (GAN) (Goodiellow et al. 2014) are popular research for the neural network, several models (Salimans al., [016] Dumoulin et al. 2016 Dai et al. 2017) had gone more in-depth researches with GAN for semi-supervised learning. They showed remarkable results, especially image classification problems. In 2014, Kingma et al. proposed Deep generative models (Kingma et al. 2014), applying a aeriational-autennoode based generative model to semi-supervised learning. They showed good results on image classification for -une-vieee learning and became the benchmark of several datasets. However, in practice, these models require more careful tuning for parameters. Also, they usually require more neural network structures and computation resources.
Moreover, Rasmus et al. (Rasmus et al. 2015) propose semi-supervised with Ladder Networks, model structure with an autoencoder. This work is similar to denoising autoencoders and applied to every layer. It is impressive that they have got vast improvement compared with Deep Generative models (Kingma et al.) 2014). Shortly after, Miyato ey al. (Miyato et al. 2018) also achieve competitive results on the benchmark data sets with regularization term. They guide their model to minimize the change of the input and output of the network, which does not require labeled information and able to unlabeled samples for their regularization term.
Labeled propagation proposed by (Zhu and Ghahramani 2002) is also a family of methods for semi-supervised learning. By smoothing the model around the input data points, the model can extrapolate the labels of unlabeled samples. Similar to this idea, several works (Laine and Aila 2016)] Sajjadi al. 2016) had succeeded by using the random image augmentation. They try improve the generation performance of image classification for semi-supervised learning.
3 PROPOSED MODEL
For semi-supervised learning, we assume that in some particular space, samples in the same category should be in the same cluster. Following this assumption, once we can distinguish different clusters properly, and guide our network to find the decision boundary with sparser region between clusters.
In this section, we introduce an end-to-end learning method by adding our loss functions. First of all, we tried to guide our network to learn good mapping from the original input space to the embedding latent space; see see /eec. 3.T] We next defined loss functions and tried to cluster the samples that should be in the same categories together in the embedding latent space; see Sec. 3.2) Moreover, similar to some supervised learning works (Xu et al. 2005| Rennie and Srebro 2005 (Srebro et al. 2005). aim to maximize the margin between different clusters separate them well. Note that since we are lack of labeled samples, we maximize the margin within temporary clustering results for our data, instead of using the ground truth of labeled data, see more in Sec. 3.3 Overall, we named our proposed model as Maximum Cluster Margin Classifier, referred to as MCMC.
3.1 EMBEDDING LATENT SPACE
In general, measuring a clustering result is very subjective. However, to deal with various kinds of distribution, we avoid evaluating them directly by their original input space. Instead, we pull out layer from the neural network and set it as the embedding latent space. Next, we evaluate whether the quality of the embedding latent space. In our proposed model, we try to define measurement for the embedding latent space to satisfy our assumptions. This guides the previous layers to learn about good mapping from the original input space to a well-distributed embedding latent space. To strengthen the efficiency of the embedding latent space, we add simple classifier that is fully connected to the embedding latent space.
3.2 DB LOss
3.2.1
DAVIES-BOULLDIN INDEX
As index (Davies and Bouldin 1979) proposed, given a dataset with N clusters, for every cluster C₁, compute S; as a measure of scattering with the cluster, which is defined as
2

---

Under review as a conference paper at ICLR 2021
7
(a) DB problem
(b) Maximum Margin
Figure 1: (a) Since DB loss LDB consider about the centroid of cluster only, this leads the problem of overlapping between different clusters. (b) We compute Maximum Margin loss LasM by maximizing these è pairs single link.
follows:
T Σ
S: = T; x 1
where X is the jth sample of cluster C. A; is the centroid of cluster C. T; is the size of C and p is usually set as To measure the distance between different centroid of clusters C2 and Cj, we compute Mij. which is defined follows:
M, 114, - A,l,
laki
where dRii is the ith element of A,, and usually set as 2. Finally, we combine S; with Mij. which defined as follows:
S Sj
N D
D, max Riy. (t)
DB
(3)
Mi.i
N
3.2.2 DB LOss
Thanks to Davies-Bouldin index Sec. 3.2.1 (Davies and Bouldin, 1979).
have measurement
for evaluating the clustering result in embedding latent space. Eq. (1) provides us the evaluation of whether the samples in single cluster close enough to the centroid. Simultaneously, Eq. (2) separates the centroid of different clusters to become as far as possible. Moreover, Eq. 3 restricts every single cluster could gather together while different cluster has a long distance between their
centroid. We defined our DB loss LD as follows: Given points X with temporary predictions ý, then
CpB DB
where DB can be computed in Sec. 3.2.
Till now, DB loss LDD provides us the capability to guide the samples the same category gather in index (Davies and Bouldin 1979), the
the same cluster. However, when look into measurement considers only about the centroid of each cluster. This causes the issue that different clusters may be overlap with each other. That is, the margin between different clusters has vanished, see Fig.3. For this reason, we came up with Maximum Margin loss, see Sec. 3.3
3.3 MAXIMUM MARGIN LOSS
we need to maximize the margin between different clusters to separate them. We hence came up with Maximum Margin Loss LMM- To push each cluster far as we can, we compute the sum of & pairs single link (Gower and Ross 1969) between different clusters. When maximizing those pairs of single link, it also implies maximizing the margin between
As we mentioned at the start of Sec. 3
3

---

Under review as a conference paper at ICLR 2021
different clusters. We define the Maximum Margin loss LMM follows:
Given clusters C₁ C2 a.y CN. let Vig be the tt pair single link between C2 and Cj.
Also, we define Er f=1 V to be k pairs single link distance,,y Finally, we have
N
LMM
E
(le pairs single link distances,)).
(S)
where & the number of pairs between different clusters, and N the number of the clusters.
Maximum Margin loss LMи provides the information of k k-pair single link distance between different clusters. This loss can guide the model ameliorate the overlap problem and maximize the margin between different clusters. Fig. I(b) shows the main idea of Maximum Margin loss LMM. which helps our network avoid the problem illustrated in Fig. I(a) Overall, by combining DB loss LDB and Maximum Margin loss LarM, we can find our ideal evaluation for clustering results in embedding latent space, Our network is now able to gather the samples that should be in the same categories and separate different clusters far as possible.
3.4 TOTAL LOSS
In this section, we took a more in-depth exploration of how we deal with labeled data and unlabeled data for our total loss separately. For labeled data, we applied loss with cross-entropy (CE) (De Boer et al. 2005), which is widely used for classification tasks in supervised learning. Note that Cross- entropy (CE) (De Boer al., 2005) available change into other losses in supervised learning for personal use. Though labeled data might have small number of samples, they still play an essential role in guiding the network learning the right predictions. Moreover, they can also provide the neural network to learn a rough decision boundary in the early iterations. This leads our network to get a good start and modify the boundary into better results. As we mentioned in Sec. [32.22 and Sec. 3.3 we need DB loss LD guide the samples in the same cluster to group together. At the same time, Maximum Margin loss LM M pushes the different clusters as far as possible, i.e., maximize the margin between different clusters. We define the total loss Lotal as follows:
CTotal = LDB + LCE + LMM..
(6)
3.5 COMPUTING FLOW
For our network training, since need several points to compute our loss in a single iteration, we assume every batch has enough samples. During our computations, we separate DB loss LD into two parts. First of all, we sampled some data as a batch from all training data, including labeled data and unlabeled data. We compute our DB loss for all training data LT B by the embedding latent space and temporary predictions. Next, we compute another DB loss for labeled data L with the embedding latent space and their ground truth labels. We found that adding this loss can speed up training and make our network more stable. It is reasonable since we use ground truth instead of temporary predictions for computing DB loss for labeled data Lb.. This can guarantee that labeled samples in the same categories gather in the same cluster within their ground truth. Hence, it plays an important role in finding a well-distributed embedding latent space that satisfics our assumptions mentioned in the start of
During our training, we give two different batch sizes for Lt and Dh Please note that batch for LbB can always get the labeled samples. Another side, batch for L sampled data from all training data, has the possibility to get samples with no labeled data for single batch. This fact also shows the importance L/B to speed up gathering the samples in the same category together. Overall, the final DB loss Lp is defined as follows:
CDB = CbB + L
Moreover, to maximize the margin between different clusters, we compute Maximum Margin loss LMM by the same process DB loss CY for training data. We now divide computing flow into
4

---

Under review as a conference paper at ICLR 2021
loss (training data)
Maximum Margin loss
Embedding latent space
Labeled data Label information
Training data
classifter
output
Cross cross-entropy
DB loss (labeled data)
Figure 2: Computing flow for our losses in MCMC. We compute cross-entropy by the output of temporary predictions and the ground truth of labeled data, Lb by the embedding latent space and the ground truth of labeled data, 4 and Laa by the embedding latent space and the output of temporary predictions. Finally, we sum up our total loss for MCMC.
Algorithm Training for Maximum Cluster Margin Classifier
Input:
Network ==);0)); classifier g 0g): data Dr. Du, Dr Dr. U Dv; batch sizes Nr, Nr;
learning rate a
Output: Learnt network parameters z(: 02), 9(:0g)
1: Initialize parameters 2(:;0,) g(: 0,)
2: repeat
3:
(XL, y) +- NL samples from DD:
4:
(XT) + Nr samples from Dr:
5:
ZI. + z(xr.; 02); Zr + =(Xr;02)
6:
UL g(Z,1ig); + g(Zr;fg); #Temporary predictions for training data.
7:
8:
LCE CrossEntropy(y, yL):
9:
D + DBI y): b < DBT yr);
LMM < Mazimum.M www.nn.aaagggnnos.ooo ( ;
11:
LTotal + LoE LiB + Lb + LMMi
12:
0g + 0. BL.Tiull 0g + 0g 88,
OLCg.
86
until stopping criterion is true
labeled data and training data, which can also imply learning the correctness and modifying better decision boundary separately. Fig. shows our entire computing flow for each loss and and Algorithhm ] introduces computing details for our training. For the convenience during experiments, we set 10 for LMM without fine-tuning.
4 ToY PROBLEMS
To justify our theory, we first conduct study with two classic experiments, Two Half-moon, and
Triple Circle problem. our experiments, we generate samples with Scikit-learn (Pedregosa et al.
2011) and randomly select labeled samples reproduce the situational simulation of semi-supervised learning. We present the transformation process of the decision boundary. Also, we show the failure case for supervised learning demonstrate the improvements between MCMC and supervised learning. Finally, we take a closer observation of our proposed model.
5

---

MNIST dataset consists of handwriting digit images with image size 1x28x28, including 60000 training data and 10000 testing data. Following the works of Kingma et al. (Kingma et al. (014). we use different numbers of samples for D, from the training set. Note that all classes are balanced. i.c.. each class has the same number of labeled samples. Since our network structure is easy to change into any neural network, in this task, we replace our network into a convolution neural network. We followed the SonwPool-CNN--e structure from Springenberg add our embedding space before the last layer, which is set as 300 and fully connected to 10 neural optimizer, with learning Under review as a conference paper at ICLR 2021
(a) Two Half-moon experiment
(b) Triple Circle experiment
Figure 3: Main figures show the initial labeled samples and the transformation process of the decision boundary for MCMC. The dashed box shows the failure case of supervised learning with labeled samples. (a) Two Half-moon experiment. (b) Triple Circle experiment.
Two Half-moon problem For Two Half-moon problem, we randomly sampled 1000 points with Scikit-learn (Pedregosa et al. (2011) and draw 9 points of them as labeled data. As As 3(a) shows, when adding our loss functions, the decision boundary was modified to better result with considering the spatial distribution information. Moreover, we notice that unlike supervised learning, when our proposed model is trying to separate two clusters, tries to maintain the correctness of the labeled red points simultancously.
Triple Circle problem We applied our proposed model to Triple Circle problem. We sampled 1500 points with Scikit-learn (Pedregosa et al., (20TT) randomly and draw 13 points as labeled data. Fig shows our impressive improvements when considering the spatial distribution with unlabeled samples. On the other side, the result of supervised learning gets unfortunate result though all labeled data are actually in the right categories.
We notice that for multi categories problem, it is still crucial learn rough decision boundary in the early iterations. Overall, we our original hypothesis in this experiment. We show that our proposed loss function can provide the spatial distribution information from unlabeled data and adjust the decision boundary without losing the correctness during training. By decreasing loss within more iterations, the decision boundary can maximize the margin between different clusters. This is precisely in line with our original expectations and is able to extract spatial distribution information from unlabeled samples for semi-supervised learning. These results not only strengthen our confidence for our loss function but also shows impressive works during multi categories problem.
EVALUATION ON REAL DATA
We evaluate model four real data sets, MNIST, USPS, ETH-80, and COIL-100. We reproduce the experiments of the MNIST data set followed by Kingma et al. (Kingma et al. 2014) and compare them with different works. As for USPS, ETH-80. and COIL-100 data sets, follow Zhang al. (Zhang et al.) 2020) and summarize the result of performance comparison. For these experiments, we implement our model within PyTorch (Paszke et al., (2077). Thanks for the tool; helped us done differentiation automatically. Our open source will release in Github after our paper is accepted.
5.1 RESULTS FOR THE MNIST DATASET
al. (Springenberg et al. 2014). We
(10 classes). For training, used ADAM (Kingma and Ba 2014)

---

Under review as a conference paper at ICLR 2021
Text accuracy
accuracy
MCNC
MCNC DBfibbeled
MCMC (ours)
MCMC with no MM
# of labeleed datia
iterations
(a) MCMC vs. SL.
(b) Ablation study
Figure 4: (a) Test accuracy with different amounts of labeled data on MNIST. Figure shows the result of supervised learning (SL) and our proposed model 'MCMC". (b) The transitions of accuracy during different methods on MNIST with 100 labeled data. We tested with four methods, supervised learning (SL), MCMC, MCMC without DB loss for labeled data, and MCMC without Maximum Margin loss. The red box shows the differs the early iterations between different methods.
rate of 0.001, and exponentially decayed the rate during training. We use 50 and 300 for the batch size of labeled data and training data separably. Note that did not build validation data but fixed our training epochs as 30 epochs for every experiment. Also, we used & = 10 for LM and did not fine-tune any parameters or structures. We repeated the experiments ten times with different random seeds for labeled data.
We ran some experiments with different settings on MNIST for 100 labeled samples. As Fig. 4(a) shows, supervised learning within small amount of labeled data shows poor test accuracy and elevates the performance as having more labeled data. Compared with SL, MCMC shows significant improvements in the performance of test accuracy. Fig. 4(b) provides the ablation study results on MNIST with 100 labeled data. In Fig. 4(b) MCMC shows the strongest stability with high accuracy, and supervised learning performs the worst. For MCMC without DB loss for labeled data, the model takes more iterations to achieve better accuracy. As we introduced in Sec. 3.5) this loss is important for the network to learn faster during training. For MCMC without Maximum Margin loss, found the instability for the network during the training. Without Maximum Margin loss, the embedding latent space will occur the overlap problem that showed in Fig. I(a) This causes the failure case for semi-supervised learning. As result, it is crucial for us to add both Maximum Margin loss and DB loss for labeled data, making our model become
stable and further increase the accuracy.
Table summarizes the results of MNIST dataset and shows the reports of the mean and standard deviation with different amounts of labeled data. The mean and standard deviation of error rate both decrease as we add more labeled data. This also implies that labeled samples stand critical role when training model once again. In Table I our proposed method shows remarkable results and outperforms with other methods.
Table I: Test performance on MNIST dataset with the different number of labeled data.
Test error % with used labels
100
1000
ALL
TSVM (Collobert al. 2006)
16.81
5.38
1,40
MTC (Ruta) et al. I
12.03
3.64
0.81
AtlasRBF PPtelis
2014
8.10 (+.0.95
3.68 (+0.11)
1.31
DGM (M1 M2) (Kingma et al.
2014
3.33 (+.1.4
2.40 (+0.02)
0.96
CatGAN (Spinggeberrg 2015)
1.91 (+00.10)
1.73 (+0.18)
0.91
Conv-CatGAv (Springenberg 2015)
1.39 (+0.28)
0.48
VAT (Miyato al.] 2018))
1.36 (+:0.03)
1.27 (+0.11)
0.64 (#0005)
Ladder, full (Rasmus et al. 2015
1.06 (+:0.37)
0.84 (+0.08)
0.57 (+0002)
Ladder, CNN-I (Rasmus ei al.
2015
0.89 (+0.50)
MarginGAN (Dong and Lin 2019
3.53 (+0.57)
2.87(=0.71)
FlowGMM-com GMM- (Izmailov et al. 2019
1.8
1.0
MCMC (ours)
1.01 (E:0.13
0.80 (#:0.07)
0.37 (+0.05)
7

---

Under review as a conference paper at ICLR 2021
5.2 RESULTS FOR THE USPS, ETH-80, COIL-100 DATASET
To evaluate our method in more real problems, we took experiments on USPS, ETH-80, and COIL- 100. Followed by Zhang et al. (Zhang et al. (020), we took 12 different tests with various settings for each data set, and compare with other works.
USPS data set USPS (Hull, 1994) a database of handwritten digits, includes 11000 images with image size 1x16x16. We took experiments with different amounts of labeled samples per class. The number of the labeled digit is set 5, 10, .... 60, corresponds 12 experiments.
ETH-80 data set ETH-80 (Leibe and Schiele, 2003) contains 3280 object images with 8 broad categories and ten subcategories for each category. During our preprocessing, we resized every image into 3x32x32. We took experiments with different amounts of labeled samples per class. The number of the labeled digit set to 10, 25...., 175, corresponds to 12 experiments.
COIL-100 data set COIL-100 (Nene et al., 1996) includes 7200 images with 100 different objects. During our preprocessing, we resized every image into 3x32x32. We took experiments with different amounts of labeled samples per class. The number of the labeled digit is set to 2, 4...., 24, corresponds to 12 experiments.
Table 2 shows the statistic 12 experiments for each data set and the comparison with other works. Different from the concept of label propagation similar works, we do not define the similarity between samples. We assume that the similar samples are also similar in our embedding space, and gather them together by optimizing converge into the same cluster in our embedding space and make our proposed method outperformed with other works.
loss function. This force the dense samples
Table 2: Performance comparison of USPS, ETH-80 and COIL-100
USPS
ETH-80
COIL-100
Mca
Meant.S7D
Mcan=STD
GFHF
(Zhu et al.
2003)
72.57 (+8.72)
71.86 (=5.50)
71.86
LLGC
Zhou et al.
2004
85.34 (#..52)
81.62 (=2.71)
86.36
(13.78)
LNP Wang and ZZhang
2007
86.57 (+-6.25)
80.86 (+2.688)
86.97
(+3.49)
SLP
(Nie et
201O)
85.89 (+-5.25)
80.30 (+-2.89)
87.36 (+3.38)
PN-LP (Zodd et al. (2166)
80.23 (+..24)
72.41 (#4.44)
87.18 (#3.83)
CD-LNP (Zhang al. (015
73.47 (#9.22)
70.11 (+:4.83)
82.91 (14.81)
SparseNP (Zhang et al. (201cc)
89.93 (+2.42)
82.81 (#2.02)
76.36 (#5.34)
ProjLP Zhang et al. 20155)
89.68 (#3.22)
02.39(#1.0)
90.41 (#3.10)
AdaptiveNP (Jia et al. 2016
82.37 (+5.99)
73.29 (+4.34)
78.42 (=4.54)
ALP-TMR (Zhang et al. 2020)
91.12 (43304)
82.26 (+1.95)
91.51 (+4.13)
MCMC (ours)
94.28 (E2.38)
93.38 (=5.91)
94.74
6 CONCLUSION
In this paper, we proposed a new approach, which we called Maximum Cluster Margin classifier (MCMC), for semi-supervise learning. The key idea is that we define a novel loss function for clustering results. In particular, the quality of embedding latent space plays an important role for semi-supervised learning. In our embedding latent space, we guide our network to gather the samples which should in the same categories. Simultaneously, pushing these clusters as far as possible, and find better boundary passing through the sparse region. We showed the indispensability for MCMC to have high stability during training for our losses. Also, we compared with other works and showed outstanding improvements for semi-supervised learning these data sets.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
C. M. Bishop. Training with noise is equivalent to tikhonov regularization, Neural computation, 7(1): 108-116, 1995.
O. Chapelle, B. Scholkopf, and A. Zien. Semi-supervised learning (chapelle, et al., eds.; (boo) reviews]. IEEE Transactions on Neural Networks, 20(3):542-542) 2009.
R. Collobent, F. Sinz, Weston, and L. Bottou, Large scale transductive svms. Journal of Machine Learning Research, 7(Aug):1687-1712, 2006.
7. Dai, 7. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov. Good semi-supervised learning that requires a bad gan. In Advances neural information processing systems, pages 6510-6520, 2017.
D. L. Davies and D. W. Bouldin. A cluster separation measure. IEEE trunsactions on pallern analysis and machine intelligence, (2):224-22) 227, 1979.
P. De Boer, P. Kroese, Mannor, and R. Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1):19--0)) 67, 2005.
J. Dong and T. Lin. Margingan: Adversarial training in semi-supervised learning. In Advances in Neural Information Processing Systems, pages 10440-10449, 2019.
V. Dumoulin, Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, and A. Courville. Adversarially learned inference. arXiv preprint arXiv:1606.0077 2016.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
C. Gower and J. Ross. Minimum spanning trees and single linkage cluster analysis. Journal of the Royal Statistical Society: Series C (Applied Statisties), 18(1):54-64 1969.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-553 5070005-0777000 2006.
Hull. database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-555 0544, 1994.
P. Izmailov, P. Kirichenko, M. Finzi, and A. G. Wilson. Semi-supervised learning with normalizing flows. arXiv preprint arXiv:1912. 2019.
L. Jia, Z. Zhang, L. Wang, W. Jiang. and M. Zhao. Adaptive neighborhood propagation by joint 12, 1-norm regularized sparse coding for representation and classification. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 201-210. IEBE, 2016.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6900, 2014.
D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581-3589, 2014.
S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. arXiv:1610.02242. 2016.
arXiv preprint
D.-H. Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop Challenges in Representation Learning, ICML, volume 3, page 2, 2013.
B. Leibe and B. Schiele. Analyzing appearance and contour based methods for object categorization. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume pages 11-409. IEEE, 2003.
T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: regularization method for supervised and sumi-super learning. IEEE transactions on pattern analysis and machine intelligence, 1799 2018.

---

Under review as a conference paper at ICLR 2021
S. A. Nene, S. K. Nayar, and H. Murase. object image library (coil-100. Technical report, 1996.
Nie, S. Xiang, Y. Liu, and C. Zhang. A general graph-based semi-supervised learning with novel class discovery. Neural Computing and Applications, 190oa) 19(4):59-555, 2010.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation pytorch. 2017.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, al. Scikit-learn Machine learning python. Journal of machine learning research, 12(00c)::225-28800 2011.
N. Pitelis, C. Russell, and L.. Agapito. Semi-supervised learning using unsupervised atlas. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 565-580. Springer, 2014.
A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. In Advances in neural information processing systems, pages 3546-3554, 2015.
J. D. Rennie and N. Srebro. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd international conference on Machine learning, pages 713-719, 2005.
S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and X. Muller. The manifold tangent classifier. In Advances in neural information processing systems, pages 2294-2302, 2011.
M. Sajjadi, M. Javanmardi, and Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pages 1163-1 171, 2016.
Salimans, I. Goodfellow, W. Zaremba, Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances neural information processing systems, pages 2234-2242, 2016.
T. Springenberg. Unsupervised and supevsupe learning with categorical generative adversarial networks. arXiv preprint arXiv: 1XXi::5511066 2015.
J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 1412 2014.
N. Srebro, J. Rennie, and T. S. Jaakkola. Maximum-marg -maginn matrix factorization. In Advances in neural information processing systems, pages 1329-1336, 2005.
N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()): 1929-1958, 2014.
F. Wang and C. Zhang. Label propagation through linear neighborhoods. IEEE Transactions on Knowledge and Data Engineering, 20(1):55-67, 2007.
Weston, Ratle, Mobahi, and R. Collobert. Deep learning via semi-supervised embedding. In Neural Networks: Tricks of the Trade, pages 639.655. Springer, 2012.
L. Xu, J. Neufeld, B. Larson, and Schuurmans. Maximum margin clustering. In Advances in neural information processing systems, pages 1537-1544, 2005.
C. Zhang, S. Wang, D. Li, J. Yang, and H. Chen. Prior class dissimilarity based linear neighborhood propagation. Knowledge-
Systems, 83:58 65, 2015a.
H. Zhang. Z. Zhang, M. Zhao, Q. Ye, M. Zhang, and M. Wang. Robust mateemmarrxx-eoveryy based weightee label propagation for classification. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Z. Zhang, W. Jiang, F. Li, L. Zhang, M. Zhao, and L. Jia. Projective label propagation by label embedding. In International Conference on Computer Analysis of Images and Patterns, pages 470-481. Springer, 2015b.
10

---

Under review as a conference paper at ICLR 2021
Z. Zhang, L. Zhang, M. Zhao, W. Jiang, Y. Liang, and F. Li. Semi-supervised image classification by nonnegative sparse neighborhood propagation. In Proceedings of the 5th ACM on International Conference Multimedia Retrieval, pages 139-146, 2015c.
D. Zhou, O. Bousquet, T. N. Lal, Weston, and B. Schölkopf. Learning with local and global consistency. In Advances in neural information processing systems, pages 321-328, 2004.
X. Zhu, Z. Ghahramani, and J. D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings the 20th International conference on Machine learning (ICML-03), pages 912-919, 2003.
X. Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin- Madison Department of Computer Sciences, 2005.
X. Zhu and Z. Ghahramani. Learning from labeled and unlabeled data with label propagation. 2002.
O. Zoidi, A. Tefas, N. Nikolaidis, and I. Pitas. Positive and negative label propagations. lEEE Transactions on Circuits and Systems for Video Technology, 28(2):342-355 2016.


Reviews:
[{"review_id": "ikdGRvylMJy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] In this work, authors dealt with the problem of improving classification performance in the setting of semi-supervised learning. Authors exploit inherent knowledge on the samples distribution via clustering to improve the latent space. The work heavily relies on well-studied concepts including the  Davies-Bouldin index and maximum margin clustering.\n\n\n[Cons]\n-- The general idea to improve the semi-supervised classification task by exploiting knowledge from the data through clustering constraints.\n\n\n[Pros]\n-- The lack of a formalist to describe the technique turns down the paper. This also makes unclear the level of novelty.\n-- The results are ultimately less informative than one would like and the experimental setting is limited. This leads to the results being no more than case studies which demonstrates only limited advantages in both qualitative and quantitative terms. \n\n\n\nDetailed comments for authors:\n\n\n--  [Novelty] The novelty in the proposed technique is unclear as it leverages on already well-studied principles including Davies-Bouldin index and maximum margin clustering. Authors need to strongly  detail the main advantages and key insights of the technique. As it is described so far, it is hard to appreciate the level of contribution.\n\n-- [Lack of formalism] A major drawback that turns down the paper is the lack of formalism to describe the proposed technique (which also makes unclear the level of novelty). That is, the core part of the technique is  (6) and previous expressions (1)-(5) are drawn from already well-studied concepts (Davies-Bouldin index and maximum margin clustering). One would expect that the authors provide further details (6) such as explicitly define the optimisation process.  Not only that, but several expressions are not properly set -- for example (5) authors needs to set the explicit definition rather that the text ‘k pairs single link distance’ as there are several works using the principles of maximum margin clustering, the reader needs to see the specifics of each component in the technique (even (5) draw from a very well-studied principle yet it is not defined properly). Overall, it is not enough to set (6) but authors should reduce (1)-(5) which are well-established expressions and detail (6) along with important parts such as the optimisation. \n\n-- A major drawback of the components are not considered. For example, it is well-known that the Davies-Bouldin index can report good results, however, it does not imply the best information retrieval. Why then chose Davies-Bouldin index? There are several works such as [*] that can be used to generalise the latent space well whilst avoiding the disadvantage of the authors proposed components and affect positively the outcome from SSL [**].  \n\n[*]Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018). Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 132-149).\n[**] Sellars, Philip, Angelica Aviles-Rivero, and Carola Bibiane Schönlieb. \"Two cycle learning: Clustering based regularisation for deep semi-supervised classification.\" arXiv preprint arXiv:2001.05317 (2020).\n\n\n-- Experimental setting. The results are ultimately less informative than one would like and the experimental setting is limited. This leads to the results being no more than case studies which demonstrates only limited advantages in both qualitative and quantitative terms. \n-- Authors open the paper with a big statement “the comparison results confirm our model’s outstanding performance over semi-supervised learning” However, authors use very small and not well-representative datasets such as CIFAR-100, ImageNet (or even a smaller version mini-ImageNet). Likewise the comparison setting is limited, there are no comparisons with recent other techniques such as FixMatch, UDA. Deep Label propagation etc. Otherwise the strong claims of the authors are unsupported. \n-- The findings are less ultimate than one would like. The results from the tables are far from being well-explained. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T9Izk3xoR-f", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a novel loss function for semi-supervised learning. Arguing that SOTA semi-supervised learning methods neglect spatial information (latent clustering structure) in the data, the authors propose a loss function which combines clustering objectives with classification objectives. The proposed loss function combines the cross-entropy loss, the within-cluster scatter as known from k-means, the distance between centroids and the margin between classes. The proposed loss is notably non-continuous since it makes use of the maximum function. The authors employ  ADAM with a learning rate of 0.001 with exponential decay to optimize the novel loss function. Experiments on MNIST and three comparably small datasets show that the proposed method is able to achieve high accuracy with only few labeled data points.\n\nReview:\nCombining the elements of clustering and classification makes total sense for the case of semi-supervised learning. In that view, I find the conducted approach very interesting. However, the paper leaves many details unclear to the reader - in particular with regard to the definition of the loss function and the optimization (see minor issues below). I am missing a discussion about the optimization issues of the proposed loss function. The loss function is not continuous and in practice, this makes a loss susceptible to local minima. There might also be cases where numerical instabilities occur, for example if the centroids are close to each other and the DB loss is dividing by a small constant. In addition, I wonder about the sensitivity to the optimization parameters, such as the learning rate scheduling.\nThe experiments on synthetic data (half moons and three circles) look nice, but don't add much value. The classes/clusters in the datasets can be detected by simple clustering algorithms, additional information by a few labeled data points is not needed. Hence, these datasets are not suitable for demonstrating the power of semi-supervised methods. I would be more interested in examples where the points in one class are actually not connected/close to each other. \nThe experiments on real world data show a good performance of the proposed optimization of the novel loss function. Although  I don't believe that one always has to show experiments on a big, big dataset like imagenet, I think that at least a dataset of the magnitude of Cifar-10 and Cifar-100 should be considered. The learning rate is chosen to be comparatively small. How does that affect the convergence rate of the method on larger datasets? In addition, I would encourage a description of characteristics of the real world datasets which shows why they pose an interesting case (balanced/unbalanced classes, number of classes, easy/difficult to classify). \n\nMinor issues:\n* How are the centroids computed and when? If not all the data is labeled, then we can not compute the centroids as the average of points assigned to one cluster/class, because we need the centroid to determine the assignment of points to clusters/classes. Since DB is also reflecting the distance between centroids, the centroids are apparently additionally learned parameters? This aspect would become more clear if the loss functions would be denoted as functions. That is, instead of denoting a loss as, e.g., DB, denote it as DB(A,X). In Algorithm 1, L_{DB} is defined as DBI(Z_L,y). DBI is not defined as far as I see. I assume that DBI should actually reflect DB. Z are described as the network parameters, but what that entails is unclear.\n* In deep learning, there is actually an inherent connection to k-means clustering. In fact, SOTA deep networks compute a latent embedding where every class is represented by a centroid. This is discussed here:\nHess, Sibylle, Wouter Duivesteijn, and Decebal Mocanu. \"Softmax-based Classification is k-means Clustering: Formal Proof, Consequences for Adversarial Attacks, and Improvement through Centroid Based Tailoring.\" arXiv preprint arXiv:2001.01987 (2020).\nThis insight might be useful, since it says that the embedded space induced by cross-entropy learning *just* has to be tweaked in order to reflect a suitable clustering as well. This might lead to a more simple loss function. In addition, it puts the proposed approach more in perspective and adds theoretical value.\n* What is the definition of k pairs single link distance in Eq.(5)?\n* I wouldn't call the method MCMC, this is known as Markov Chain Monte Carlo.\n* Table 1 returns the test error and Table 2 the test accuracy, use the same metric!\n* Use the citation in the text, instead of saying \"X et al. (X et al.) propose\" say \"X et al. propose\" if the template allows it, where X et al. is the citation or say \"(X et al.) propose\".\n* Define the parameters used in Algorithm 1.\n* How many classes does the ETH dataset have? The description says only 8 categories with 10 subcategories.\n* Please go over the text and use tools such as grammarly to improve the writing. For some senences I couldn't make out their meaning. For example: \"Note that cross-entropy is is available o change into other losses in supervised learning for personal use.\"\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PG0xY51egep", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to address the semi-supervised learning topic by proposing a method based on an aggregated loss considering both cross-entry and Davies-Bouldin Index. Cross-entropy is used to ensure the maximum margin between classes and Davies-Bouldin Index is applied to the labeled data and to the whole dataset, respectively, to ensure a high quality of clustering. Evaluations in four small and simple datasets are reported to demonstrate the effectiveness of the proposed method.\n\nThe idea behind the proposed method is pretty simple and straightforward. It makes sense and so I believe that the method proposed is correct conceptually. However, a correct method may not necessarily be effective, which would require an extensive evaluation. That is what the paper lacks. Let me elaborate in detail.\n\nFirst, all the four datasets used in the evaluations are small in scale and simple in class distributions. This is NOT the sate-of-the-art anymore. There are larger in scale and more complex datasets available and why did the authors fail to use these datasets instead?\n\nSecond, the comparison studies reported in Fig. 4 are not clearly documented and possibly are not fair either. Which specific supervised learning method did you use in this comparison? Did you use the same number of labeled data samples for the supervised learning method? The same questions are also raised for the toy example experiment reported in Fig. 3. If the supervised learning method uses the same number of the labeled samples, it is not a fair comparison obviously as your method uses more regularization constraints in the loss function. If a different number of the labeled samples is used, it is not a fair comparison either. Note that more constraints you use in the loss function, the number of labeled samples required can vary more, depending upon the different data distributions. Consequently, the comparison studies as well as the ablation studies reported in Fig. 4 can be misleading, possibly just showing the best case. More datasets with varying complexity in distributions may need to further the comparison studies before any meaningful conclusion can be reached.\n\nThird, Davies-Bouldin Index is just one of the intrinsic evaluation metics for cluster analysis and there are others such as silhouette coefficient and Calinski-Harabasz score. In principle, you may also use these metics. I wonder whether you have investigated this and why you picked up Davies-Bouldin index.\n\nFinally, the paper has a lot of grammatical errors and typos that should have been fixed before the paper was sent in for review.\n\nAs a minor comment, I don’t like the acronym MCMC for this proposed method. The same acronym is already in use as a very popular statistical machine learning method in the literature. The authors may want to change to another name.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6dXNH1GXgGz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed algorithm is not properly put in the context of other semi-supervised learning methods that are closely related. These include label propagation, clustering auto encoders, entropy minimization / margin maximization, prototype networks and semi-supervised clustering. Examples of these lines of research are: \n\nZhu and Ghahramani, Learning From Labeled and Unlabeled Data With Label Propagation,2002\nXie, Girschick and Farhadi, Unsupervised Deep Embedding for Clustering Analysis, 2016\nGrandvalet and Bengio, Semi-supervised learning by entropy minimization, 2005\nSnell et al, Prototypical Networks for Few-shot Learning, 2017\n\nThe idea of the paper is to penalize maximize distances between pairs of samples from different clusters as determined by the classifier output as a proxy for unlabeled samples. The paper claims previous works do not consider the spatial information provided by unlabeled examples. This is not correct. Many methods take advantage of such information such as label propagation (uses proximity between examples to propagate labels), entropy minimization (uses density of examples), etc. The novelty of the method over these previous techniques is not clear to me\n\nThe experiments section of the paper has some shortcomings as well. The datasets used in Figure 3 are also easily solved by other semi-supervised learning methods. The discussion of these figures does not lend insight into what type of problems that are challenging for previous methods are addressed by the proposed approach. Some of the experimental results do not appear to be compared to the state-of-the-art. For instance, Sajjadi et al, Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning, 2016 reports significantly better results for the MNIST dataset with 100 labeled examples compared to the results included in Table 1. Finally, the datasets used with the exception of MNIST are not standard ones used in recent semi-supervised learning papers. CIFAR-10 and STL-10 would have been better choices. \n\nThe clarity of the paper and English use is also below average. Some sentences contain informal phrases that might not be suitable for a scientific paper such as “People try…”, \"Thanks for the tool; it helped us done differentiation automatically.\"  or vague statements such as \"in some particular space, samples in the same category should be in the same cluster\". The paper goes to lengths describing simple concepts such as cluster centroids and distance between cluster centroids (equations 1-3) yet concepts more crucial to the algorithm such as ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the writing is not precise. It does not help to find any novelty in the ideas, and the limited and too quickly described experiences are not convincing enough to forgive this problem. The authors chose not to oppose or comment on the detailed arguments provided by the reviewers. I agree with the reviewers in recommending the rejection of this paper. 
Decision: Reject