AC-VAE: Learning Semantic Representation with VAE for Adaptive Clustering
Authors: Xingyu Xie
Minjuan Zhu
Yan Wang
Lei Zhang
Venue:  (2021)
Keywords: Unsupervised Representation Learning
Neighbor Clustering
Variational Autoencoder
Unsupervised Classification

Abstract: Unsupervised representation learning is essential in the field of machine learning, and accurate neighbor clusters of representation show great potential to support unsupervised image classification. This paper proposes a VAE (Variational Autoencoder) based network and a clustering method to achieve adaptive neighbor clustering to support the self-supervised classification. The proposed network encodes the image into the representation with boundary information, and the proposed cluster method takes advantage of the boundary information to deliver adaptive neighbor cluster results.  Experimental evaluations show that the proposed method outperforms state-of-the-art representation learning methods in terms of neighbor clustering accuracy. Particularly, AC-VAE achieves 95\% and 82\% accuracy on CIFAR10 dataset when the average neighbor cluster sizes are 10 and 100. Furthermore, the neighbor cluster results are found converge within the clustering range ($\alpha\leq2$), and the converged neighbor clusters are used to support the self-supervised classification. The proposed method delivers classification results that are competitive with the state-of-the-art and reduces the super parameter $k$ in KNN (K-nearest neighbor), which is often used in self-supervised classification.

Under review as a conference paper at ICLR 2021
AC-VAE: LEARNING SEMANTIC REPRESENTATION WITH VAE FOR ADAPTIVE CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised representation learning is essential in the field of machine learning. and accurate neighbor clusters of representation show great potential to support unsupervised image classification. This paper proposes VAE ( (Variational Au- toencoder) based network and a clustering method achieve adaptive neighbor clustering to support the self-supervised classification. The proposed network en- codes the image into the representation with boundary information, and the pro- posed cluster method takes advantage of the boundary information to deliver adap- tive neighbor cluster results. Experimental evaluations show that the proposed method outperforms ofateoof-hhe-art representation learning methods in terms of neighbor clustering accuracy. Particularly, AC-VAE achieves 95% and 82% ac- curacy on CIFARIO dataset when the average neighbor cluster sizes are 10 and 100. Furthermore, the neighbor cluster results are found converge within the clus- tering range (a S 2), and the converged neighbor clusters are used to support the self- seff-supervised classification. The proposed method delivers classification results that are competitive with the state-of-the-ar and reduces the super parameter KNN (K-nearest neighbor), which is often used in self-supervised classification.
INTRODUCTION
Unsupervised representation learning is long-standing interest in the field of machine learning (Peng et al., 2016a; Chen al., 2016: 2018; Deng al., 2019; Peng et al., 2016b), which offers promising way to scale-up the usable data amount for the current artificial intelligence methods without the requirement for human annotation by leveraging on the vast amount of unlabeled data (Chen et al., 2020b;a). Recent works (Chen et al., 2020b;a; He et al., 2020) advocate to structure the unsupervised representation learning at the pre-training stage and then apply semi-supervised or self- supervised techniques the learned representations in the fine-tuning stage. So the representation learning acts as a feature extractor, which extracts semantic features from the image, and well- extracted features should lead excellent classification performance (He et al., 2020). Moreover, representation learning assigns close vectors to images with similar semantic meanings, thus making it possible cluster the same meaning images together (Xie al., 2016; Van Gansbeke et al., 2020). When no label is available, unsupervised or self-supervised classification methods rely on the neighbor clustering to provide the supervisory signal guide the self-supervised fine- fine-tuning process (Van Gansbeke et al., 2020; Xie et al., 2016). In this scenario, accurately clustering neighbors among representations is crucial for the followed classification fine-tuning.
In many of the prior unsupervised methods (Van Gansbeke et al., 2020; Xie et al., 2016). the neighbor clustering process is performed by KNN (k-nearest neighbor) based methods. However, KNN based methods introduce k as super parameter, which needs to be fine-tuned regarding different datasets. In an unsupervised setup, selecting a suitable & without any annotation or prior knowledge is not straightforward. Therefore it is desirable to have neighbor clustering process that automatically adapts to different datasets, thus eliminating the need for pre-selecting the super parameter k.
To achieve adaptive neighbors clustering, the proposed method tries to encode the image repre- sentation into the multivariate normal distribution, as the multivariate normal distribution provides distance information, such as z-score, which can naturally adapt different datasets without the help of any additional mechanism. Prior works (Kingma & Welling, 2013; Higgins et al., 2016; Burgess al., 2018) 8) showed VAE's ability to encode images into multivariate normal distributions; nonethe-

---

Under review as a conference paper at ICLR 2021
less, these works struggled to extract high-level semantic features, most of them trained by image recovery tasks, which encourages the network to focus on the low-level imagery features. Consequently, the extracted low-level features cannot be utilized in the unsupervised classification method, which needs semantic features to function.
(a; μa, Σ)
2-4core
Eneoder
T
Z
T
T
Regulation
Encoder
Decoder
My
Cluster
range
VAE Based Network
Z-score Clustering
The proposed clustering method includes a VAE based network and z-score based cluster methods. The VAE based network encodes the image into the multivariate normal distribution, and the z-score based clustering method takes advantage of the distribution's ' boundary information.
Figure I:
To provide VAE with the ability to extract the high-level semantic features, as well as to utilize its strength to produce adaptive clusters, this paper proposes framework, AC-VAE, including VAE based network and z-score based clustering method, as shown in Figure The VAE based network encodes the image into the multivariate normal distribution ((, Σ), The distribution' mean şi is taken as the representation; meanwhile, its z-score provides the boundary information that naturally adapt different datasets. The proposed clustering method takes advantage of the boundary information to achieve adaptive neighbor clustering. The proposed framework's efficacy evaluated on CIFARIO, CIFAR100-20 and SLT datasets, and surpasses the current kate-of-the- art methods in neighbor clustering on these datasets. Particularly, AC-VAE achieves 95% and 82% accuracy on CIFAR1O dataset when the average neighbor cluster sizes are 10 and 100, surpassing the current state-of tatte-of-the-rr method by margin of 10%. Our main innovations and contributions can be summarized as follows:
This work proposed a VAE based network to encode the image into the representation with its boundary information. The representation and boundary information retrieved from the multivariate normal distribution, which encoded from the image. The efficacy of the adaptive boundary demonstrated by neighbor clustering results.
In this work, a loss function is proposed based on consistency regulation train the VAE- based network for extracting the high-level semantic feature from the image. Experiments demonstrate that the proposed method assigns close vectors images with similar semantic meanings.
This work proposed a clustering method to take advantage of the adaptive boundary of each representation. The proposed method delivers high accuracy neighbor clusters. Besides, the neighbor clusters found converge within the clustering range (a < 2), and the self- supervised learning framework utilizing the converged clusters delivers competitive results without the need of pre-selecting parameter lt.
RELATED WORKS
Many frameworks cluster the dataset directly into semantic classes, and train the network in an end- to-end manner (Asano al., 2019; Caron et al., 2019; Haeusser et al., 2018; Yang et al., 2016; Xie et al., 2016). Although the end-to-end training method is easy to apply, the network's initialization largely influences these frameworks' performance. Therefore, complex mechanisms (such cluster
2

---

Under review as a conference paper at ICLR 2021
reassignment) are needed to assist the clustering process. As an alternative approach, methods (Caron et al., 2018; Hu et al., 2017: Yang et al., 2016) based on maximizing the mutual information between image augmentations are proposed to address this issue.
In contrast to end-to-end training, the multi-stage method (Van Gansbeke et al., 2020) is introduced, which first aim to obtain accurate neighbor clusters from the representation learning, then apply these neighbor clusters in the followed finetuning, and this method made breakthroughs in unsuper- vised classification. This method depend mainly the accurate neighbor cluster results from the representation learning. A large number of representation learning methods (Doersch et al., 2015; Gidaris et al., 2018; Noroozi & Favaro, 2016; Pathak et al., 2016; Zhang et al., 2016) have been introduced, and these methods usually assign pretext tasks to the network, and the network learns the image representation by solving these tasks. However, most of these methods aim to use the learned representations to serve the following supervised or semi-supervised tasks. Therefore, the neighbor clustering performance of these learned representations is not optimized, and few of these methods have strong neighbor clustering performance. SimCLR (Chen et al., 2020a) and MoCo (He et al., 2020) utilizing consistency regularization outstanding other methods in neighbor clustering performance and assisted SCAN framework (Van Gansbeke et al., 2020) to reach the state-of-the tet-hhe-rr art results in unsupervised classification tasks. However, the SCAN framework needs to pre-select the super parameter & to perform the KNN clustering from the starter.
This paper aims to provide an adaptive clustering method that needs no super parameters by creating the boundary for each representation. The representation and its boundary information retrieved from VAE based structure. VAE based networks are typically used for image generation tasks (Kingma & Welling, 2013; Razavi et al., 2019) or disentanglement tasks (Higgins al., 2016; Burgess et al., 2018). Although VAE shows the potential to encode images into multivariate normal distributions (Razavi al., 2019; Burgess et al., 2018), the efficacy of utilizing VAE to extracting high-level representations is not heavily studied. Besides, VAEs are usually trained by different forms of image recovery tasks, which keep VAE away from extracting high-level semantic features.
This paper adopts the consistency regulation to train the proposed VAE based network for extract- ing the high-level representation and its boundary information. Moreover, a clustering method is proposed to utilize this boundary information to deliver adaptive cluster results. In the end, these adaptive clusters are utilized in the unsupervised classification task.
3 METHOD
The following sections presents the generative network that produces the representation and its boundary information first, and then introduces the cluster method that benefits from the bound- ary information.
3.1 GENERATIVE MODEL.
In the unsupervised setup, the ground truth of the desired image representation not available. However, there are general assumptions about the desired presentation's behavior pattern, i.e., how desired representations should interact with each other, such as images showing the same kind of object should have similar semantic representations. This paper introduces the latent vector that controls the representation's 's behavior pattern and utilizes VEA based network to generate repre- sentation from this latent vector. The proposed network aims generate representations that follow the behavior assumption of the expected representation. In that case, the generated representation and the expected one would share the same latent vector, as the latent vector decides the represen- tation's 's behavior pattern. However, the generated representation may differ from the expected one, even when they have the same behavior pattern. is hard to directly generate the desired represen- tation, which needs the ground truth to train from the starter. Therefore, this paper adopts the latent vector as close approximation of the desired representation.
The proposed VAE based network is shown in Figure 2. multivariate normal distribution (xx; μ, Σ) encoded from the image x by an encoder e(x). Then, sample x is drawn from this distribution with a stochastic process. This random process creates variations that support tryouts to find this latent vector's 's acceptable range in the training stage. Besides, the distribution's 's mean fa also taken standard latent vector provide short-cut for
This paper states the latent vector
3

---

Under review as a conference paper at ICLR 2021
better encoder training. As the encoder is deep neural network for extracting high-level semantic features, the stochastically sampled latent vector 2 cannot provide a stable guide to train the deep encoder. The sum of the sampled latent vector and the standard latent vector μ is fed into the decoder d(x)) to generate the representation r.
Behavio ior Regulation
T
Encoder
Sampling
Decoder
x
d(x²
e(x)
N(:; pe, Σ)
Figure 2: The framework of the proposed VAE based network.
The network is trained by the behavior regulations, which impose regulations on the generated representation. This work adopts consistency regulation, commonly used behavior assumption of the semantic representation, which regulates an image and its argumentations to have the same semantic representation. The consistency regulation can be performed by minimizing the behavior loss stated in Equation (1).
BLg d(r.,r)) e(TT)))
in which, T(x;) is the augmentation of image Jss r; is the representation of x; generated by the proposed network v(-), and d(,) measures the distance of two representations.
As the proposed network based on VAE, the loss function of the vanilla VAE (Kingma Welling, 2013) is adapted to train the proposed network by replacing its image recovery loss with the behavior loss Lzg The loss function used train the proposed network is shown in Equation (2).
E Bo-g
(2)
in which, P(r(z) distribution of the representation r generate by latent vector 2, P(2) is the distribution of the latent vector, (z((x) is the distribution of given and [L)) is the KI. divergence between Q=2(=)) and P(z).
As mentioned earlier, the latent distribution will act as close approximation of the desired represen- tation. The mean μ will be regarded as the image representation, and the z-score of the distribution characterizes its boundary information.
3.2 NEIGHBBR CLUSTERING METHOD
This work clusters the images based on the boundary information based z-score. The insight is that, when an image encoded into the distribution, the image's 's close variations should be located within small z-score range of this distribution. Figure (a) illustrates the representation and its boundary information, 7-score range, in a five-dimensional distribution. For neighbor clustering, the neighbor that its means ya fall into the required 7-score ranges will be clustered, and an illustrated cluster criterion is shown in Figure (b). This cluster criterion is strict, as it requires the clustered neighbor not only close the root of the cluster but also has a similar flow as the root.
For fast calculation, this work proposes a z-score based distance-vector, in which each element of this vector corresponds to the distance each dimension. The z-score is used because the direct compression of z-scores between different normal distributions is veiled. The proposed z-score based distance-vector d d(s,,1z) between 2; and xy shows Equation (3).
abs(14: 11) 0.5.
d(s,))))
(3)
20:
4

---

Under review as a conference paper at ICLR 2021
in which, p4, is the mean of 2; 's latent distribution, and is the diagonal elements of the covariance matrix Σ. The a controls the z-score range to be used. When a = 1. the distance is normalized by the z-score range [-1, 1]. This work will expand the z-score range to increase the cluster size until it reaches [-2,2], covering more than 95% the normal distribution. However, the sample falls out of the 7-score range of [-2,2] is most unlikely from this distribution's 's population, therefore, the z-score range is limited within [-2, 2]. To be clustered, all element of the z-score based distance vector should not larger than O. Besides, d(sr.ay) may not equal ((s..r.)) as the z-score range of different representations may differ from each other, as demonstrated in Figure 3 (c).
By modifying a in Equation (3), the z-score based distance will change accordingly. The cluster threshold a indicates how strict the clustering criterion is: small will tighten the cluster restriction, and large a will reduce the restriction. As the experiment section 4 will demonstrate, cluster converging after surpasses certain value are observed on all evaluated datasets, so the a needs no fine-tuning for each dataset.
Notably, the early converge is observed in the experiments, in which the cluster size stops from increasing before reached the desired cluster number. This situation introduced by the strict clus- tering method, which requires the neighbor representation satisfy the criterion in all dimensions. In some cases, the clustering criteria are hard reach; hence the clustering process stops. To address this issue, this work introduces the loose match strategy and parameter 0 (0- < 0 < 1.. Instant of requiring a full match in every dimension as the standard clustering process, a certain mismatch,
1 - 0, is accepted, demonstrated in Figure (d). The loose match strategy is backup method and is unnecessary for the unsupervised classification, which will be demonstrated in the experiment section.
mean
Loose Match
acopet
accpet
with
z-score range
reject
Joose match
reject
BBC
(a)
(b)
(c)
(d)
Figure 3: (a) Representation A is illustrated as five-dimensional distribution. The dots represent the mean of each dimension, and its z-score range indicated by the bars; (b) Representation B is clustered by A, as all the means of B fall into the z-score range of A. However, representation C cannot join the A's cluster as some of C's means fall out of the required z-score range. (c) Represen- tation A cannot be clustered by B, even representation B is clustered by A. Because B has narrow z-score range; (d) With the loose match strategy, the representation D can be accepted to join the A's cluster, although one of its means cannot fall into the required z-score range.
4 EXPERIMENTS
This paper first evaluates the neighbor cluster performance of the proposed method. Then uses the neighbor cluster results in self-supervised classification framework to demonstrate the potential to adapt the proposed method to support self-supervised classification. At last, additional feature of the proposed method is introduced.
4.1 EXPERIMENTAL SETUP
Experiments are performed on CIFARIO (Krizhevsky et al., 2009), CIFAR100-20 (Krizhevsky et al., 2009) and STL.0 (Coates al., 2011). The proposed network is trained the training set of all
5

---

Under review as a conference paper at ICLR 2021
Table 1: Neighbor Cluster Results Training Set
CIFARIO
CIFAR 1O0-20
STL
Methods
10
50
100
10
50
100
10
50
100
Rot + KNN
65.58
62.06
60.08
51.76
51.41
50.38
60.69
54.71
50.02
InstDise KNN
74.73
73.96
73.08
56.66
56.41
54.28
71.52
61.82
55.38
InstDise KNN
71.33
67.78
64.38
57.76
56.36
55.68
66.24
59.71
54.78
MoCo KNNN
75.28
70.72
68.27
56.92
56.49
54.83
71.97
57.66
59.89
SimCLR + KNNN
81,53
76.76
73.58
61,56
54.05
49.33
75.82
67.11
62.58
VAE KNN
75.71
73.34
72.32
59.27
52.04
48.38
74.71
72.83.08
58,92
AC-VAE
95.25
87.72
82.27
68.75
65.51
62.48
92.75
87.93
85.42
datasets. For all experiences, the same set of configurations is applied. A ResNet-34 adopted as the encoder network, and two-layer full connection is utilized as the decoder network. The latent distribution dimension is set as 512, and the decoded representation vector has a size 64. Both encoder and the decoder networks are initialized randomly. Besides, the NT _ent loss (Chen al., 2020a) and its augmentation strategy are used for the behavior loss implementation.
4.2 EVALUATION OF NEIGHBOR CLUSTER
The neighbor cluster performance is evaluated under cluster sizes of 10, 50, and 100, as desired that clustering method can maintain high accuracy when keep increasing the cluster size. The accuracy of the neighbor cluster is obtained by averaging all neighbor clusters' accuracy. While in the proposed method, neighbor clusters have different cluster sizes, the average cluster size is used for convenient comparison. For comparison, two instant discrimination based methods (Caron et al., 2018; 2019), Rot (Coates et al., 2011), SimCLR (Doersch al., 2015), and MoCo (Gidaris et al., 2018) are chosen, as they all use consistency regulation to perform unsupervised represen- tation learning. To get the cluster results, KNN is applied to the learned representation. For the proposed VAE based network, both KNN and z-score based methods are employed for clustering. The comparing methods are not suitable use the z-score based clustering method, as it requires the distributional information. The neighbor cluster accuracy on the training set is studied, as the neigh- bor cluster results on the training set will be utilized to support the self-supervised classification training.
The neighbor cluster accuracy comparison on the training set shown in Table 1. The proposed VAE based network with KNN is compared with others to demonstrate its ability to project the same meaning images to close representations. With KNN clustering. the proposed method's accuracy lower than the state-of-the-an the proposed network creates uncertainty in the representation, which contributes the decline of accuracy. After the KNN is replaced by the z-score based clustering method, the proposed methods (AC-VAE) outperformed all other methods all cases. Notably, around 10% increases are found when cluster size is 10 on CIFARIO and STL datasets. This performance comes from the z-score based cluster method, which uses the boundary information to exclude those nearby samples that do have the same flow shape. Figure 4 is used demonstrate the efficacy of the proposed clustering method. In Figure 4, the representations from different classes overlap each other in some areas. It hard for the method that only considers the distance to deliver highly accurate cluster.
SimCLR. This is because the sampling process introduced by
To utilize the neighbor cluster method in unsupervised classification, the parameter a's effect on the cluster size and cluster accuracy also studied. The results are shown Figure 5. Clusters have been found naturally converged in the training set of all three datasets within the range of < < 2. As shown Figure 5 (a), the cluster size remains the same after the threshold a reaches a certain point. These results benefit from encoding the image as the distribution. For distribution, most of its population will fall into its z-score range of [-2, 2] (a = 2), covering 95% the distribution population. During the network training, the VAE-based network should push samples with the similar meanings into this high- sossiiossiillity region. This analysis matches the experiment results, in which the converges happened before the 7-score range expands to [-2, 2), as shown in Figure 5.

---

Under review as a conference paper at ICLR 2021
Figure 4: Visualization of representation produced by VAE based network using t-sne (Maaten & Hinton, 2008) for dimensionality reduction.
Table 2: The and 0 applied in the experiments
Dataset
CIFARIO
CIFAR100-20
STL
Cluster Size
10
50
100
10
50
100
10
50
100
0.52
0.94
1.28
1.45
1.72
0.57
1.24
1.42
N/A
N/A
N/A
N/A
0.95
0.91
N/A
0.89
0.92
To get the results mentioned in Table 1, the and 0 listed in Table 2 are applied. However, these super parameters precisely selected for easy compassion to the KNN based clustering results. In practice, there is no need fine-select a nor reach a specific cluster size unless particular neighbor cluster size is highly desired.
Clustered Neightus
120
Cluster Accuracy
0.85
80
0.8
0.75
0.7
0.65
0
0.6
0.5
1
(b)
Figure 5: (a) the clustered size increases increases and reaches converged number within S 2; (b) the cluster accuracy decreased as the a increases and maintains the accuracy within o: S 2.
4.3 SELF SUPERVISED CLASSIRICATION
In order to demonstrate the efficacy of utilizing the proposed method to perform unsupervised classi- fication, the self-supervised classification framework SCAN (Van Gansheke al., 2020) adapted by replacing its KNN based neighbor clustering method with the proposed method to perform the unsupervised classification. The converged clusters showed in section 4.2 are utilized, in which all the clusters naturally converged without using the loose match strategy nor fine-tuning There- fore, the self-adaptive cluster results are used to perform the unsupervised classification. As Table demonstrated, the framework utilizing the adaptive cluster results outperform most of the comparing methods. When compared to the sfate-of-the-ans (SCAN, Van Gansbeke et al. (2020)), the proposed method still delivers competitive results without selecting super parameter k.

---

Under review as a conference paper at ICLR 2021
Table 3: The Unsupervised Classfication results
Methods
CIFARIO
CIFARI00-20
STL
DEC (Xie et al., 2016)
30.1
18.5
35,9
ADC (Haeusser et al., 2018)
32.5
16.0
53.0
DeepCluster (Caron al., 2018)
37.4
18.9
33,4
DAC (Chang al., 2017)
52.2
23.8
47.0
IIC (Ji al., 2019)
61.7
25.7
59.6
SCAN with KNN clustering
87.6
45.9
76.7
SCAN with (C-VAB
79.2
40.2
71.8
4.4
OTHER ADVANTAGES
The proposed method also has additional advantages, desired in the unsupervised setup, such as prototype selecting. As different representation has different boundary information, some samples will cluster far neighbors than others under the same z-score range, each class's prototype can easily identified as those who cluster the most neighbors. The selected prototypes of each dataset are shown in Figure 6.
(b)
(c)
Figure 6: Prototype images on the different datasets: (a) Cifar10, (b) 10-2r100-20, (c) STL.
CONCLUSION
This paper proposes AC-VAE, including a VAE based network and clustering method. The VAE based network encodes the image into the multivariate normal distributions. The semantic represen- tation and its boundary can be retrieved from this distribution. The clustering method takes advan- tage of the boundary information to achieve adaptive neighbor clustering. Experiments demonstrate that the proposed VAE-based network has the ability to project images with the same semantic meaning into close representations. Experiments also show the efficacy of the proposed method to from adaptive clusters. This work attempts to push the edge fully unsupervised classification by omitting a critical super-parameter & in the state-of-the-art method. Experiments show that the nat- urally converged cluster supports the unsupervised classification framework to deliver competitive results.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv: 2019.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03500 03599, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. Proceedings of the IEEE International Conference on Computer Vision, pp. 2959-2968. 2019.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In Proceedings of the IEEE international conference on computer vision, pp. 5879-5887, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised edels are strong supervipd learners. arXiv preprint wxXi:2200.100029 2020b.
Yuanyuan Chen, Lei Zhang, and Zhang Yi. A novel low rank represennaation algorithm for subspace clustering. Inf. J. Pattern Recognit. Artif. Intell., 30444401610000071111600000000110 2016. doi: 10. 00155028801416500000 URL https: /oii http://i.c/...../1///..
65000075.
Yuanyuan Chen, Lei Zhang, and Zhang Yi. Subspace clustering using low-rank constrained autoencoder. Inf. Sci., 424:27-38, 2018. doi: 10.10166j.ns.2017.000.47
URL https:
ins 2017 201.0.9.........
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223,00 2011.
Hongli Deng, Lei Zhang, and Lituan Wang. Global context-dependent recurrent neural net- work language model with sparse feature learning. Neural Comput. Appl., 31(S-2):999- 1011, 2019, doi: 10.1007/600521-017-3006-- s0C521-010 -17--065-----
URL https://dci.org: wtt.:i..orr11...............1... 1007/
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422 1422-1430, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07726. 2018.
Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers. Associative deep clustering: Training classification network with no labels. In German Conference on Pattern Recognition, pp. 18-32. Springer, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7729,9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.

---

Under review as a conference paper at ICLR 2021
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-auumented training. arXiv preprint arXiv:1702.08720, 2017.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9570v):2579-2665 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solvingjjgsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-25544 2016.
Xi Peng, Huajin Tang, Lei Zhang, Zhang Yi, and Shijie Xiao. A unified framework for representation-basee subspace clustering of out-of-sample and large-scale data. IEEE Trans. Neu- ral Nenworks Learn. Syst., (7(1)):2499-25111 2016a. doi: https: / //oiioi.o.o /tt.////.... 1109 TNNLS. 2015 2490080.
24900000... URL
Xi Peng, Miaolong Yuan, Zhiding Yu, Wei-Yun Yau, and Lei Zhang. Semi-supervised subspace learning with 12graph. Neurocomputing, 208: 143-152, 2016b. doi: 00.1016/).neucom.2200 112. URL httpt:://////// http::///oo..ooo..o 111.1/1.....1111 nww.om..eemmmmmmmmmmmmmm 201..11.........................
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876, 2019.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis. Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. European Conference on Com- puter Vision (ECCV), 2020.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487, 2016.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5 5147-5156, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
10


Reviews:
[{"review_id": "uy6XbDuWMzc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adaptive neighbor clustering method by estimating normal distribution on the representation space. The proposed neighbor clustering can utilize the acceptable range for each dimension of each instance from the estimated variance, which leads to different size of neighbors for each instance, and improved the neighbor clustering performances. In addition, the proposed neighbor clustering method can replace the KNN-based neighbor clustering in the previous SCAN (Semantic Clustering by Adopting Nearest neighbors) framework for image semantic clustering.\n\nOverall, the paper is hard to follow and understand with many typos, incorrect notations, and especially the main term, VAE. Why can it be called VAE without a decoder which decodes some original input from a compact representation? I think the proposed network contains only the encoder (transforming z to r is also encoding), and it applies the stochastic process in the middle of encoder (512-d z instead of 64-d r). Therefore, it seems that the proposed training can be considered as the previous contrastive learning with stochastic regularization. And, it naturally raises the question that why not imposing the normal distribution on the final representation r? In addition, I do not understand why the sum of the sampled z and \\mu is fed into for obtaining r. What if z is solely fed into? What is v in Eq. (1)? Is it d \\circ e? How to explicitly connect Eq. (1) and (2)?\n\nIn Table 3, The clustering performance drop of SCAN with AC-VAE is not marginal compared to the previous SCAN with KNN, even though the proposed AC-VAE seems to produce significantly improved neighbor clustering results in Table 1. However, there is no analysis on this.\n\nThe amortized inference of normal distribution on the (middle of) representation space, and the use of its estimated variance in obtaining reliable neighbors for each instance seems to be make sense, however the motivation, insight, and contribution of the derived framework seem to be very limited.\n\nTypos: SLT -> STL, VEA -> VAE, argumentations -> augmentations, d(x_i, x_i) -> d(x_j, x_i)", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajd0eJ-sTQp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\n\nThis paper proposes an encoder-decoder architecture for clustering images based on their semantic (high-level) features as well as creating a semantic representation for downstream tasks. The model takes as a basis the encoder part of a variational auto-encoder (VAE) and uses the properties of the distributions induced in the latent space to define cluster boundaries and avoiding the need of setting the parameter K (number of clusters), usually required in methods based on K-Means. In order to induce the encoder to focus on high-level features, a decoder loss based on consistency regulation is defined, which aims at mapping variations of each image, obtained by data augmentation transformations, to similar decoded representation.\n\nQuestions and Suggestions:\n\nShouldn’t be abs(mu_i - x_j) In Equation (3)?\n\nz-scores are usually defined as (mu-x)/std. However, in Equation (3), defining d(x_i, x_j), there is a “-0.5” term. Although this equation is supposed to represent a distance, it can result in negative values when abs(mu_i-mu_j) is small. I suggest removing this term and compare the result against 0.5 instead of 0.0. Also, include the 2 in the denominator in the alpha parameter and make it range in [0.0, 1.0] instead of [-2.0, 2.0].\n\nEquation (2) is not an equation, since it has no equal sign. As it defines a loss function, I suggest using equality to introduce a symbol defining the loss.\n\nExplain Figure 2 in the caption, indicating what the terms mean as well as the doted and solid lines.\n\nFix typos: \n“argumentations” -> “augmentations” (section 3.1)\n“Compression” -> “Comparison” (section 3.2)\n\nPros:\n\nThe paper is focusing on important problems in the field of representation learning.\nIt presents good clustering accuracy results in challenging datasets\nIt seems to boost self-supervised classification.\n\nCons:\n\nThe presentation and clarity of the article should be better and it would benefit from a review of the English\n\nThe use of terminology is not precise, leaving the reader oftentimes confused about what is meant. To give a few examples:\n- “Autoencoders”: The architecture does not map inputs to inputs themselves, so strictly speaking it is not an AE model.\n- “Adaptive Clustering”: This can mean different things in the literature, for instance, it is also used to mean methods that deal with non-static distributions. But in the article, the term is not clearly defined and just means establishing the number of clusters.\n\n- The use of the words neighbor/neighborhoods: is confusing: “neighbor cluster performance”: is this evaluating the capacity of the model to map similar clusters close to each other (making them neighbors) or of mapping similar samples close to each other  (regular clustering evaluation). Also, “neighbor cluster accuracy”: Is this the same as “Clustering Accuracy”, a standard metric to evaluate clustering methods, or is this evaluating something else? How is it computed? \n\n- Define what these terms mean: “Behaviour Pattern”, “Root of cluster”, “Similar flow as the root”\n\nThe review of the literature misses some important articles, which should also be compared with in the experiments:\nVincent Fortuin et al, 2019 “SOM-VAE: Interpretable Discrete Representation Learning on Time Series”\nFlorent Forest et al 2019 “Deep Embedded SOM: Joint Representation Learning and Self-Organization”\n\nEvaluating Clustering is tricky and the use of only one metric is usually not enough and misleading. I suggest including other metrics such as Normalized Mutual Information and Purity to make the evaluation more sound.\n\nThe use of only CIFAR in evaluation is also a problem, as clustering datasets usually present a great variability with different methods. Other papers also consider MNIST and FASHION MNIST, for instance.\n\nThe article replaces the parameter K with two other parameters: alpha and theta. Although it states that these parameters are easier to adjust than K, the experiments are not enough to show that, It is missing especially one experiment evaluating the results of other methods with different values of K. It can be the case that with K high enough other methods will produce similar results in downstream tasks.\n\nHow many clusters the does method find? Is it the expected number? This is not covered by clustering accuracy.\n\nConclusion:\n\nAlthough the paper is interesting and has some novelty aspects, I am not convinced about the soundness of the results due to the lack of evaluation with other metrics and different datasets. Also, I believe that other similar methods, mentioned above, should be included in the comparison. Finally, experiments showing that alpha and theta are easier to adjust than K are needed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ca5ehvl9QwK", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes the approach  for clustering the image data using z-scores for semantic representation.\n\nWhile the method seems novel, there are questions to be clarified, and the final rating is dependent on the answers on those questions:\n\n1. The paper states that the method performs adaptive clustering. This statement needs clarification. From that claim I would generally expect that it means adapting to non-stationary data but it has apparently a different meaning in the context of the description of the method. Instead, as the reviewer can see from Figure 3, it is a z-score based cluster assignment. It is unclear what is the precise meaning behind the ‘adaptive’ quality. \n2. It is unclear what would happen if the same point is matched to different clusters. Would it be possible to formalise it all as an algorithm so that the described procedure is reproducible?\n3. Section 3.2 Besides, $d(x_i,x_j)$ may not be equal to $d(x_i,x_i)$. There is an apparent typo: should be $d(x_j,x_i)$?\n4. On section 4.4, while the prototypes are shown, it would also be important to see the examples of neighbours of the prototype by their increased distance.\n5. It is said in the discussion of the Eq 1 that *\" The consistency regulation can be performed by minimizing the behavior loss stated in Equation (1): $BL_x = d(r_i,r_i^′) = d(v(x_i),v(T(x_i)))$ in which, $T(x_i)$ is the augmentation of image $x_i$, $r_i$ is the representation of $x_i$ generated by the proposed network $v(\\cdot)$, and $d(\\cdot)$ measures the distance of two representations.”* It is not clear how exactly the augmentation of image is constructed and what distance is used. Could this loss function be described in more detail?\n\n6. For Table 1, the results are shown with specially selected parameters. As the comment goes that it is possible to do it without task-specific $\\alpha$ parameter, would it be possible to show these results for the common value of $\\alpha$?\n7. It is said in section 4.2 that *'With KNN clustering, the proposed method’s accuracy is lower than the state-of-the-art, SimCLR. This is because the sampling process introduced by the proposed network creates uncertainty in the representation, which contributes the decline of accuracy.’* As I understand, the stochastic representation is essential for the z-score (but not for KNN). Therefore, is there a way to support this claim by the ablation study, i.e. produce a deterministic (autoencoder based, for example) representation and see what the accuracy would be with the deterministic representation + KNN ?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses automatically learning the neighborhood size (they call adaptive neighbor support) for unsupervised representation learning with a VAE.  The neighborhood size is determined based on z-scores from by estimating a normal distribution in the latent space. 

The paper is poorly written.  There are several grammatical errors and typos that distracts from understanding the paper.  In addition, the use of terminology is not precise, which adds to the confusion, as pointed out by the reviewers.

AC-VAE is better than VAE+KNN in Table 1 but worse in SCAN with KNN in Table 3.  Further analysis to understand why this is so is needed.

Additional measures of cluster quality is recommended.

As pointed out by the reviewers, this paper is below the acceptance threshold for ICLR.  The reviewers provided several constructive suggestions.  Please refer to detailed reviewer comments to help you improve your paper.

Decision: Reject