Deep Clustering and Representation Learning that Preserves Geometric Structures
Authors: Lirong Wu
Zicheng Liu
Zelin Zang
Jun Xia
Siyuan Li
Stan Z. Li
Venue:  (2021)
Keywords: Deep Clustering
Manifold Representation Learning

Abstract: In this paper, we propose a novel framework for Deep Clustering and multimanifold Representation Learning (DCRL) that preserves the geometric structure of data. In the proposed DCRL framework, manifold clustering is done in the latent space guided by a clustering loss. To overcome the problem that clusteringoriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving intra-manifold structure locally and a ranking loss for inter-manifold structure globally. Experimental results on various datasets show that the DCRL framework leads to performances comparable to current state-of-the-art deep clustering algorithms, yet exhibits superior performance for manifold representation. Our results also demonstrate the importance and effectiveness of the proposed losses in preserving geometric structure in terms of visualization and performance metrics. The code is provided in the Supplementary Material.

Under review as a conference paper at ICLR 2021
DEEP CLUSTERING AND REPRESENTATION LEARNING THAT PRESERVES GEOMETRIC STRUCTURES
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we propose a novel framework for Deep Clustering and multi- manifold Representation Learning (DCRL) that preserves the geometric structure of data. In the proposed DCRL framework, manifold clustering is done in the latent space guided by a clustering loss. To overcome the problem that clustering- oriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving aitra-manifold structure locally and ranking loss for inter-manifole structure globally. Experimental results on various datasets show that the DCRL, framework leads to performances compa- rable to current state-of-the--rt deep clustering algorithms, yet exhibits superior performance for manifold representation. Our results also demonstrate the impor- tance and effectiveness of the proposed losses in preserving geometric structure in terms of visualization and performance metrics. The code is provided the Supplementary Material.
INTRODUCTION
Clustering, a fundamental tool for data analysis and visualization, has been an essential research topic in data science and machine learning. Conventional clustering algorithms such K-Means (MacQueen, 1965), Gaussian Mixture Models (GMM) (Bishop, 2006), and spectral clustering (Shi & Malik, 2000) perform clustering based on distance or similarity. However, handcrafted distance or similarity measures are rarely reliable for large-scale nigh-dimensional data, making it increasingly challenging to achieve effective clustering. An intuitive solution is to transform the data from the high-dimensional input space to the low-dimensional latent space and then to cluster the data in the latent space. This can be achieved by applying dimensionality reduction techniques such as PCA (Wold et al., 1987), t-SNE (Maaten & Hinton, 2008), and UMAP (McInnes et al., 2018). However, since these methods are not specifically designed for clustering tasks, some of their properties may be contrary to our expectations, e.g., two data points from different manifolds that close in the input space will be closer in the latent space derived by UMAP. Therefore, the first question here is how to learn the manifold representation that favers clustering?
The two main points for the multi-manifol representation learning are Point (1) preserving the local geometric structure within each manifold and Point (2) ensuring the discriminability between different manifolds. Most previous work seems to start with the assumption that the label of each data point is known, and then design the algorithm in a supervised manner, which greatly simplifies the problem of multi-manifold learning. However, it is challenging to decouple complex cross- over relations and ensure discriminability between different manifolds, especially in unsupervised settings. One natural strategy is to achieve Point (2) through performing clustering in the input space to get pseudo-labels and then performing representation learning for each manifold. However, clustering is in fact contradictory to Point (1) (which will be analyzed in detail in Sec. 3.3), making it important to alleviate this contradiction so that clustering helps both point (1) and point (2). Thus, the second question here is how cluster data that favors learning manifold representation?
To answer these two questions, some pioneering work has proposed to integrate deep clustering and representation learning into unified framework by defining a clustering -rrennee loss. Though promising performance has been demonstrated on various datasets, we observe that a vital factor has been ignored by these work that the defined clustering-oriented loss may deteriorate the geometric

---

Under review as a conference paper at ICLR 2021
structure of the latent space' ', which in turn hurts the performance of visualization, clustering gener- alization, and manifold representation. In this paper, we propose to jointly perform deep clustering and multi-manifold representation learning with geometric structure preservation. Inspired by Xie et al. (2016), the clustering centers are defined as a set of learnable parameters, and we use a clus- tering loss to simultaneously guide the separation of data points from different manifolds and the learning of the clustering centers. To prevent clustering loss from deteriorating the latent space, an isometric loss and a ranking loss are proposed to preserve the intra-manifold structure locally and inter- manifla structure globally. Finally, we achieve the following three goals related to clustering, geometric structure, and manifold representation: (1) Clustering helps to ensure mnter-oanifold dis- criminability; (2) Local structure preservation can be achieved with the presence of clustering; (3) Geometric structure preservation helps clustering.
The contributions of this work are summarized as below:
Proposing to integrate deep clustering and multi-manifole representation learning into a unified framework with local and global structure preservation.
learning algorithms that deal with all point pair rela- tionships between different manifolds simultancously, we set the clustering centers as a set of learnable parameters and achieve global structure preservation in a faster, more efficient, and easier to optimize manner by applying ranking loss to the clustering centers.
Unlike conventional
Analyzing the contradiction between two optimization goals of clustering and local struc- ture preservation and proposing an elegant training strategy to alleviate it.
The proposed DCRL algorithm outperforms competing algorithms in terms of clustering effect, generalizability to out-of-sample, and performance in manifold representation.
2 RELATED WORK
Clustering analysis. As fundamental tool in machine learning, it has been widely applied in various domains. One branch of classical clustering is K-Means (MacQueen, 1965) and Gaus- sian Mixture Models (GMM) (Bishop, 2006), which are fast, easy to understand, and can be ap- plied to a large number of problems. However, limited by Euclidean measure, their performance on high- nigh-dimensional data is often unsatisfactory. Spectral clustering and its variants (such as SC- Neut (Bishop, 2006)) extend clustering to high-dimensional data by allowing more flexible distance measures. However, limited by the computational efficiency of the full Laplace matrix, spectral clustering is challenging to extend to large-scale datasets.
Deep clustering. The success of deep learning has contributed to the growth of deep clustering. One branch of deep clustering performs clustering after learning a representation through existing unsupervised techniques. For example, Tian al. (2014) use autoencoder to learn low dimensional features and then run K-Means to get clustering results (AE+K -Meens). Considering the geometric structure of the data, N2D (McConville al., 2019) applies UMAP to find the best clusterable mani- fold of the obtained embedding, and then rum K MMMeans to discover higher-quality clusters. The other category of algorithms tries optimize clustering and representation learning jointly. The closest work to us is Deep Embedding Clustering (DEC) (Xie et al., 2016), which learns mapping from the input space to a low dimensional latent space through iteratively optimizing clustering-orienter objective. As modified version of DEC, while IDEC (Guo et al., 2017) claims to preserve the local structure of the data, in reality, their contribution is nothing more than adding a reconstruction loss. JULE (Yang et al., 2016b) unifies unsupervised representation learning with clustering based on the CNN architecture to improve clustering accuracy, which can be considered as a neural exten- sion of hierarchical clustering. DSC devises dual autoencoder to embed data into latent space, and then deep spectral clustering (Shaham et al., 2018) is applied to obtain label assignments (Yang et al., 2019). ASPC-DA (Guo al., 2019) combines data augmentation with self-paced learning to encour- age the learned features to be suster-oriented While sometimes they both evaluate performance in terms of accuracy, we would like to highlight that deep clustering and visual relf-supervised learn- ing (SSL) are two different research fields. SSL typically uses more powerful CNN architecture (applicable only to image data), and uses sophisticated techniques such as contrastive learning (He
'This claim was first made by IDEC (Guo et al., 2017). but they did not provide experiments to support it. In this paper, however, we show that the geometry of the latent space is indeed disrupted by visualization of learned embeddings (Fig. 4), visualization of clustering process (Fig. A3), and statistical analysis (Fig. A5).
2

---

Under review as a conference paper at ICLR 2021
et al,, 2020), data augmentation (Chen al., 2020), and clustering (Zhan et al., 2020; Ji et al., 2019; Van Gansbeke et al., 2020) for better performance on large-scale datasets such as ImageNet. Deep clustering, however, uses general MLP architecture (applicable to both image and vector data), difficult to scale directly to large datasets without considering those sophisticated techniques.
Manifold Representation Learning. Isomap, learning, aims to capture global nonlinear features and seek an optimal subspace that best preserves the geodesic distance between data points (Tenenbaum et al., 2000). In contrast, some algorithms, such as the Locally Linear Embedding (LLE) (Roweis & Saul, 2000), are more concerned with the preservation of local neighborhood information. Combining DNN with manifold learning, the re- cently proposed Markov-Lipschit Deep Learning (MLDL) algorithm achieves the preservation of local and global geometries by imposing Locally Isometric Smoothness (LIIS) prior constraints (Li et al., 2020). Furthermore, multi-manifold learning is proposed to obtain intrinsic properties of dif- ferent manifolds. Yang et al. (2016a) proposed supervised discriminant isomap where data points partitioned into different manifolds according to label information. Similarly, Zhang et al. (2018) proposed a cemi-supervised learning framework that applies the labeled and unlabeled training sam- ples to perform the joint learning of local weighborhoodd-pessee on multi-manifol learning, the problem considered from the perspective that the label is known or partially known, which significantly simplifies the problem. However, it is challenging decouple multiple overlapping manifolds in unsupervised settings, and that is what this paper aims to explore.
representative algorithm of single-manifold
features. In most previous work
PROPOSED METHOD
Consider a dataset X with N samples, and each sample I; E Rd is sampled from C different manifolds {Mo)ez Assume that each category in the dataset lies in compact low-dimensional manifold, and the number of manifolds is prior knowledge. Define two nonlinear mapping 2 = fz; 0f) and y; g(z,, 0g), where E R is the embedding of in the latent space, y; is the reconstruction of x;- The j-th cluster center is denoted as H4 Rm where {1z); is defined as a set learnable parameters. We aim to find optimal parameters O, and μ so that the embeddings { can achieve clustering with local and global structure preservation. To this end, denoising autoencoder (Vincent et al., 2010) shown in Fig is first pre-trained in an unsupervised manner to learn initial latent space. Denoising autoencoder aims to optimize the self-reconstructionn loss LAE = MSE(f, v). where the is a copy of x with Gaussian noise added, that is, & = + o2) Then the autoencoder is finetuned by optimizing the following clustering-oriente losses {Lrank (2, p), LLIS (x. Lalign (2, p)}. Since the clustering should be performed features of clean data, instead of noised data è that is used in denoising autoencoder, the clean data x is used for fine-tuning.
loss (Leluster (2, [}}} and structure-oriente
LAE
Lia8
$ 0/x
y
Letuster Latton
Lrank
μ
Figure 1: The framework of the proposed DCRL method. The encoder, decoder, latent space, and cluster centers marked as blue, red, green, and purple, respectively.
3

---

Under review as a conference paper at ICLR 2021
3.1 CLUSTERING-ORIENTER LOSS
First, the cluster centers { in the latent space are initialized (the initialization method will be introduced Sec 4.1). Then the similarity between the embedded point 2; and cluster centers {uy1zm1 j=1 is measured by Student's t-distribution:
(1 || -1011111111) -1:l1?)
9ij
(I)
Σj 1 11 || - 13-112)
The auxiliary target distribution is designed to help manipulate the latent space, defined as: R
where f3 Σ
Pij Eydyy1s
93
where fj is the normalized cluster frequency, used to balance the size of different clusters. Then the encoder optimized by the following objective:
Loluster KL(PIIQ) ΣΣ
Pij log Pij
(3)
hi
The gradient of Leasste with respect to each learnable cluster center U4 can computed as:
aLduster - (1+11) -0ll²) (pis - 9bp) (z+ pcy)
Touuster facilitates the aggregation of data points within the same manifold, while data points from different manifolds are kept away from each other. However, we find that the clustering-oriented loss may deteriorate the geometric structure of the latent space, which hurts the clustering accuracy and leads to meaningless representation. To prevent the deterioration of clustering loss, we introduce isometry loss LLIS and ranking loss Lyank to preserve the local and global structure, respectively.
3.2 STRUCTURE-ORIENTER LOSs
Intra-manifold Isometry Loss. The intra-manifodd local structure is preserved by optimizing the following objective:
N
LLIS Σ
Σ t=1 JEN?
Idx (II, - dz (21, (((x;;) = I(x,))
(5)
where 2 represents the neighborhood of data point z; in the feature space Z, and the ANN is ap- plied to determine the neighborhood. π--) E {0, }} is an indicator function, and ((x;) is a manifold determination function that returns the manifold s; where sample 2, is located, that 8; 1(x,)
arg max; Pij. Then we can derive C manifolds ( [M,] M. = {I,; 8; = c, 1,2, N). In nutshell, the loss LLIS constrains the isometry within each manifold.
Inter-manifold Ranking Loss. The inter-manifold global structure is preserved by optimizing the following objective:
Lyank
dz (p) - dx (15% x
(6)
where {x:x) IC y 1 is defined as the centers of different manifolds in the original input space X with u/" JM, ΣieM, I4 (j = 2,,,.... C). The parameter determines the extent to which different manifolds move away from each other. The larger x is, the further away the different manifolds are from each other. The derivation for the gradient of Lyank with respect each learnable cluster cen- ter nj is placed in Appendix A.1. Note that Lrank is optimized in an iterative manner, rather than by initializing (15)g once and then separating different clusters based only the initialization
4

---

Under review as a conference paper at ICLR 2021
results. Additionally, contrary us, the conventional methods for dealing with inter-manite sep- aration typically impose push-away constraints on all data points from different manifolds (Zhang et al., 2018; Yang et al., 2016a), defined
NN
Leep
dz (z)) T(((xt) + ((x,))
1=1
The main differences between Lrank and Lsep are as follows: (1) seep imposes constraints on embedding points { which in turn indirectly affects the network parameters 0y. In contrast, Irank imposes preeeerreervttoon constrains directly on learnable parameters { in the form of regularization item to control the separation the clustering centers. (2) Lrank is easier to optimize, faster to process, and more accurate. Lsep is imposed which involves X point-to-point relationships. This means that each point may be subject to the push-away force from other manifolds, but at the same time, each point has to meet the isometry constraint with its neighboring points. Under these two constraints, optimization difficult and is easy to fall into local optimal solution and output inaccurate results. In contrast, Leank is imposed directly on the clustering centers, involving only C'C cluster-to-cluster relationships, which avoids the above problem and makes easier optimize. (3) The parameter introduced in Lrank allows us to control the extent of separation between manifolds for specific downstream tasks.
all data points from different manifolds,
Alignment Loss. Note that the global ranking loss Lrank imposed directly the learnable parameter {my))==- so optimizing Trank only updates (11) rather the encoder's parameter 0-- However, the optimization of { not only relies on Trankk but also constrained by Lelusters which ensures that data points remain roughly distributed around cluster centers and do not deviate significantly from them during the optimization process. Alignment loss Lalign
auxiliary
term, aims to help align learnable cluster centers (nz);= with real cluster centers {ug CC and make this binding stronger:
Lalign Σ |1ag of11
(8)
where {0715 13=1 are defined as of = My ΣiEM, 24 (j = 1,2,...,0 The derivation for the
gradient of Latign with respect to each learnable cluster center μı is placed in Appendix A.1.
3.3
TRAINING STRATEGY
3.3.1
CONTRADICTION
The contradiction between clustering and local structure preservation is analyzed from the forces analysis perspective. As shown in Fig 2, we assume that there exists a data point (red point) and its three nearest neighbors (blue points) around cluster center (gray point). When clustering and local structure preserving are optimized simultancously, it very easy to fall into a local optimum, where the data point is in steady-state, and the resultant force from its three nearest neighhors is equal in magnitude and opposite to the gravitational forces of the cluster. Therefore, the following training strategy applied to prevent such local optimal solutions.
Figure 2: Force analysis of the contradiction between clustering and local structure preservation.
5

---

Under review as a conference paper at ICLR 2021
3.3.2 ALTERNATING TRAINING AND WEIGHT CONTINUATION
Alternating Training. To solve the above problem and integrate the goals of clustering and structure preservation into a unified framework, we take an alternating training strategy. Within each epoch, we first jointly optimize Leluster and Lrank in mini-batch, with joint loss defined
L1 LAE a
Lrank
(9)
where a is the weighting factor that balances the effects of clustering and global aunk-preservttion. Then at each epoch, we optimize isometry loss LLIS and Lolign on the whole dataset, defined as
L BLLI + Latign
(10)
Weight continuation. At different stages of training, we have different expectations for the cluster- ing and structure- preserving.... At the beginning of training, successfully decouple the overlapping manifolds, we hope that the Lauster will dominate and LL.S will be auxiliary. When the margin between different manifolds is sufficiently pronounced, the weight a for Leinster can be gradually reduced, while the weight ,3 for I.LIS can be gradually increased, focusing on the preservation of Appendix A.2.
the local isometry. The whole algorithm is summarized in Algorithm
Three-stage explanation. The training process can be roughly divided into three stages, as shown in Fig 3, explain the training strategy more vividly. At first, four different manifolds overlap. At Stage 1, Laluster dominates, thus data points within each manifold converge towards cluster centers form spheres, but the local structure of manifolds destroyed. At Stage Lrank dominates, thus different manifolds in the latent space move away from each other to increase the manifold margin and enhance the discriminability. structure from the spherical shape with LLIS dominating. It worth noting that the above losses may coexist with each other rather than being completely independent at different stages, but that the role played by different losses varies due to the alternating training and weight continuation.
stage 3, the manifolds gradually recover their original local
Stage
Stage
Stage 3
B
Figure 3: Schematic of training strategy. Four different colors and shapes represent four intersecting manifolds, and three stages involve the clustering, separation, and structure recovery of manifolds.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
In this section, the effectiveness of the proposed framework evaluated in 6 benchmark datasets: MNIST-fulL, MNIST-test, USPS, Fashion-MNIST, REUTERS- -OK and HAR, on which our method compared with other methods mentioned in See 2 in evaluation metrics including metrics designed specifically for clustering and manifold representation learning. The brief descriptions of the datasets are given in Appendix A.3.
Parameters settings. Currently we use MLP architecture for this version and will extend it to ConvAE in the future. The encoder structure of the input data, and the decoder is its mirror. After pretraining. in order initialize the Iearnable clustering centers, the t-SNE is applied to transform the latent space Z dimensions further, and then the K-Means algorithm is to obtain the label assignments for each data point 2, The centers of each category in the latent space are set as initial cluster centers {uy } The batch size is set 256, the epoch set to 300, the parameter l for nearest neighbor set and the parameter
50000500-50002000000 10 where is the dimension
"Since cluster centers (mz) are learnable and updated in an iterative manner, we believe that a proper initialization is sufficient, and the exploration of initialization methods beyond the scope of this paper.

---

Under review as a conference paper at ICLR 2021
is set to 3 for all datasets. Sensitivity analysis for parameters i and is available in Appendix A.12. Besides, Adam optimizer (Kingma & Ba, 2014) with learning rate X=0.001 is used. As described in See 3.3.2, the weight continuation is applied to train the model. The weight parameter for duster decreases linearly from 0.1 to O within epoch 0-150. In contrast, the weight parameter 3 for LLIS increases linearly from 0 to 1.0 within epoch 0-150. In this paper, each set of experiments is run times with different 5 random seeds, and the results are averaged into the final performance metrics. The implementation uses the PyTorch library running on NVIDIA v100 GPU.
Evaluation Metries. Two standard evaluation metrics: Accuracy (ACC) and Normalized Mutual Information (NMI) (Xu al., 2003) are used to evaluate clustering performance. Besides, six evalu- ation metrics are adopted in this paper to evaluate the performance of multi-manifold representation learning, including Relative Rank Error (RRE), Trustworthiness (Trust), Continuity (Cont), Root Mean Reconstruction Error (RMRH), Locally Geometric Distortion (LGDD and Cluster Rank Accu- racy (CRA). Limited by space, their precise definitions are available in Appendix A.4.
4.2
EVALUATION OF CLUSTERING
4.2.1
QUANTITATIVE COMPARISON
The metries ACC/NMI of different methods on various datasets are reported in Tab 1. For those comparison methods whose results are not reported or the experimental settings are not clear on some datasets, we run the released code using the hyperparameters provided in their paper with the same random seeds and initialization, then report their average performance, and label them with (*). While ASPC-DA achieves the best performance on three datasets (MNIST-test, MNIST-full, and USPS), its performance gains do not come directly from clustering, but from sophisticated modules such as data augmentation and self-paced learning. Once these modules are removed, there is a very large degradation in its performance. For example, with data augmentation removed, ASPC-DA achieves less competitive performance, e.g., an accuracy of 0.931 (vs 0.988) on MNIST-full, 0.813 (vs 0.973) on MNIST-test and 0.768 (vs 0.982) on USPS. Though ASPC-DA is based on the MLP architecture, its mmage-based Data Augmentation (DA) cannot be applied directly to vector data, which explains why ASPC has no performance advantage on the vector-based REÜTERS- and HAR datasets (even compared to DEC and IDEC).
In a fairer comparison (without considering ASPC-DA), we find that DCRL outperforms K-Means and SC-Ncut by significant margin and surpasses the other seven competing DNN-based algo- rithms on all datasets except MNIST-test. Even with the MNIST-test dataset, we still rank second, outperforming the third by 1.1% In particular, we obtained the best performance on the Fashion- MNIST and HAR (vector) dataset, and more notably, our clustering accuracy exceeds the current SOTA method by 5.1% and 4.9%, respectively.
Table 1: Clustering performance (ACC/MM) of different algorithms on six datasets.
Algorithms
MNIST-fail
MNIST-test
USPS
Fashion-MNIST
REUTERS-IOK
HAR
K-Means
0.3320.500
301
0.668/0.001
0.474X0.312
0,395103775
0.599/0,588
SC-Neut
0.656/0.331
0.66000.704
0.649/0.794
0.508/0.57
0.658/0.401
0,538/0.741
AE+K Means
0.818/0.747
0.815007844
0.662/0.693
0.566/0.585
0.721/0.432*
0.674/0.670*
DEC
0.903/0.854
0.885/18811
0. 889/0.8873*
0.554/0.576*
0.773/0.528 522-
0.759/0 695*
IDEC
0.918/0.86*
0.876/0.817
0. 893/0.876*
0,572/0.601
0.785/0.541
0.7860.718*
JULE
0.96400.933
0.961/0.915
0.950/0.913
0.563/0 608
0.978/0.941
0.980/0.946
0.869/08857
0.662X1645
ASPO-DA
0:988/0.966
0.973/00936
0.982/09951
0.591/0.68 654
ASPC (wlo DA)
0.931/0.886+
798
0. 867
0.576/0.6324
0.692/0.418*
0.769/0.822
N
0.969/0.928
0.954/0.897
0 0994411898*
0.67 1/6667
0.784/0.536*
0.79600721
DCRL (ours)
0.980/0.946
0.972/0.930
0.960/0902
0.710/0.655
0.836/0.590
0.845/0.788
4.2.2 GENERALIZABILITY EVALUATION
Tab 2 demonstrates that learned DCRL can generalize well to unseen data with high clustering accuracy. Taking MNIST-full as an example, DCRL was trained using 50,000 training samples and then tested on the remaining 20,000 testing samples using the learned model. In terms of the metrics ACC and MNI, our method is optimal for both training and testing samples. More importantly, there is hardly any degradation in the performance of our method on the testing samples compared
7

---

Under review as a conference paper at ICLR 2021
to the training samples, while all other methods showed significant drop in performance, e.g., DEC from 84.1% to 74.8%. This demonstrates the importance of geometric structure preservation for good generalizability. The testing visualization available in Appendix A.5 shows that DCRL still maintains clear inter-cluster boundaries even on the test samples, which demonstrates the great generalizability of our method.
Table 2: Generalizability evaluated by ACC/NMI.
Algorithms training samples
testing samples
AE+ -Means
0,815/0.736
0.751/0.711
DEC
0.841/0.773
0.748/0.706 704
IDEC
0.845/0.88 860
0.826/0.86 842
JULE
0.958/0.907
0.921/0.895
DSC
0.975/0.939
0.969/0.921
N2D
0.974/0.930
0.965/0.91 911
DCRL (ours)
0.978/0.941
0.978/0.941
4.2.3 CLUSTERING VISUALIZATION
The visualization of DCRL with several comparison methods is shown in Fig 4 (visualized using UMAP). From the perspective of clustering, our method is much better than the other methods. Among all methods, only DEC, IDEC, and DCRL can hold clear boundaries between different clusters, while the cluster boundaries of the other methods are indistinguishable. Although DEC and IDEC can successfully separate different clusters, they group many data points from different classes into the same cluster. Most importantly, due to the use of the clustering-oriented oriented loss, the embedding learned by algorithms such as DEC, IDEC, JULE, and DSC (especially DSC) tend to form spheres and disrupt the original topological structure. Instead, our method overcomes these problems and achieves almost perfect separation between different clusters while preserving the local and global structure.
Additionally, the embedding of latent space during the training process is visualized in Appendix A.6, which is highly consistent with the three-stage explanation mentioned in Sec 3.3.2, showing that clustering-orienter does indeed do deteriorate the local geometric structure the latent space, and designed LLIS helps recover it. In addition, in the above experiments, the cluster number C' is assumed to be a known prior (which is consistent with the assumptions almost all deep clustering algorithms). Therefore, we provide additional experiment to explore what happens when C' is larger than the number of true clusters. It is found that there exists splitting of the clusters, but the different categories still maintain clear boundaries and are not mixed together, somewhat similar to hierarchical clustering. See Appendix A.7 for detailed experimental settings and analysis.
(a) AE+X-Means
(b) DEC
(c) IDEC
(d) JULE
(e) DSC
(1 N2D
(g) DCRL (Ours)
Figure 4: Visualization of the embeddings learned by different algorithms on MNIST-full dataset.
8

---

Under review as a conference paper at ICLR 2021
4.3 EVALUATION OF MULTI-MANIFOLD REPRESENTATION LEARNING
Although numerous previous work has claimed that they brought clustering and representation learn- ing into unified framework, they all, unfortunately, lack an analysis of the effectiveness of the learned representations. In this paper, we compare DCRL with the other five methods in six evalu- ation metrics on six datasets. (Limited by space, only MNIST-full results are provided in the Tab and the complete results are in Appendix A.8). The results show that DCRL outperforms all other methods, especially the CRA metric, which is not only the best on all datasets but also reaches 1.0. This means that the "rank" between different manifolds in the latent space is completely preserved and undamaged, which proves the effectiveness of our global ranking loss Lyank-
Moreover, statistical analysis is performed in this paper show the extent to which local and global structure is preserved the latent space for each algorithm. Limited by space, they are placed in Appendix A.9. Furthermore, we also evaluated whether the learned representations are meaningful through downstream tasks, and this experiment is available in Appendix A.10.
Table 3: Performance for
representation learning.
Methods
RRE
Trust
Cont
d-RMSE
L.GD
CRA
DEC
0,099
0.844
0,948
44.85
4.379
0.28
IDEC
0.009
0.998
0.979
24.58
1.714
0.33
JULE
0.026
0.936
0.983
28.34
2.129
0.27
DSC
0.097
0.873
0.925
6.98
1.198
0.23
N2D
0,010
0.992
0.984
5.71
0.699
0.21
DCRL
0.005
0,999
0.987
5.49
0.691
1.00
4.4
ABLATION STUDY
This evaluates the effects of the loss terms and training strategies in the DCRL with five sets of ex- periments: the model without (A) Structure-oriented Loss (SL); (B) Clustering-oriented Loss (CL); (C) Weight Continuation (WC); (D) Alternating Training (AT), and (E) the full model. Limited by space, only MNIST-full results are provided in Tab 4, and results for the other four datasets are in Appendix A.11. After analyzing the results, we can conclude: (1) CL is the most important factor for obtaining good clustering, the lack of which leads to unsuccessful clustering, hence the numbers in the table not very meaningful and are shown in gray color. (2) SL not only brings subtle improvements in clustering but also greatly improves the performance of multi-manifold represen- tation learning. (3) Our elegant training strategies (WC and AT) both improve the performance of clustering and muntii-mniifol representation learning to some extent, especially on metrics such as RRE, Trust, Cont, and CRA.
Table 4: Ablation study of loss items and training strategies on MNIST-full dataset.
Datasets
Methods
ACC/NMI
RRE
Trust
Cont
d-RMSE
LGD
CRA
w/o SL
0.976/0.939
0.0093
0.9967
0,9816
24.589
1.6747
0.32
w/o CL
0.814/0.736
0.0004
0.9998
0.9990
7,458
0.0487
1.00
MNIST-full WC
0.977/0.943
0.0065
0.9987
0.9860
5.576
0.6968
0.98
w/o AT
0.978/0.944
0.0069
0.9986
0.9851
5.617
0.7037
0.96
full model
0.980/0.946
0.0056
0,9997
0.9871
5.498
0.6916
1.00
CONCLUSION
The proposed DCRL framework imposes clustering-oriented oriented and crructure-orienter constraints to optimize the latent space for simultaneously performing clustering and multi-manifold represen- tation learning with local and global structure preservation. Extensive experiments on image and vector datasets demonstrate that DCRL is not only comparable to the saate-of-the-ae deep clustering algorithms but also able to learn effective and robust manifold representation, which is beyond the capability of those clustering methods that only care about clustering accuracy. Future work will focus on the adaptive determination of manifolds (clusters) numbers and extend our work CNN architecture for large-scale datasets.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.0570%3 2020.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In IJCAI, pp. 1753-1759, 2017.
Xifeng Guo, Xinwang Liu, En Zhu, Xinzhong Zhu, Miaomiao Li, Xin Xu, and Jianping Yin. Adap- tive self-paced deep clustering with data augmentation. IEEE Transactions on Knowledge and Data Engineering, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
David Lewis, Yiming Yang, Tony G Rose, and Fan Li. Revl: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.
Stan Z. Li, Zelin Zhang, and Lirong Wu. WrXiv:2006.082.0 08256, 2020.
Markov-lipschttz deep learning. arXiv preprint
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov):557926603 2008.
J MacQueen. Some methods for classification and analysis of multivariate observations. In Proc. 5th Berkeley Symposium Math., Stat., and Prob, pp. 281, 1965.
Ryan McConville, Raul Santos- Roorio-, Robert J Piechocki. and Ian Craddock. N2d:(not too) deep clustering via clustering the local manifold of an autoencoded embedding, arXiv preprint arXiv: /908. 05968, 2019.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.0344 00300.0004262... 2018.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2855 2011.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2333-2336 2000.
Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. arXiv preprint arXiv:800.001887 2018.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-995 -00889905111 2000.
Joshua Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5550):2319-32233 2000.
10

---

Under review as a conference paper at ICLR 2021
Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie- Yan Liu. Learning deep representations for graph clustering. In Aaai, volume 14, pp. 1293-1299. Citeseer, 2014.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 11(12), 2010.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and
intelligent laboratory systems, ((1-3):37-52, 1987.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mann mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1705$ 077447 2017.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning, pp. 478-487, 2016.
Wei Xu, Xin Liu, and Yihong Gong. Document clustering based non-negative matrix factoriza- tion. Proceedings of the 26th anmual international ACM SIGIR conference on Research and development in informaion retrieval, pp. 267-273, 2003.
Bo Yang, Ming Xiang, and Yupei Zhang. Multi-manifold discriminant isomap for visualization and classification. Pattern Recognition, 55:215-230, 2016a.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2016b.
Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1866-4075,,
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clus- tering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Yan Zhang, Zhao Zhang, Jie Qin, Li Zhang, Bing Li, and Fanzhang Li. Semi-supervised local multi- manifold isomap by lincar embedding for feature extraction. Patter Recognition, 76:662-674 2018.

---

Under review as a conference paper at ICLR 2021
APPENDIX
A. GRADIENT DERIVATION
In the paper, we have emphasized time and again that { is a set of learnalbe parameters, which means that we can optimize it while optimizing the network parameter 0y. In Eq. (4) of the paper, we have presented the gradient of Leluster with respect to ki- In addition to Ldusstes both Trrank and aliggn are involving Py. Hence, the detailed derivations for the gradient of Lyank and Laaignn with respect to M4 are also provided. The gradient of Lrank with respect each learnalbe cluster center Hj can be computed as:
aLrank 0pg
2 EEE Ei=1 ldz (KV,Hy)) dx (o0)
c
0 dz (Wv, Myr K dx (u5², 105),
8pg
#=15'=1
The Euclidean metric is used for both the input space and the hidden layer space, i.e., dz (Jzz, (ly) liue - My ||. In addition, the symbols are somewhat abused for clear derivation, representing K dx (1r) oz) with K. Accordingly, Eq. (1I) can be further derived as follows:
Ilronk 8pl
C
C
2 ldz (mv, x dx (Vz) , v
8pg
#=1 5'=1
C 0
Ilue -Mzill - K
Öpt
/=1
0 || -Myll - K
2 11a -1yll K
Öpt
y'=
1 (11m - -Myll - K)
11 - -1g11 || - K
014
11 - Myl K
1 (11mg My - K)
|14 M4 K
1145 - M3 |1 K
=1=
3 11 -Wll
1
M1l - K
[[|p M1ll K
1 111g - Myrl |1
|lag 1lr1 || K
[1ag pez
K
M4 M4
|1g M4 ||
K
|| 11 K
||M - M4
[14 - M4 - K
|1 Myl
[14 - -,y1 | K
|ug 14 || K
2
|114 Mel
[14 - M4
K
7=1
K dx (uzx u.x)
Σ
M4 - 142
[1u4 - |4l
|14 - Mel ||
K. dx (u:), 015))
dz (1z, [p)
dx (u),ee 1z²) - -x dx (v), uz))
Σ
Mj p4
dz (M5,Hw)
ddz (1), mv)
12

---

Under review as a conference paper at ICLR 2021
The gradient of Lalign with respect to each learnalbe cluster center My be computed as:
Lalion
EE 5'=1
11as
1
Opl
Opt
ply 11/11
Öpe
p=1
allky 0/11
Op.
0(y - uf)
Pi uz
Jpg
11g 0z
11 oz
A.2 ALGORITHM
Algorithm Algorithm for Deep Clustering and Representation Learning
Input:
Input samples: X; Number of clusters: C; Number of batches: B; Number of iterations: E.
Output:
Autoencoder's weights: 0y and Og: Cluster labels [s:] Trainable cluster centers (μ5) 1: Initialize the weight {ny}, f-1
1- fy and fg. and obtain initialized soft label assignment {st]
for epoch E FF. do
3: Compute embedded points ) and distribution Q:
Update target distribution P:
5: Compute soft cluster centers {0," x C
Jim1 and (v]" y
6: for batch (O.1.....13 .B) do
Pick up one batch of samples Xhatch from X; Compute corresponding distribution Quarch and it's reconstruction Yaateh Pick target distribution batch Poutch from P:
8:
9:
10:
Compute loss Laer Leluster and Lranki
11:
Update the weight Oy. 0g and {15) end for
12:
13:
Compute Tusso and Traiign on all samples; Update the weight 0, and (pz)g
14:
15:
Assign new soft labels (s.h
16: end for
17: return Dy, Og. ( (15)
A.3 DATASETS
To show that our method works well with various kinds of datasets, we choose the following six image and vector datasets. Some example images shown in Fig A1, and the brief descriptions of the datasets are given in Tab A1.
MNIST-full (LeCun et al., 1998): The MNIST-full dataset consists of 70,000 handwritten digits of 28 28 pixels. Each gray image is reshaped to 784-dimensional vector.
MNIST-test (LeCun et al., 1998): The MNIST-test is the testing part of the MNIST dataset, which contains a total of 10000 samples.
USPS 3; The USPS dataset is composed of 9298 gray-scale handwritten digit images with size of 16x16 pixels.
13

---

Under review as a conference paper at ICLR 2021
Table Al: Description of Datasets.
Dataset
Samples
Categories
Data Size
MNIST-full
70000
10
28x28x
MNIST-test
10000
10
28x28x x
USPS
9298
10
16x 16x
Fashion- MNIIT
70000
10
28x28 x
REUTERS-IOK
10000
4
2000
HAR
10299
6
561
Fashion-MNIST
(Xiao et al., 2017): This Fashion-MNIST dataset has the same number of images and the same image size as MNIST-full, but it is fairly more complicated. Instead of digits, it consists of various types of fashion products.
REUTERS-IOK REUTERS (Lewis et al., 2004) is composed of around 810000 English stories labeled with category tree. Four root categories (corporate/industraa, gov- ernment/social, markets, and economics) are used as labels and excluded all documents with multiple labels. Following DEC (Xie al., 2016), subset of 10000 examples are randomly sampled, and the tf-idf features on the 2000 most frequent words are computed. The sampled dataset is denoted REUTERS-IOK.
HAR: HAR is time series dataset consisting of 10,299 sensor samples from a smartphone. It was collected from people performing six different activities: walking, walking up- stairs, walking downstairs, sitting, standing, and laying.
O /
0 9
MNIST
0
33456789 9
(b) USPS
(c) Beshion-MNISS
Figure A1: The image samples from three datasets (MNIST, USPS, and Fashion-MNIST)
A.4 DEFINITIONS OF PERFORMANCE METRICS
The following notations are used for the definitions:
dx (i, the pairwise distance between x; and æy in input space X;
dz (i, j): the pairwise distance between 2; and æj in latent space Z;
N N the of indices the k-nearest neighbor (kNN) of æ; in input space X;
Arh:2 the set of indices to the k-nearest neighbor (kNN) of 2; in latent space Z;
Tx (i; the rank of the closeness (in Euclidean distance) of xj to æ; in input space X;
rz (i: the rank of the closeness (in Euclidean distance) of 2j to in latent space Z. The eight evaluation metrics are defined below:
(1) ACC (Accuracy) measures the accuracy of clustering:
MN
(4 = m (8:))
ACC max
N
14

---

Under review as a conference paper at ICLR 2021
where 1 and 8; the true and predicted labels for data point z,, respectively, and m².)) is all possible one-to-one mappings between clusters and label categories.
(2) NMI (Normalized Mutual Information) NMI calculates the normalized measure of simi- larity between two labels of the same data
T(1:8)
NMT
max (1), H(s)
where s)) is the mutual information between the real label and predicted label and (-) represents their entropy.
(3) RRE (Relative Rank Change) measures the average of changes in neighbor ranking be- tween two spaces X and Z:
RRE
{M + M
(k2
ki
+
k=k
where kn and ly are the lower and upper bounds of the k-NN
[rx(i,j) rz (i, rz(1,5)
MR Hi
JEA
(rx(i,)) - (i(
MR +x
H Σ Σ
(x(1,))
1=1
JENN
where Hy is the normalizing term, defined as
H N N 27
(4) Trust (Trustworthiness) measures to what extent the k nearest neighbors of point preserved when going from the input space to the latent space:
N Σ Σ
Trust
(rx(i.)) k)
k2
k1
NA(2N 3/ 1) 1=1
km
JEN, JENN.
where k and 22 are the bounds of the number of nearest neighbors.
(5) Cont (Continuity) is defined analogously to Trust, but checks to what extent neighbors are preserved when going from the latent space to the input space:
Cont
2
Σ
(rz(i,)) - k)
h
h
Nk(2N 3k
JENK 2 JEAE..
where kn and 22 are the bounds of the number of nearest neighbors.
(6) d-RMSE (Root Mean Square Error) measures what extent the distributions of dis- tances coincide:
N
N
d RMSE
ΣV
Σ
N2
15

---

Under review as a conference paper at ICLR 2021
(7) LGD (Locally Geometric Distortion) measures how much corresponding distances be- tween neighboring points differ in two metric spaces and is the primary metric for isometry, defined as:
kg
Σ sextias (di(i,)) - du (i.
LGD
(k2 Ki + M(#N.)
k=k1
where 1 and h2 are the lower and upper bounds of the k-NN (8) CRA (Cluster Rank Accuracy) measures the changes in ranks of cluster centers from the input space X and the latent space Z:
CRA Ei 1(xx(1X,00) -x)) r2(f 1
C
where C' is the number of clusters, v/ is the cluster center of the jth cluster in the input space X, vz the cluster center of the jth cluster in the latent space Z, rx (vz u denotes the rank of closeness (in terms of Euclidean distance) of vz* to vy in space X in the input space X, and (" u denotes the rank of the closeness (in terms of Euclidean distance) of ou to o in space Z.
A.5 VISUALIZATION IN GENERALIZABILITY
The visualization results on the testing samples are shown in Fig A2; even for testing samples, our method still shows distinguishable inter-cluster discriminability, while all the other methods without exception coupled different clusters together.
(a) A-+a-ss
(b) DEC
(c) IDEC
(d) JULE
DSC
(f) N2D
(g) DCRL (ours)
Figure A2: The visualization of the obtained embeddings on the testing samples to show the gener- alization performance of different algorithms on MNIST-full dateset.
A.6 VISUALIZATION IN DIFFERENT STAGES
The embedding visualization of the latent space during the training process is visualized in Fig A3 for depicting how both clustering and structure- preeuurrrppeeeiiigg.g. is achieved. We can see that the different closely adjacent. In the early stage of training,
clusters initialized by pretrained autoencoder with clustering loss Latuster and global ranking loss Lyank- different manifolds are separated from each other, each manifold loses its local structure, and all of them degenerate into spheres. As
16

---

Under review as a conference paper at ICLR 2021
the training progresses, the weight a for Latuster gradually decreases, while the weight 3 for Liso increases and the optimization gradually focused from global to local, with each manifold gradually recovering its original geometric structure from the sphere. Moreover, since our local isometry loss Liso is constrained within each manifold, the preservation of local structure will not disrupt the global ranking. Finally, obtain representations in which cluster boundaries are clearly distinguished, and local and global structures are perfectly preserved. This shows that clustering- oriented loss does deteriorate the local geometric structure of the latent space, and designed LLrs helps to recover it.
(a) Epoch
(h) Epoch 9
(c) Epoch 19
(d) Epoch 29
(c) Epoch 69
(f) Epoch 119
(g) Epoch 159
(h) Epoch 209
(i) Epoch 249
i Epoch 299
Figure A3: Clustering visualization at different stages of training on MNIST-full dateset.
A.7 EXPLORATION ON THE ASSUMED CLUSTER NUMBER C
Taking MNIST-test dataset as an example, present the embedding visualization with assumed number of clusters C being 10, 11, and 12, respectively. We find that when C is larger than the number of true clusters (10), data originally belonging the same cluster will be split, e.g., cluster is split into two, but the different categories of data still hold clear boundaries and are not mixed together, somewhat similar hierarchical clustering.
(a) C=10
(b) C=11
(c) C=12
Figure A4: Clustering visualization with different assumed cluster number C MNIST-test dateset.
17

---

Under review as a conference paper at ICLR 2021
A.8 QUANTITATIVE EVALUATION OF REPRESENTATION LEARNING
Our method is compared with the other five methods in six evaluation metrics on six datasets. The complete results in Tab A2 demonstrate the superiority of method, especially metrics RRE, Trust, Cont, and CRA. As shown in Tab A2, DCRL outperforms all other methods, especially in only the best on all datasets but also reaches 1.0. This means that
the CRA metric, which the "rank" between different manifolds in the latent space is completely preserved and undamaged, which proves the effectiveness of our global ranking loss Irank-
Table A2: Representation learning performance of different algorithms on five datasets.
Datasets
Algorithms
RRE
Trust
Cont
d-RMSE
LGD
CRA
DEC
0.09988
0.84499
0,94805
44.8535
4.37986
0.28
IDEC
0.00984
0.99821
0.97936
24.5803
1.71484
0.33
JULE
0.02657
0,93675
0,98321
28.3412
2.12955
0.27
MNIST-full
DSC
0.09785
0.87315
0,92508
6.98098
1.19886
0.23
N2D
0.01002
0.99243
0,98466
5.7162
0.69946
0.21
DCRL
0.00567
0.99978
0.98716
5,4986
0.69168
1.0
DEC
0.12800
0.81841
0.91767
14,6113
2.29499
0.19
IDEC
0.01505
0,99403
0,97082
7,4599
1,08350
0.38
JULE
0.04122
0.92971
0,97208
9,4768
1.17176
0.42
MNIST-test
DSC
0.10728
0.85498
0.92254
7.1689
1.19239
0.26
N2D
0,01565
0.98764
0.97572
5.0120
0.97454
0.33
DCRL
0.01090
0.99811
0,97612
5.8000
0,93394
1.0
DEC
0.07911
0,88871
0,94628
16.4355
1,77848
0.31
IDEC
0.01043
0.99726
0.97960
13,0573
1.11689
0.30
JULE
0.02972
0.98763
0.98810
14.6324
1.43426
0.33
USPS
DSC
0.06319
0.9151
0.93988
8.4412
1.02131
0.27
N2D
0.01337
0.98769
0.98135
8.1961
0.54967
0.37
DCRL
0.00577
0.99979
0.98701
6,4980
0.53180
1.0
DEC
0.04787
0.93896
0.95450
39.3274
3.87731
0.37
IDEC
0.01089
0.99683
0.97797
25.4024
1.91385
0.27
JULE
0,03013
0,97732
0.97923
15.2213
1,43642
0.43
Fasion-MNIST
DSC
0,05168
0.95013
0.96121
17.2201
1.42091
0.36
N2D
0.00894
0.99062
0.98054
14.49079
1.28180
0.26
DCRL
0.00836
0.99868
0.98203
13.3788
1,33893
1.0
DEC
0.26192
0.65518
0,80477
40.4671
4,00423
0.63
IDEC
0.05981
0.95840
0.90550
43.9556
2.01365
0.75
JULE
REUTERS-IOK
DSC
N2D
0.03827
0,97385
0,93412
36,1042
1.69013
0.31
DCRL (ours)
0.03206
0,98380
0,93802
34.5478
2.72096
1.0
DEC
0.09060
0.89097
0.91766
10.0222
1,58691
0.30
IDEC
0.01031
0,99433
0,98132
9,9155
0,93736
0.39
JULE
HAR
DSC
N2D
0.00841
0.99281
0.97695
8.2326
0.64296
0.33
DCRL (ours)
0.00665
0.99895
0.98634
15.2876
0.46189
18

---

Under review as a conference paper at ICLR 2021
A.9 STATISTICAL ANALYSIS
The statistical analysis is presented to show the extent to which local and global structure is pre- served from the input space the latent space. Taking MNIST-full as an example, the statistical analysis of the global rank-preservation is shown in Fig A5 (a)-(f). For the i-th cluster, if the rank (in terms of Euclidean distance) between it and the j-th cluster preserved from imput space to latent space, then the grid in the i-th row and j-th column is marked as blue, otherwise yellow. As shown in the figure, only our method can fully preserve the global rank between different clusters, while all other methods fail.
Finally, we perform statistical analysis for the local isometry property of each algorithm. For each sample in the dataset, it forms a number of point pairs with its neighborhood samples {(rs; = 2, N; E EN*). We compute the difference the distance of these point pairs from the imput space to the latent space {dz(z,, Zz, dx (Iz, 2y)li 1,2, .... 2; E .M,, and plot as a histogram. As shown Fig A5 (g), the curves DCRL are distributed on both sides of the 0 value, with maximum peak height and minimum peak-botton width, respectively, which indicates that DCRL achieves the best local isometry. Although IDEC claims that they can preserve the local structure well, there is still big gap between their results and ours.
DEC
IDEC
1.0
JULE
1.0
DSC
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
04
0.2
0.2
02
51
5 6
0.0
0.0
2
000
Rank
Rank
Rank
Rank
(a) DEC
(b) IDEC
(c) JULE
(d) DSC
N2D
1.0
DCRL
Distance Histogram
25000
0.8
IDEC
0.6
BALE
15000
0.4
0.4
N2D
0.2
0.2
5000
2
0.0
0
0.9
0.33
-0.2
-0.1
0.0 ADIs
0.1
0.2
0.3
Rank
Rank
(e) N2D
(I) DCRL
(g) Local Isometry
Figure A5: Statistical analysis of different algorithms compare the capability of global and local structure preservation from the input space to the latent space.
19

---

Under review as a conference paper at ICLR 2021
A.10 QUANTITATIVE EVALUATION OF DOWNSTREAM TASKS
Numerous deep clustering algorithms have recently claimed to obtain meaningful representations, however, they do not analyze and experiment with the so-called "meaningful" ones. Therefore, we interested in whether these proposed methods can indeed learn representations that useful for downstream tasks. Four different classifiers, including a linear classifier (Logistic Regression; LR), Iwo nonlinear classifiers (MLP, SVM), and tree-based classifier (Random Forest Classifier; RFC) are used as downstream tasks, all of which use default parameters and default implementations in sklearn (Pedregosa et al., 2011) for fair comparison. The learned representations are frozen and used as input for training. The classification accuracy evaluated on the test set serves as metric to evaluate the effectiveness of learned representations. In Tab A3, DCRL outperformed the other methods overall on all six datasets, with MIP, RFC, and LR as downstream tasks. Additionally, we surprisingly find that with MLP and RFC as downstream tasks, all methods other than DCRL do not even match the accuracy of AE on the MNIST-full dataset. Notably, DEC and IDEC show a sharp deterioration performance on downstream tasks, falling short of even the simplest AEs, again showing that clustering-orientee loss disrupt the geometry of the data.
Table A3: Performance of different algorithms in downstream tasks.
Datasets
Algorithms MLP
RFC
SVM
LR
AE
0.9746
0.9652
0,9859
0.9565
DEC
0.8647
0.8706
0.8707
0.8566
IDEC
0.9797
0.9737
0.9852
0,9650
MNIST-full
JULE
0.9802
0.9825
0.9787
0,9743
DSC
0,9622
0.9501
0.9837
0.9752
N2D
0.9796
0.9803
0.9799
0.9792
DCRL
0.9851
0.9874
0.9869
0.9841
AE
0.9415
0.9420
0.9745
0,9495
DEC
0.8525
0.8605
0.8725
0.8685
IDEC
0.9740
0.9725
0.9845
0.9655
MNIST-test
JULE
0.9775
0.9845
0.9800
0.9825
DSC
0.9535
0.9740
0.9825
0,9795
N2D
0.9715
0.9760
0.9725
0.9725
DCRL
0.9855
0.9875
0.9865
0.9855
AE
0.9421
0.9469
0.9677
0.9073
DEC
0.8289
0.8668
0.8289
0.8294
IDEC
0.9482
0.9556
0.9656
0.9125
USPS
JULE
0.9576
0.9617
0.9703
0.9476
DSC
0.9351
0.9572
0.9612
0.9342
N2D
0.9569
0.9569
0.9569
0.9541
DCRL
0.9656
0.9651
0.9604
0.9551
AE
0.8613
0,9932
0,8314
0.7588
DEC
0.6268
0.9853
0.6377
0.6245
IDEC
0.8367
0.9918
0.8607
0.7514
Easion-MNIST
JULE
0.8541
0,9892
0.8566
0.7723
DSC
0.8084
0,9823
0,8618
0.7676
N2D
0.8412
0.9493
0.8230
0.7753
DCRL
0.8642
0.9942
0.8468
0.7768
AE
0.9325
0,9170
0,9375
0.8205
DEC
0.7985
0.7880
0.8105
0.7450
IDEC
0.9225
0.8930
0.9280
0.7705
REUTERS-
JULE
DSC
N2D
0.9205
0.9080
0,9240
0.8335
DCRL (ours)
0.9360
0.9185
0.9390
0.8475
AE 0.9181
0.9139
0.9201
0.8849
DEC
0.7696
0.7847
0.7628
0.7634
IDEC
0.8973
0.9031
0,9041
0.8822
HAR
JULE
DSC
N2D
0.9138
0.9083
0,9174
0.8799
DCRL (ours)
0.9235
0,9193
0.9293
0.8996
20

---

Under review as a conference paper at ICLR 2021
A.11 MORE ABLATION EXPERIMENTS
The results of the ablation experiments on the MNIST-full dataset have been presented in Tab 4 in Sec 4.3. Here, provide four more of ablation experiments the other four datasets. The conclusion is similar (note that the clustering performance of the model without clastering-oriented losses is very poorly, so the "best" metric numbers are not meaningful and are shown in gray color): (I) CL is very important for obtaining good clustering. (2) SL is beneficial for both clustering and representation learning.
(3) Our training strategies (WC and AT) are very superior in improving metrics such as ACC, RRE, Trust, Cont, and CRA.
Table A4: Ablation study of loss items and training strategies used in DCRL.
Datasets
Methods
ACC/NMI
RRE
Trust
Cont
d-RMSE
LGD
CRA
w/o SL
0.976/0.939
0.0093
0.9967
0.9816
24.589
1.6747
0.32
w/o CL.
0.814/0.736
0.0004
0.9998
0.9990
7.458
0.0487
1.00
MNIST-full
w/o WC
0.977/0.943
0,0065
0.9987
0.9860
5,576
0.6968
0,98
w/o AT
0.978/0. 944
0.0069
0.9986
0.9851
5.617
0.7037
0.96
full model
0.980/0.946
0.0056
0.9997
0.9871
5,498
0.6916
1.00
w/o SL
0.973/0.932
0.0146
0,9928
0.9727
7,701
1,0578
0.31
w/o CL
0,773/0. 747
0.0020
0.9994
0,9954
7.229
0,0809
1.00
MNIST-test
w/o WC
0,956/0. 904
0.0132
0.9955
0.9735
5,470
0,9364
1.00
w/o AT
0.970/0.929
0.0118
0.9974
0,9747
5.567
0.9404
1.00
full model
0.972/0.930
0.0109
0.9981
0,9761
5.800
0.9339
1.00
w/o SL.
0.958/0.902
0.0095
0.9967
0.9812
14.609
0,9847
0.29
w/o CL.
0.664/0.658
0.0020
0.9996
0,9952
2.934
0,0687
1.0
USPS
w/o WC
0.956/0.896
0.0060
0.9991
0.9868
6.572
0.5335
1.00
AT
0.947/0.885
0.0080
0.9979
0,9833
5.960
0.4967
1.00
full model
0.960/0.902
0.0057
0.9997
0.9870
6.498
0,5318
1.00
w/o SL.
0.706/0.682
0.0T08
0.9964
0.9781
25.954
1.8936
0.30
wo CL
0.576/0.569
0,0004
0.9994
0,9995
7,654
0.0523
1.00
Fasion-MNIST
WC
0.702/0.695
0.0084
0.9972
0.9814
13.238
1.3474
1.00
w/o AT
0.708/0.694
0.0097
0.9975
0.9798
13.354
1.3611
1.00
full model
0.710/0.685
0.0083
0.9986
0.9820
13.378
1.3389
1.00
w/o SL
0.819/0.564
0.0529
0.9610
0,9185
44,481
1,9090
0.38
CL
0.542/0.279
0.0277
0.9868
0,9456
37.018
2,2294
1.00
REUTERS-IOK
w/o WC
0.830/0.583
0.0420
0.9667
0.9361
35.302
2,8286
1.00
w/o AT
0,825/0.563
0,0440
0.9650
0,9330
39,275
2,9146
1.00
full model
0.836/0.590
0.0320
0.9838
0.9380
34.547
2.7209
1.00
SL
0.835/0.
0.0116
0.9944
0.9792
8.168
0.8882
0.33
w/o CL
0.744/0.6 615
0.0024
0.9986
0.9948
15.060
0,2193
1.00
HAR
w/o WC
0.786/0.701
0.0130
0.9950
0.9756
15,398
0.6171
1.00
w/o AT
0.834/0. 745
0.0089
0.9965
0,9835
15.726
0.4734
1.00
full model
0.845/0.758
0.0066
0.9989
0.9863
15.287
0.4618
1.00
21

---

Under review as a conference paper at ICLR 2021
A.12 PARAMETER SENSITIVITY
We also evaluated the sensitivity of parameters k: and k. on the MNIST-test dataset and the results shown in Tab A5. The parameters and found have little effect the clustering performance (ACC/NMI, and some combinations of and x even produce better clustering perfor- mance than the metrics reported the main paper. However, the effect of and on representation learning is more pronounced, and different combinations of & and к may increase or decrease per- formance. general, this paper focuses on the design of the algorithm itself and has not performed the parameter search to find the best performance.
Table A5: Parameter Sensitivity with different parameters l and on the MNIST-test dataset.
Parameters
ACC/NMI
RRE
Trust
Cont
d-RMSE
LGD
CRA
k=1, k=3
0.975/0.936
0.0125
0.9944
0.9756
5.757
0.8868
1.00
k= k=3
0.973/0.931
0.0114
0.9970
0.9757
5.805
0.9207
1.00
k=5, x=3
0.972/0.930
0.0109
0.9981
0.9761
5.800
0.9339
1.00
k=8, k=3
1.972/0.929
0.0104
0.9989
0.9765
5.810
0.9476
1.00
k=10, k=3
0.972/0.929
0.0105
0.9990
0.9764
5.704
0.9487
1.00
k=5, x=1
0.967/0.912
0.0068
0.9993
0.9845
5.409
0.2524
1.00
k=5, k=3
0.972/0.930
0.0109
0.9981
0.9761
5,800
0.9339
1.00
k=5, k=5
0.972/0.929
0.0146
0.9964
0.9691
15.0653
1.5719
1.00
k=5, k=8
0.972/0.929
0.0190
0.9943
0.9615
29.4607
2.5410
1.00
k=5, x=10
0.972/0.929
0.0195
0.9951
0.9597
37.7661
3.1434
1.00
22


Reviews:
[{"review_id": "lRvy-Ra4oUL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method which can simultaneously perform clustering and represent learning with local and global structure preservation. The method imposes clustering-oriented and structure-oriented in order to optimize the latent space. The experimental results show its effectiveness. However, here are some issues.\n(1) This paper is not innovative enough. From the perspective of the entire clustering process, the proposed clustering method essentially makes the intra-cluster distance as small as possible and the inter-cluster distance as large as possible. There is no significant innovation in the idea of clustering.\n(2) In the initialization of the algorithm, first use t-SNE to transform the latent space Z into two dimensions, and then run K-Means to get the label of each data point. In this step, why do you need to perform dimension conversion first instead of directly performing K-Means clustering on the original data?\n(3) In this method, the author uses K-Means for initialization. However, K-Means is not a stable clustering algorithm. Using it to initialize the label, are the results stable? Has the author analyzed the effect of different label initialization methods on the results?\n(4) The data sets used in the experiment are too single. In the adopted data sets, three are all about MNIST, and there is little difference between MNIST-test and MNIST-full. Moreover, similar to MNIST, USPS is also a handwritten digital image data set. The data sets used by the author are repetitive, and it is difficult to reflect the experimental performance of the method.\n(5) From the quantitative evaluation results of downstream tasks, compared with the classic classification algorithms, the representation learning obtained by the proposed method for classification does not show great advantages in classification accuracy.\nOverall, I think this paper is not ready yet.", "extra_scores": {}, "extra_text": {}}, {"review_id": "igFHxUs777p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contributions of this work are the inclusion of some new optimization objectives into the standard deep clustering formulations.\nThe cluster orientated loss seems to come from DEC but as the authors point out this causes issues that affect the quality of clusters.\nThus to preserve the manifolds within the embedding the propose new objectives in addition to this. One objective tries to ensure representation clusters are internally coherent with the original manifold Another objective tries to ensure global structure is preserved by optimizing directly the centroids of the learned representation \n\nThere are a number of strengths to this work. \n\nThe idea of including the geometric structure is well founded and their approach appears to account for this well. Unlike the typical work in this area, the authors choose to evaluate their representations on downstream tasks, and not just clustering, included. This really demonstrates the benefits (and limitations) outside of the typical clustering focus. I also liked the inclusion of the ablation study which shows benefit of Weight Graduality (WG) and Alternating Training (AT). They also perform sufficient experimentation to validate their approach.\n\nOne concern w.r.t. there proposed approach is that one of their objective functions only updates the centroids and thus they need another objective to align these. This seems like a rather arbitrary way of going about this. \n\nThe authors also state that \"It is worth noting that all of the above losses coexist rather than independently at different stages\" However, the final stage corresponds to L_LIS corresponds outside of the batching, which is a little misleading. I would not say they are not independent in this case.\n\nI have some questions w.r.t. the evaluation.. \nWhy is tSNE used after pretraining when the work of N2D (which you reference) shows UMAP performs better? This is more of a question about the authors choice, rather that a criticism of the work. \nI also am unclear why they reduce the dimensionality to 2. This is intuitive when you are visualizing the dataset, but not necessarily for clustering. The above paper N2D (which you reference) mentions that the cluster quality lessens with 2 dimensions as opposed to higher.\nAgain, despite these choices, the method outperforms others, thus these are less a direct criticism of the work and more about the lack of justification these choices. \n\nI believe the authors are missing some other relevant work in their comparisons, for e.g. [1], which reports higher performance than the authors on some datasets.\n\nFurther, like much of these works, the method is only tested on smaller datasets. Other approaches are tested on datasets such as ImageNet-10, the inclusion of more larger image datasets would strengthen this work. \n\n[1] Guo, Xifeng, et al. \"Adaptive self-paced deep clustering with data augmentation.\" IEEE Transactions on Knowledge and Data Engineering (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "YScr21rxUJr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a clustering methods that builds upon DEC and N2D+tSNE with extra loss terms to preserve geometric structures. They provide an extensive analysis but not always fair. In the end, the benefits of \"preserving the geometric structures\" are not clear or well defended.\n\nOverall, the paper is well written and easy to follow.\nThe authors provide arguments to several aspects of their contribution. However it is also scattered with claims.\nThe maths are clear despite some atypical notations (eg. the scale parameter).\n\n1/The introduction raises well the problem of \"learning a representation that favors clustering\" (please use \\emph instead of bold).\nConversely, the paragraph claiming that we need a clustering that also favors representation learning is not clear. From this point of view, it appears to me to be two completely different things.\n\n2/ The second section reviews the most important and related works in the field of deep clustering. \nI would rather describe JULE as a neural extension of hierarchical clustering.\nSince I am not an expert in manifold representation, several abbreviation remains unclear to me: LIS, MMD, MLDL, LLE, SSMM.\n\n3/ Since you wish to learn a representation and clustering, you aim to find the optimal \\theta AND \\mu.\n3.1/ The derivations here are the same as DEC (citation), except that you rightfully name the underlying distribution.\n\"However, we find that the clustering-oriented loss may deteriorate the geometric structure of the latent space\"\nThis claim is not justified. \n\n3.2/ Eq 5. the multiplication dot is unnecessary and disturbing.\nEq 6. The \"scale\" parameter is very disturbing. \nI understand \"rank\" as the number of clusters. This term ensures that the distance between the manifolds centers in the input space and between the centroids in the features space remain the same/proportional. The separation of the clustering centers depends on their initialization. I wonder what happens when C is larger than the number of true clusters. \n\"so L rank is easier to optimize, faster to process, and more accurate.\" Faster to process, yes. However easier and more accurate require more argument. It is not clear relatively to what is it more accurate. Moreover, the conservation of the distances imposed by L_rank  appears to me more difficult to satisfy, especially since the euclidean distance is used in both spaces. \n\nThe alignment loss replaces the update of the \\mu in DEC.\n\n3.3 It is clear and well explained. Although, Fig 2 and its caption could be more informative and self-contained.\n\n4. The experiments involves several baselines and varied data-sets. However, the experimental setting is not clear and weights a lot in my final opinion. \nDo you report best score? Average? What about the baselines? DCLR and N2D reports very similar scores. Are they statistically different? This apply to all the tables.\nIn the same way that (I)DEC is very dependent on the AE initialization, the results (DCLR vs N2D) suggest the same for the propose method. \n\n4.3.1 Table 3: Most of these measure check if neighbors on the input are also neighbors in the feature space.\nIs RMSE relevant since the f is non linear? Note that for LGD, since it stays in the neighborhood, it is not such a big deal.\nWhy is a high CRA desirable? Besides if DCLR did not report a very high value, that would  suggests that L_rank fails.\n\n4.3.2 \"Significantly, the performance of DEC on downstream tasks deteriorates sharply and even shows a large gap with the simplest AEs, which once again shows that the clustering-oriented loss may damage the data geometric structure.\"\nA direct competitor to DCLR is rather IDEC than DEC because of the missing reconstruction loss.\nWithout statistical test, the differences are not interpretable (SVM).\n\n4.4 Here again, and especially for the last three lines of Table 5, the missing statistical test prevent us from drawing any conclusion.\n\nA.7 It should be say more clearly that the rank of j relatively to i is in terms of distance. It is also related to CRA.\nN2D and DCLR report very similar scores on MNIST (Table 1), although the curves on Fig A.3.g are shifted. Hence the argument on the benefit of having a centered curves does not hold.\n\nAs a summary, the paper is well written but includes several claims that are not justified. The experimental section is not reproducible (best score? average? same init for all the baselines?) and miss statistical tests. \nIndeed some geometric aspects are preserved by the projection, but are relevant for clustering? or classification?\nBesides, the benefits over N2D do not appear clear to me. It seams that the method depends on its initialization. I wonder how (I)DEC would perform with the same initialization.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8yxc--Jisi0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nIn this paper, the authors proposes a deep clustering model to enable the clustering and representation learning to favor each other via preserving the geometric structure of data. The proposed DCRL framework integrates an isometric loss for local intra-manifold structure and a ranking loss for global inter-manifold structure. The authors evaluate the proposed framework on five datasets and the experimental results show that the proposed framework brings certain improvements over the baseline approaches. \nGenerally speaking, it is a good work with good results. Considering there are some weaknesses to be solved, I’d like to vote for a rating of 6.\n\n[Pros]\n+ The paper is well organized and easy to follow.\n+ The idea of preserving global and local structure is reasonable and seems to be effective.\n+ The description of experimental setting is sufficient, should be easy to reproduce. The authors also conduct comprehensive experiments with good results achieved.\n\n[Cons]\n- The proposed intra-manifold isometry loss and inter-manifold ranking loss seem to be modifications of $L_{LIS}$ proposed in [1] so the framework is interesting but not surprisingly new to me.\n- Though the authors claim that ‘the clustering-oriented loss may deteriorate the geometric structure of the latent space’ at the very beginning of this paper, this claim is not supported by any proof until it reaches the end of the experiment section. It would be better to illustrate this phenomenon by some figures or case studies in the introduction to help the readers understand this. Moreover, the relationship between geometric structure and the clustering performance is also recommended to be analyzed to show the necessity of preserving the local data structure.\n- In this paper, the meaningfulness of learned representations is tested on downstream classification tasks. However, it seems not very reasonable to adopt such evaluation, since intuitively the representation learning should focus on improving the clustering performance and it seems not very necessary to take the downstream task into account for the clustering models (at least for me it is not very practical to use representations learned by deep clustering models to do some classifications).\n- Experiments are only conducted only with MLP-AE backbones. It is not clear whether the data structure will be preserved for ConvAEs which is also commonly used in real-world applications.\n- There are some minor issues, e.g., 'learining' in page 2 and page 17 should be 'learning'. In Fig.4, ‘AE+K-mean’ should be ‘AE-K-means’.\n[1] Stan Z Li, Zelin Zhang, and Lirong Wu. Markov-lipschitz deep learning. arXiv preprint arXiv:2006.08256, 2020.\n\n[Questions]\n1. Could you please provide some explanation for why the clustering-oriented loss may deteriorate the geometric structure of the latent space and why preserving the structure is necessary for the clustering?\n2. Will the data structure be preserved for ConvAEs backbones?\n3. How do the hyperparameters $k$ and $scale$ affect the performance?\n4. How to set the the range of hyperparameters $\\alpha, \\beta$ in practice?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: During discussion, the reviewers acknowledge improvement of the revised version of the paper through author rebuttal and agree with that the paper is overall well written.

However, the novelty of the paper is not strong, and reviewers share the concern that distinction between self supervised learning and deep clustering is not convincing. Also, the concerns (2) raised by the Reviewer #2 is important, which is about the effectiveness of the proposed two-step procedure: first apply t-SNE to transform into two dimensions, followed by running K-means, while the answer is not well  justified. In my opinion, parameter sensitivity should be studied more carefully. The authors mention that $k$ and $\kappa$ have little effect on clustering performance, while this could imply that the associated terms do not have impact on the proposed method. This point should be clarified further.

Overall, the paper is still not ready for publication, I will therefore reject the paper.
Decision: Reject