Year: 2020

Abstract:
In this paper, we propose a novel framework for Deep Clustering and multimanifold Representation Learning (DCRL) that preserves the geometric structure of data. In the proposed DCRL framework, manifold clustering is done in the latent space guided by a clustering loss. To overcome the problem that clusteringoriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving intra-manifold structure locally and a ranking loss for inter-manifold structure globally. Experimental results on various datasets show that the DCRL framework leads to performances comparable to current state-of-the-art deep clustering algorithms, yet exhibits superior performance for manifold representation. Our results also demonstrate the importance and effectiveness of the proposed losses in preserving geometric structure in terms of visualization and performance metrics. The code is provided in the Supplementary Material.

text: In this paper, we propose a novel framework for Deep Clustering and multimanifold Representation Learning (DCRL) that preserves the geometric structure of data. In the proposed DCRL framework, manifold clustering is done in the latent space guided by a clustering loss. To overcome the problem that clusteringoriented losses may deteriorate the geometric structure of embeddings in the latent space, an isometric loss is proposed for preserving intra-manifold structure locally and a ranking loss for inter-manifold structure globally. Experimental results on various datasets show that the DCRL framework leads to performances comparable to current state-of-the-art deep clustering algorithms, yet exhibits superior performance for manifold representation. Our results also demonstrate the importance and effectiveness of the proposed losses in preserving geometric structure in terms of visualization and performance metrics. The code is provided in the Supplementary Material.

heading: 1 INTRODUCTION
text: Clustering, a fundamental tool for data analysis and visualization, has been an essential research topic in data science and machine learning. Conventional clustering algorithms such as K-Means (MacQueen, 1965), Gaussian Mixture Models (GMM) (Bishop, 2006), and spectral clustering (Shi & Malik, 2000) perform clustering based on distance or similarity. However, handcrafted distance or similarity measures are rarely reliable for large-scale high-dimensional data, making it increasingly challenging to achieve effective clustering. An intuitive solution is to transform the data from the high-dimensional input space to the low-dimensional latent space and then to cluster the data in the latent space. This can be achieved by applying dimensionality reduction techniques such as PCA (Wold et al., 1987), t-SNE (Maaten & Hinton, 2008), and UMAP (McInnes et al., 2018). However, since these methods are not specifically designed for clustering tasks, some of their properties may be contrary to our expectations, e.g., two data points from different manifolds that are close in the input space will be closer in the latent space derived by UMAP. Therefore, the first question here is how to learn the manifold representation that favors clustering?
The two main points for the multi-manifold representation learning are Point (1) preserving the local geometric structure within each manifold and Point (2) ensuring the discriminability between different manifolds. Most previous work seems to start with the assumption that the label of each data point is known, and then design the algorithm in a supervised manner, which greatly simplifies the problem of multi-manifold learning. However, it is challenging to decouple complex crossover relations and ensure discriminability between different manifolds, especially in unsupervised settings. One natural strategy is to achieve Point (2) through performing clustering in the input space to get pseudo-labels and then performing representation learning for each manifold. However, clustering is in fact contradictory to Point (1) (which will be analyzed in detail in Sec. 3.3), making it important to alleviate this contradiction so that clustering helps both point (1) and point (2). Thus, the second question here is how to cluster data that favors learning manifold representation?
To answer these two questions, some pioneering work has proposed to integrate deep clustering and representation learning into a unified framework by defining a clustering-oriented loss. Though promising performance has been demonstrated on various datasets, we observe that a vital factor has been ignored by these work that the defined clustering-oriented loss may deteriorate the geometric
structure of the latent space 1, which in turn hurts the performance of visualization, clustering generalization, and manifold representation. In this paper, we propose to jointly perform deep clustering and multi-manifold representation learning with geometric structure preservation. Inspired by Xie et al. (2016), the clustering centers are defined as a set of learnable parameters, and we use a clustering loss to simultaneously guide the separation of data points from different manifolds and the learning of the clustering centers. To prevent clustering loss from deteriorating the latent space, an isometric loss and a ranking loss are proposed to preserve the intra-manifold structure locally and inter-manifold structure globally. Finally, we achieve the following three goals related to clustering, geometric structure, and manifold representation: (1) Clustering helps to ensure inter-manifold discriminability; (2) Local structure preservation can be achieved with the presence of clustering; (3) Geometric structure preservation helps clustering.
The contributions of this work are summarized as below:
• Proposing to integrate deep clustering and multi-manifold representation learning into a unified framework with local and global structure preservation.
• Unlike conventional multi-manifold learning algorithms that deal with all point pair relationships between different manifolds simultaneously, we set the clustering centers as a set of learnable parameters and achieve global structure preservation in a faster, more efficient, and easier to optimize manner by applying ranking loss to the clustering centers.
• Analyzing the contradiction between two optimization goals of clustering and local structure preservation and proposing an elegant training strategy to alleviate it.
• The proposed DCRL algorithm outperforms competing algorithms in terms of clustering effect, generalizability to out-of-sample, and performance in manifold representation.

heading: 2 RELATED WORK
text: Clustering analysis. As a fundamental tool in machine learning, it has been widely applied in various domains. One branch of classical clustering is K-Means (MacQueen, 1965) and Gaussian Mixture Models (GMM) (Bishop, 2006), which are fast, easy to understand, and can be applied to a large number of problems. However, limited by Euclidean measure, their performance on high-dimensional data is often unsatisfactory. Spectral clustering and its variants (such as SCNcut (Bishop, 2006)) extend clustering to high-dimensional data by allowing more flexible distance measures. However, limited by the computational efficiency of the full Laplace matrix, spectral clustering is challenging to extend to large-scale datasets.
Deep clustering. The success of deep learning has contributed to the growth of deep clustering. One branch of deep clustering performs clustering after learning a representation through existing unsupervised techniques. For example, Tian et al. (2014) use autoencoder to learn low dimensional features and then runK-Means to get clustering results (AE+K-Means). Considering the geometric structure of the data, N2D (McConville et al., 2019) applies UMAP to find the best clusterable manifold of the obtained embedding, and then runK-Means to discover higher-quality clusters. The other category of algorithms tries to optimize clustering and representation learning jointly. The closest work to us is Deep Embedding Clustering (DEC) (Xie et al., 2016), which learns a mapping from the input space to a low dimensional latent space through iteratively optimizing clustering-oriented objective. As a modified version of DEC, while IDEC (Guo et al., 2017) claims to preserve the local structure of the data, in reality, their contribution is nothing more than adding a reconstruction loss. JULE (Yang et al., 2016b) unifies unsupervised representation learning with clustering based on the CNN architecture to improve clustering accuracy, which can be considered as a neural extension of hierarchical clustering. DSC devises a dual autoencoder to embed data into latent space, and then deep spectral clustering (Shaham et al., 2018) is applied to obtain label assignments (Yang et al., 2019). ASPC-DA (Guo et al., 2019) combines data augmentation with self-paced learning to encourage the learned features to be cluster-oriented. While sometimes they both evaluate performance in terms of accuracy, we would like to highlight that deep clustering and visual self-supervised learning (SSL) are two different research fields. SSL typically uses more powerful CNN architecture (applicable only to image data), and uses sophisticated techniques such as contrastive learning (He
1This claim was first made by IDEC (Guo et al., 2017), but they did not provide experiments to support it. In this paper, however, we show that the geometry of the latent space is indeed disrupted by visualization of learned embeddings (Fig. 4), visualization of clustering process (Fig. A3), and statistical analysis (Fig. A5).
et al., 2020), data augmentation (Chen et al., 2020), and clustering (Zhan et al., 2020; Ji et al., 2019; Van Gansbeke et al., 2020) for better performance on large-scale datasets such as ImageNet. Deep clustering, however, uses general MLP architecture (applicable to both image and vector data), so it is difficult to scale directly to large datasets without considering those sophisticated techniques.
Manifold Representation Learning. Isomap, as a representative algorithm of single-manifold learning, aims to capture global nonlinear features and seek an optimal subspace that best preserves the geodesic distance between data points (Tenenbaum et al., 2000). In contrast, some algorithms, such as the Locally Linear Embedding (LLE) (Roweis & Saul, 2000), are more concerned with the preservation of local neighborhood information. Combining DNN with manifold learning, the recently proposed Markov-Lipschitz Deep Learning (MLDL) algorithm achieves the preservation of local and global geometries by imposing Locally Isometric Smoothness (LIS) prior constraints (Li et al., 2020). Furthermore, multi-manifold learning is proposed to obtain intrinsic properties of different manifolds. Yang et al. (2016a) proposed a supervised discriminant isomap where data points are partitioned into different manifolds according to label information. Similarly, Zhang et al. (2018) proposed a semi-supervised learning framework that applies the labeled and unlabeled training samples to perform the joint learning of local neighborhood-preserving features. In most previous work on multi-manifold learning, the problem is considered from the perspective that the label is known or partially known, which significantly simplifies the problem. However, it is challenging to decouple multiple overlapping manifolds in unsupervised settings, and that is what this paper aims to explore.

heading: 3 PROPOSED METHOD
text: Consider a dataset X with N samples, and each sample xi ∈ Rd is sampled from C different manifolds {Mc}Cc=1. Assume that each category in the dataset lies in a compact low-dimensional manifold, and the number of manifolds C is prior knowledge. Define two nonlinear mapping zi = f(xi, θf ) and yi = g(zi, θg), where zi ∈ Rm is the embedding of xi in the latent space, yi is the reconstruction of xi. The j-th cluster center is denoted as µj ∈ Rm, where {µj}Cj=1 is defined as a set of learnable parameters. We aim to find optimal parameters θf and µ so that the embeddings {zi}Ni=1 can achieve clustering with local and global structure preservation. To this end, a denoising autoencoder (Vincent et al., 2010) shown in Fig 1 is first pre-trained in an unsupervised manner to learn an initial latent space. Denoising autoencoder aims to optimize the self-reconstruction loss LAE = MSE(x̂, y), where the x̂ is a copy of x with Gaussian noise added, that is, x̂ = x + N(0, σ2). Then the autoencoder is finetuned by optimizing the following clustering-oriented loss {Lcluster(z, µ)} and structure-oriented losses {Lrank(x, µ), LLIS(x, z), Lalign(z, µ)}. Since the clustering should be performed on features of clean data, instead of noised data x̂ that is used in denoising autoencoder, the clean data x is used for fine-tuning.

heading: 3.1 CLUSTERING-ORIENTED LOSS
text: First, the cluster centers {µj}Cj=1 in the latent space Z are initialized (the initialization method will be introduced in Sec 4.1). Then the similarity between the embedded point zi and cluster centers {µj}Cj=1 is measured by Student’s t-distribution:
qij =
( 1 + ‖zi − µj‖2 )−1 ∑
j′
( 1 + ‖zi − µj′‖2 )−1 (1) The auxiliary target distribution is designed to help manipulate the latent space, defined as:
pij = q2ij/fj∑ j′ q 2 ij′/fj′ , where fj = ∑ i qij (2)
where fj is the normalized cluster frequency, used to balance the size of different clusters. Then the encoder is optimized by the following objective:
Lcluster = KL(P‖Q) = ∑ i ∑ j pij log pij qij
(3)
The gradient of Lcluster with respect to each learnable cluster center µj can be computed as:
∂Lcluster ∂µj = − ∑ i ( 1 + ‖zi − µj‖2 )−1 · (pij − qij) (zi − µj) (4)
Lcluster facilitates the aggregation of data points within the same manifold, while data points from different manifolds are kept away from each other. However, we find that the clustering-oriented loss may deteriorate the geometric structure of the latent space, which hurts the clustering accuracy and leads to meaningless representation. To prevent the deterioration of clustering loss, we introduce isometry loss LLIS and ranking loss Lrank to preserve the local and global structure, respectively.

heading: 3.2 STRUCTURE-ORIENTED LOSS
text: Intra-manifold Isometry Loss. The intra-manifold local structure is preserved by optimizing the following objective:
LLIS = N∑ i=1 ∑ j∈NZi |dX (xi, xj)− dZ (zi, zj)| · π(l(xi) = l(xj)) (5)
where NZi represents the neighborhood of data point zi in the feature space Z, and the kNN is applied to determine the neighborhood. π(·) ∈ {0, 1} is an indicator function, and l(xi) is a manifold determination function that returns the manifold si where sample xi is located, that is si = l(xi) = argmaxj pij . Then we can derive C manifolds {Mc} C c=1: Mc = {xi; si = c, i = 1, 2, ..., N}. In a nutshell, the loss LLIS constrains the isometry within each manifold.
Inter-manifold Ranking Loss. The inter-manifold global structure is preserved by optimizing the following objective:
Lrank = C∑ i=1 C∑ j=1 ∣∣dZ (µi, µj)− κ · dX (vXi , vXj )∣∣ (6) where {vXj }Cj=1 is defined as the centers of different manifolds in the original input space X with vXj = 1 |Mj | ∑ i∈Mj xi (j = 1, 2, ..., C). The parameter κ determines the extent to which different manifolds move away from each other. The larger κ is, the further away the different manifolds are from each other. The derivation for the gradient of Lrank with respect to each learnable cluster center µj is placed in Appendix A.1. Note that Lrank is optimized in an iterative manner, rather than by initializing {µj}Cj=1 once and then separating different clusters based only on the initialization
results. Additionally, contrary to us, the conventional methods for dealing with inter-manifold separation typically impose push-away constraints on all data points from different manifolds (Zhang et al., 2018; Yang et al., 2016a), defined as:
Lsep = − N∑ i=1 N∑ j=1 dZ (zi, zj) · π(l(xi) 6= l(xj)) (7)
The main differences between Lrank and Lsep are as follows: (1) Lsep imposes constraints on embedding points {zi}Ni=1, which in turn indirectly affects the network parameters θf . In contrast, Lrank imposes rank-preservation constrains directly on learnable parameters {µj}Cj=1 in the form of regularization item to control the separation of the clustering centers. (2)Lrank is easier to optimize, faster to process, and more accurate. Lsep is imposed on all data points from different manifolds, which involves N×N point-to-point relationships. This means that each point may be subject to the push-away force from other manifolds, but at the same time, each point has to meet the isometry constraint with its neighboring points. Under these two constraints, optimization is difficult and it is easy to fall into a local optimal solution and output inaccurate results. In contrast, Lrank is imposed directly on the clustering centers, involving onlyC×C cluster-to-cluster relationships, which avoids the above problem and makes it easier to optimize. (3) The parameter κ introduced in Lrank allows us to control the extent of separation between manifolds for specific downstream tasks.
Alignment Loss. Note that the global ranking loss Lrank is imposed directly on the learnable parameter {µj}Cj=1, so optimizing Lrank only updates {µj} C j=1 rather the encoder’s parameter θf . However, the optimization of {µj}Cj=1 not only relies on Lrank, but is also constrained by Lcluster, which ensures that data points remain roughly distributed around cluster centers and do not deviate significantly from them during the optimization process. Alignment loss Lalign, as an auxiliary term, aims to help align learnable cluster centers {µj}Cj=1 with real cluster centers {v Z j }Cj=1 and make this binding stronger:
Lalign = C∑ j=1 ||µj − vZj || (8)
where {vZj }Cj=1 are defined as vZj = 1|Mj | ∑
i∈Mj zi (j = 1, 2, ..., C). The derivation for the gradient of Lalign with respect to each learnable cluster center µj is placed in Appendix A.1.

heading: 3.3 TRAINING STRATEGY

heading: 3.3.1 CONTRADICTION
text: The contradiction between clustering and local structure preservation is analyzed from the forces analysis perspective. As shown in Fig 2, we assume that there exists a data point (red point) and its three nearest neighbors (blue points) around a cluster center (gray point). When clustering and local structure preserving are optimized simultaneously, it is very easy to fall into a local optimum, where the data point is in steady-state, and the resultant force from its three nearest neighbors is equal in magnitude and opposite to the gravitational forces of the cluster. Therefore, the following training strategy is applied to prevent such local optimal solutions.

heading: 3.3.2 ALTERNATING TRAINING AND WEIGHT CONTINUATION
text: Alternating Training. To solve the above problem and integrate the goals of clustering and structure preservation into a unified framework, we take an alternating training strategy. Within each epoch, we first jointly optimize Lcluster and Lrank in a mini-batch, with joint loss defined as
L1 = LAE + αLcluster + Lrank (9)
where α is the weighting factor that balances the effects of clustering and global rank-preservation. Then at each epoch, we optimize isometry loss LLIS and Lalign on the whole dataset, defined as
L2 = βLLIS + Lalign (10)
Weight continuation. At different stages of training, we have different expectations for the clustering and structure-preserving. At the beginning of training, to successfully decouple the overlapping manifolds, we hope that the Lcluster will dominate and LLIS will be auxiliary. When the margin between different manifolds is sufficiently pronounced, the weight α for Lcluster can be gradually reduced, while the weight β for LLIS can be gradually increased, focusing on the preservation of the local isometry. The whole algorithm is summarized in Algorithm 1 in Appendix A.2.
Three-stage explanation. The training process can be roughly divided into three stages, as shown in Fig 3, to explain the training strategy more vividly. At first, four different manifolds overlap. At Stage 1, Lcluster dominates, thus data points within each manifold converge towards cluster centers to form spheres, but the local structure of manifolds is destroyed. At Stage 2, Lrank dominates, thus different manifolds in the latent space move away from each other to increase the manifold margin and enhance the discriminability. At stage 3, the manifolds gradually recover their original local structure from the spherical shape with LLIS dominating. It is worth noting that the above losses may coexist with each other rather than being completely independent at different stages, but that the role played by different losses varies due to the alternating training and weight continuation.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUPS
text: In this section, the effectiveness of the proposed framework is evaluated in 6 benchmark datasets: MNIST-full, MNIST-test, USPS, Fashion-MNIST, REUTERS-10K and HAR, on which our method is compared with 9 other methods mentioned in Sec 2 in 8 evaluation metrics including metrics designed specifically for clustering and manifold representation learning. The brief descriptions of the datasets are given in Appendix A.3.
Parameters settings. Currently we use MLP architecture for this version and will extend it to ConvAE in the future. The encoder structure is d-500-500-500-2000-10 where d is the dimension of the input data, and the decoder is its mirror. After pretraining, in order to initialize the learnable clustering centers, the t-SNE is applied to transform the latent space Z to 2 dimensions further, and then the K-Means algorithm is run to obtain the label assignments for each data point 2. The centers of each category in the latent space Z are set as initial cluster centers {µj}Cj=1. The batch size is set to 256, the epoch is set to 300, the parameter k for nearest neighbor is set to 5, and the parameter κ
2Since cluster centers {µj}Cj=1 are learnable and updated in an iterative manner, we believe that a proper initialization is sufficient, and the exploration of initialization methods is beyond the scope of this paper.
is set to 3 for all datasets. Sensitivity analysis for parameters k and κ is available in Appendix A.12. Besides, Adam optimizer (Kingma & Ba, 2014) with learning rate λ=0.001 is used. As described in Sec 3.3.2, the weight continuation is applied to train the model. The weight parameter α for Lcluster decreases linearly from 0.1 to 0 within epoch 0-150. In contrast, the weight parameter β for LLIS increases linearly from 0 to 1.0 within epoch 0-150. In this paper, each set of experiments is run 5 times with different 5 random seeds, and the results are averaged into the final performance metrics. The implementation uses the PyTorch library running on NVIDIA v100 GPU.
Evaluation Metrics. Two standard evaluation metrics: Accuracy (ACC) and Normalized Mutual Information (NMI) (Xu et al., 2003) are used to evaluate clustering performance. Besides, six evaluation metrics are adopted in this paper to evaluate the performance of multi-manifold representation learning, including Relative Rank Error (RRE), Trustworthiness (Trust), Continuity (Cont), Root Mean Reconstruction Error (RMRE), Locally Geometric Distortion (LGD) and Cluster Rank Accuracy (CRA). Limited by space, their precise definitions are available in Appendix A.4.

heading: 4.2 EVALUATION OF CLUSTERING

heading: 4.2.1 QUANTITATIVE COMPARISON
text: The metrics ACC/NMI of different methods on various datasets are reported in Tab 1. For those comparison methods whose results are not reported or the experimental settings are not clear on some datasets, we run the released code using the hyperparameters provided in their paper with the same random seeds and initialization, then report their average performance, and label them with (*). While ASPC-DA achieves the best performance on three datasets (MNIST-test, MNIST-full, and USPS), its performance gains do not come directly from clustering, but from sophisticated modules such as data augmentation and self-paced learning. Once these modules are removed, there is a very large degradation in its performance. For example, with data augmentation removed, ASPC-DA achieves less competitive performance, e.g., an accuracy of 0.931 (vs 0.988) on MNIST-full, 0.813 (vs 0.973) on MNIST-test and 0.768 (vs 0.982) on USPS. Though ASPC-DA is based on the MLP architecture, its image-based Data Augmentation (DA) cannot be applied directly to vector data, which explains why ASPC has no performance advantage on the vector-based REUTERS-10K and HAR datasets (even compared to DEC and IDEC).
In a fairer comparison (without considering ASPC-DA), we find that DCRL outperforms K-Means and SC-Ncut by a significant margin and surpasses the other seven competing DNN-based algorithms on all datasets except MNIST-test. Even with the MNIST-test dataset, we still rank second, outperforming the third by 1.1%. In particular, we obtained the best performance on the FashionMNIST and HAR (vector) dataset , and more notably, our clustering accuracy exceeds the current SOTA method by 5.1% and 4.9%, respectively.

heading: 4.2.2 GENERALIZABILITY EVALUATION
text: Tab 2 demonstrates that a learned DCRL can generalize well to unseen data with high clustering accuracy. Taking MNIST-full as an example, DCRL was trained using 50,000 training samples and then tested on the remaining 20,000 testing samples using the learned model. In terms of the metrics ACC and MNI, our method is optimal for both training and testing samples. More importantly, there is hardly any degradation in the performance of our method on the testing samples compared
to the training samples, while all other methods showed a significant drop in performance, e.g., DEC from 84.1% to 74.8%. This demonstrates the importance of geometric structure preservation for good generalizability. The testing visualization available in Appendix A.5 shows that DCRL still maintains clear inter-cluster boundaries even on the test samples, which demonstrates the great generalizability of our method.

heading: 4.2.3 CLUSTERING VISUALIZATION
text: The visualization of DCRL with several comparison methods is shown in Fig 4 (visualized using UMAP). From the perspective of clustering, our method is much better than the other methods. Among all methods, only DEC, IDEC, and DCRL can hold clear boundaries between different clusters, while the cluster boundaries of the other methods are indistinguishable. Although DEC and IDEC can successfully separate different clusters, they group many data points from different classes into the same cluster. Most importantly, due to the use of the clustering-oriented loss, the embedding learned by algorithms such as DEC, IDEC, JULE, and DSC (especially DSC) tend to form spheres and disrupt the original topological structure. Instead, our method overcomes these problems and achieves almost perfect separation between different clusters while preserving the local and global structure.
Additionally, the embedding of latent space during the training process is visualized in Appendix A.6, which is highly consistent with the three-stage explanation mentioned in Sec 3.3.2, showing that clustering-oriented does indeed do deteriorate the local geometric structure of the latent space, and designed LLIS helps to recover it. In addition, in the above experiments, the cluster numberC is assumed to be a known prior (which is consistent with the assumptions of almost all deep clustering algorithms). Therefore, we provide an additional experiment to explore what happens when C is larger than the number of true clusters. It is found that there exists splitting of the clusters, but the different categories still maintain clear boundaries and are not mixed together, somewhat similar to hierarchical clustering. See Appendix A.7 for detailed experimental settings and analysis.

heading: 4.3 EVALUATION OF MULTI-MANIFOLD REPRESENTATION LEARNING
text: Although numerous previous work has claimed that they brought clustering and representation learning into a unified framework, they all, unfortunately, lack an analysis of the effectiveness of the learned representations. In this paper, we compare DCRL with the other five methods in six evaluation metrics on six datasets. (Limited by space, only MNIST-full results are provided in the Tab 3 and the complete results are in Appendix A.8). The results show that DCRL outperforms all other methods, especially in the CRA metric, which is not only the best on all datasets but also reaches 1.0. This means that the “rank” between different manifolds in the latent space is completely preserved and undamaged, which proves the effectiveness of our global ranking loss Lrank.
Moreover, statistical analysis is performed in this paper to show the extent to which local and global structure is preserved in the latent space for each algorithm. Limited by space, they are placed in Appendix A.9. Furthermore, we also evaluated whether the learned representations are meaningful through downstream tasks, and this experiment is available in Appendix A.10.

heading: 4.4 ABLATION STUDY
text: This evaluates the effects of the loss terms and training strategies in the DCRL with five sets of experiments: the model without (A) Structure-oriented Loss (SL); (B) Clustering-oriented Loss (CL); (C) Weight Continuation (WC); (D) Alternating Training (AT), and (E) the full model. Limited by space, only MNIST-full results are provided in Tab 4, and results for the other four datasets are in Appendix A.11. After analyzing the results, we can conclude: (1) CL is the most important factor for obtaining good clustering, the lack of which leads to unsuccessful clustering, hence the numbers in the table are not very meaningful and are shown in gray color. (2) SL not only brings subtle improvements in clustering but also greatly improves the performance of multi-manifold representation learning. (3) Our elegant training strategies (WC and AT) both improve the performance of clustering and multi-manifold representation learning to some extent, especially on metrics such as RRE, Trust, Cont, and CRA.

heading: 5 CONCLUSION
text: The proposed DCRL framework imposes clustering-oriented and structure-oriented constraints to optimize the latent space for simultaneously performing clustering and multi-manifold representation learning with local and global structure preservation. Extensive experiments on image and vector datasets demonstrate that DCRL is not only comparable to the state-of-the-art deep clustering algorithms but also able to learn effective and robust manifold representation, which is beyond the capability of those clustering methods that only care about clustering accuracy. Future work will focus on the adaptive determination of manifolds (clusters) numbers and extend our work to CNN architecture for large-scale datasets.

heading: APPENDIX

heading: A.1 GRADIENT DERIVATION
text: In the paper, we have emphasized time and again that {µj}Cj=1 is a set of learnalbe parameters, which means that we can optimize it while optimizing the network parameter θf . In Eq. (4) of the paper, we have presented the gradient of Lcluster with respect to µj . In addition to Lcluster, both Lrank and Lalign are involving µj . Hence, the detailed derivations for the gradient of Lrank and Lalign with respect to µj are also provided. The gradient of Lrank with respect to each learnalbe cluster center µj can be computed as:
∂Lrank ∂µj
= ∂ ∑C i′=1 ∑C j′=1 ∣∣dZ (µi′ , µj′)− κ ∗ dX (vXi′ , vXj′ )∣∣ ∂µj
= C∑ i′=1 C∑ j′=1
∂ ∣∣dZ (µi′ , µj′)− κ ∗ dX (vXi′ , vXj )∣∣
∂µj
The Euclidean metric is used for both the input space and the hidden layer space, i.e., dZ (µi′ , µj′) = ‖µi′ − µj′‖. In addition, the symbols are somewhat abused for clear derivation, representing κ ∗ dX ( vXi′ , v X j′ ) with K. Accordingly, Eq. (11) can be further derived as follows:
∂Lrank ∂µj = C∑ i′=1 C∑ j′=1
∂ ∣∣dZ (µi′ , µj′)− κ ∗ dX (vXi′ , vXj′ )∣∣
∂µj
= C∑ i′=1 C∑ j′=1
∂ ∣∣ ‖µi′ − µj′‖ −K∣∣
∂µj
= C∑ i′=1
∂ ∣∣ ‖µi′ − µj‖ −K∣∣
∂µj + C∑ j′=1
∂ ∣∣ ‖µj − µj′‖ −K∣∣
∂µj
= C∑ i′=1 ∂ (‖µi′ − µj‖ −K) ∂µj · ‖µi ′ − µj‖ −K∣∣ ‖µi′ − µj‖ −K∣∣
+ C∑ j′=1 ∂ (‖µj − µj′‖ −K) ∂µj · ‖µj − µj ′‖ −K∣∣ ‖µj − µj′‖ −K∣∣
= C∑ i′=1 ∂ ‖µi′ − µj‖ ∂µj · ‖µi ′ − µj‖ −K∣∣ ‖µi′ − µj‖ −K∣∣
+ C∑ j′=1 ∂ ‖µj − µj′‖ ∂µj · ‖µj − µj ′‖ −K∣∣ ‖µj − µj′‖ −K∣∣
= C∑ i′=1 µj − µi′ ‖µj − µi′‖ · ‖µj − µi ′‖ −K∣∣ ‖µj − µi′‖ −K∣∣ + C∑ j′=1 µj − µj′ ‖µj − µj′‖ · ‖µj − µj ′‖ −K∣∣ ‖µj − µj′‖ −K∣∣
= 2 C∑ i′=1 µj − µi′ ‖µj − µi′‖ · ‖µj − µi ′‖ −K∣∣ ‖µj − µi′‖ −K∣∣
= 2 C∑ i′=1 µj − µi′ ‖µj − µi′‖ · ‖µj − µi′‖ − κ ∗ dX
( vXi′ , v X j )∣∣ ‖µj − µi′‖ − κ ∗ dX (vXi′ , vXj ) ∣∣ = 2
C∑ i′=1 µj − µi′ dZ (µj , µi′) · dZ (µj , µi′)− κ ∗ dX
( vXi′ , v X j )∣∣dZ (µj , µi′)− κ ∗ dX (vXi′ , vXj )∣∣
The gradient of Lalign with respect to each learnalbe cluster center µj can be computed as:
∂Lalign ∂µj
= ∂ ∑C
j′=1 ||µj′ − vZj′ || ∂µj
= C∑ j′=1 ∂||µj′ − vZj′ || ∂µj
= ∂||µj − vZj ||
∂µj
= ∂(µj − vZj ) ∂µj · µj − vZj∥∥µj − vZj ∥∥ = µj − vZj∥∥µj − vZj ∥∥

heading: A.2 ALGORITHM
text: Algorithm 1 Algorithm for Deep Clustering and Representation Learning

heading: Input:
text: Input samples: X; Number of clusters: C; Number of batches: B; Number of iterations: E. Output:
Autoencoder’s weights: θf and θg; Cluster labels {si}Ni=1; Trainable cluster centers {µj} C j=1. 1: Initialize the weight {µj}Cj=1, θf and θg , and obtain initialized soft label assignment {si} N i=1. 2: for epoch ∈ {0,1,· · · ,E} do 3: Compute embedded points {zi}Ni=1 and distribution Q; 4: Update target distribution P ; 5: Compute soft cluster centers { vXi }C i=1 and { vZi }C i=1
. 6: for batch ∈ {0,1,· · · ,B} do 7: Pick up one batch of samples Xbatch from X; 8: Compute corresponding distribution Qbatch and it’s reconstruction Ybatch; 9: Pick up target distribution batch Pbatch from P ;
10: Compute loss Lae, Lcluster and Lrank; 11: Update the weight θf , θg and {µj}Cj=1. 12: end for 13: Compute Liso and Lalign on all samples; 14: Update the weight θf and {µj}Cj=1; 15: Assign new soft labels {si}Ni=1. 16: end for 17: return θf , θg , {si}Ni=1, {µj} C j=1.

heading: A.3 DATASETS
text: To show that our method works well with various kinds of datasets, we choose the following six image and vector datasets. Some example images are shown in Fig A1, and the brief descriptions of the datasets are given in Tab A1.
• MNIST-full (LeCun et al., 1998): The MNIST-full dataset consists of 70,000 handwritten digits of 28 × 28 pixels. Each gray image is reshaped to a 784-dimensional vector.
• MNIST-test (LeCun et al., 1998): The MNIST-test is the testing part of the MNIST dataset, which contains a total of 10000 samples.
• USPS 3: The USPS dataset is composed of 9298 gray-scale handwritten digit images with a size of 16x16 pixels.
3https://cs.nyu.edu/∼roweis/data.html
Table A1: Description of Datasets. Dataset Samples Categories Data Size MNIST-full 70000 10 28×28×1 MNIST-test 10000 10 28×28×1 USPS 9298 10 16×16×1 Fashion-MNIST 70000 10 28×28×1 REUTERS-10K 10000 4 2000 HAR 10299 6 561
• Fashion-MNIST (Xiao et al., 2017): This Fashion-MNIST dataset has the same number of images and the same image size as MNIST-full, but it is fairly more complicated. Instead of digits, it consists of various types of fashion products. • REUTERS-10K: REUTERS (Lewis et al., 2004) is composed of around 810000 English
news stories labeled with a category tree. Four root categories (corporate/industrial, government/social, markets, and economics) are used as labels and excluded all documents with multiple labels. Following DEC (Xie et al., 2016), a subset of 10000 examples are randomly sampled, and the tf-idf features on the 2000 most frequent words are computed. The sampled dataset is denoted REUTERS-10K. • HAR: HAR is a time series dataset consisting of 10,299 sensor samples from a smartphone.
It was collected from 30 people performing six different activities: walking, walking upstairs, walking downstairs, sitting, standing, and laying.
(a) MNIST
(b) USPS
(c) Fashion-MNIST
Figure A1: The image samples from three datasets (MNIST, USPS, and Fashion-MNIST)

heading: A.4 DEFINITIONS OF PERFORMANCE METRICS
text: The following notations are used for the definitions:
dX(i, j): the pairwise distance between xi and xj in input space X; dZ(i, j): the pairwise distance between zi and zj in latent space Z;
N k,Xi : the set of indices to the k-nearest neighbor (kNN) of xi in input space X; N k,Zi : the set of indices to the k-nearest neighbor (kNN) of zi in latent space Z; rX(i, j): the rank of the closeness (in Euclidean distance) of xj to xi in input space X; rZ(i, j): the rank of the closeness (in Euclidean distance) of zj to zi in latent space Z.
The eight evaluation metrics are defined below:
(1) ACC (Accuracy) measures the accuracy of clustering:
ACC = max m
∑N i=1 1 {li = m (si)}
N
where li and si are the true and predicted labels for data point xi, respectively, and m(·) is all possible one-to-one mappings between clusters and label categories.
(2) NMI (Normalized Mutual Information) NMI calculates the normalized measure of similarity between two labels of the same data
NMI = I(l; s)
max{H(l), H(s)}
where I(l, s) is the mutual information between the real label l and predicted label s, and H(·) represents their entropy.
(3) RRE (Relative Rank Change) measures the average of changes in neighbor ranking between two spaces X and Z:
RRE = 1
(k2 − k1 + 1) k2∑ k=k1 { MRkX→Z +MR k Z→X } where k1 and k2 are the lower and upper bounds of the k-NN.
MRkX→Z = 1
Hk N∑ i=1 ∑ j∈Nk,Zi ( |rX(i, j)− rZ(i, j)| rZ(i, j) )
MRkZ→X = 1
Hk N∑ i=1 ∑ j∈Nk,Xi ( |rX(i, j)− rZ(i, j)| rX(i, j) ) where Hk is the normalizing term, defined as
Hk = N k∑ l=1 |N − 2l| l .
(4) Trust (Trustworthiness) measures to what extent the k nearest neighbors of a point are preserved when going from the input space to the latent space:
Trust = 1
k2 − k1 + 1 k2∑ k=k1 1− 2Nk(2N − 3k − 1) N∑ i=1 ∑ j∈Nk,Zi ,j /∈N k,X i (rX(i, j)− k)  where k1 and k2 are the bounds of the number of nearest neighbors.
(5) Cont (Continuity) is defined analogously to Trust, but checks to what extent neighbors are preserved when going from the latent space to the input space:
Cont = 1
k2 − k1 + 1 k2∑ k=k1 1− 2Nk(2N − 3k − 1) N∑ i=1 ∑ j /∈Nk,Zi ,j∈N k,X i (rZ(i, j)− k)  where k1 and k2 are the bounds of the number of nearest neighbors.
(6) d-RMSE (Root Mean Square Error) measures to what extent the two distributions of distances coincide:
d−RMSE = √√√√ 1 N2 N∑ i=1 N∑ j=1 (dX(i, j)− dZ(i, j))2
(7) LGD (Locally Geometric Distortion) measures how much corresponding distances between neighboring points differ in two metric spaces and is the primary metric for isometry, defined as:
LGD = k2∑ k=k1 √√√√ M∑ i ∑ j∈Nk,(l)i (dl(i, j)− dl′(i, j))2 (k2 − k1 + 1)2M(#Ni) .
where k1 and k2 are the lower and upper bounds of the k-NN. (8) CRA (Cluster Rank Accuracy) measures the changes in ranks of cluster centers from the
input space X and to the latent space Z:
CRA =
∑C i=1 ∑C j=1 1(rX(v X i , v X j ) = rZ(v Z i , v Z j ))
C2
where C is the number of clusters, vXj is the cluster center of the jth cluster in the input space X , vZj is the cluster center of the jth cluster in the latent space Z, rX(v X i , v X j ) denotes the rank of the closeness (in terms of Euclidean distance) of vXi to v X j in space X in the input space X , and rZ(vZi , v Z j ) denotes the rank of the closeness (in terms of Euclidean distance) of vZi to v Z j in space Z.
A.5 VISUALIZATION IN GENERALIZABILITY
The visualization results on the testing samples are shown in Fig A2; even for testing samples, our method still shows distinguishable inter-cluster discriminability, while all the other methods without exception coupled different clusters together.
(a) AE+K-Means (b) DEC (c) IDEC (d) JULE
(e) DSC (f) N2D (g) DCRL (ours)
Figure A2: The visualization of the obtained embeddings on the testing samples to show the generalization performance of different algorithms on MNIST-full dateset.
A.6 VISUALIZATION IN DIFFERENT STAGES
The embedding visualization of the latent space during the training process is visualized in Fig A3 for depicting how both clustering and structure-preserving is achieved. We can see that the different clusters initialized by pretrained autoencoder are closely adjacent. In the early stage of training, with clustering loss Lcluster and global ranking loss Lrank, different manifolds are separated from each other, each manifold loses its local structure, and all of them degenerate into spheres. As
the training progresses, the weight α for Lcluster gradually decreases, while the weight β for Liso increases and the optimization is gradually focused from global to local, with each manifold gradually recovering its original geometric structure from the sphere. Moreover, since our local isometry loss Liso is constrained within each manifold, the preservation of local structure will not disrupt the global ranking. Finally, we obtain representations in which cluster boundaries are clearly distinguished, and local and global structures are perfectly preserved. This shows that clusteringoriented loss does deteriorate the local geometric structure of the latent space, and designed LLIS helps to recover it.
(a) Epoch 0 (b) Epoch 9 (c) Epoch 19 (d) Epoch 29 (e) Epoch 69
(f) Epoch 119 (g) Epoch 159 (h) Epoch 209 (i) Epoch 249 (j) Epoch 299
Figure A3: Clustering visualization at different stages of training on MNIST-full dateset.

heading: A.7 EXPLORATION ON THE ASSUMED CLUSTER NUMBER C
text: Taking MNIST-test dataset as an example, we present the embedding visualization with assumed number of clusters C being 10, 11, and 12, respectively. We find that when C is larger than the number of true clusters (10), data originally belonging to the same cluster will be split, e.g., a cluster is split into two, but the different categories of data still hold clear boundaries and are not mixed together, somewhat similar to hierarchical clustering.
(a) C=10 (b) C=11 (c) C=12
Figure A4: Clustering visualization with different assumed cluster numberC on MNIST-test dateset.

heading: A.8 QUANTITATIVE EVALUATION OF REPRESENTATION LEARNING
text: Our method is compared with the other five methods in six evaluation metrics on six datasets. The complete results in Tab A2 demonstrate the superiority of our method, especially on metrics RRE, Trust, Cont, and CRA. As shown in Tab A2, DCRL outperforms all other methods, especially in the CRA metric, which is not only the best on all datasets but also reaches 1.0. This means that the “rank” between different manifolds in the latent space is completely preserved and undamaged, which proves the effectiveness of our global ranking loss Lrank.
Table A2: Representation learning performance of different algorithms on five datasets. Datasets Algorithms RRE Trust Cont d-RMSE LGD CRA
MNIST-full
DEC 0.09988 0.84499 0.94805 44.8535 4.37986 0.28 IDEC 0.00984 0.99821 0.97936 24.5803 1.71484 0.33 JULE 0.02657 0.93675 0.98321 28.3412 2.12955 0.27 DSC 0.09785 0.87315 0.92508 6.98098 1.19886 0.23 N2D 0.01002 0.99243 0.98466 5.7162 0.69946 0.21 DCRL 0.00567 0.99978 0.98716 5.4986 0.69168 1.0
MNIST-test
DEC 0.12800 0.81841 0.91767 14.6113 2.29499 0.19 IDEC 0.01505 0.99403 0.97082 7.4599 1.08350 0.38 JULE 0.04122 0.92971 0.97208 9.4768 1.17176 0.42 DSC 0.10728 0.85498 0.92254 7.1689 1.19239 0.26 N2D 0.01565 0.98764 0.97572 5.0120 0.97454 0.33 DCRL 0.01090 0.99811 0.97612 5.8000 0.93394 1.0
USPS
DEC 0.07911 0.88871 0.94628 16.4355 1.77848 0.31 IDEC 0.01043 0.99726 0.97960 13.0573 1.11689 0.30 JULE 0.02972 0.98763 0.98810 14.6324 1.43426 0.33 DSC 0.06319 0.9151 0.93988 8.4412 1.02131 0.27 N2D 0.01337 0.98769 0.98135 8.1961 0.54967 0.37 DCRL 0.00577 0.99979 0.98701 6.4980 0.53180 1.0
Fasion-MNIST
DEC 0.04787 0.93896 0.95450 39.3274 3.87731 0.37 IDEC 0.01089 0.99683 0.97797 25.4024 1.91385 0.27 JULE 0.03013 0.97732 0.97923 15.2213 1.43642 0.43 DSC 0.05168 0.95013 0.96121 17.2201 1.42091 0.36 N2D 0.00894 0.99062 0.98054 14.49079 1.28180 0.26 DCRL 0.00836 0.99868 0.98203 13.3788 1.33893 1.0
REUTERS-10K DEC 0.26192 0.65518 0.80477 40.4671 4.00423 0.63 IDEC 0.05981 0.95840 0.90550 43.9556 2.01365 0.75 JULE - - - - - - DSC - - - - - - N2D 0.03827 0.97385 0.93412 36.1042 1.69013 0.31 DCRL (ours) 0.03206 0.98380 0.93802 34.5478 2.72096 1.0
HAR
DEC 0.09060 0.89097 0.91766 10.0222 1.58691 0.30 IDEC 0.01031 0.99433 0.98132 9.9155 0.93736 0.39 JULE - - - - - - DSC - - - - - - N2D 0.00841 0.99281 0.97695 8.2326 0.64296 0.33 DCRL (ours) 0.00665 0.99895 0.98634 15.2876 0.46189 1.0

heading: A.9 STATISTICAL ANALYSIS
text: The statistical analysis is presented to show the extent to which local and global structure is preserved from the input space to the latent space. Taking MNIST-full as an example, the statistical analysis of the global rank-preservation is shown in Fig A5 (a)-(f). For the i-th cluster, if the rank (in terms of Euclidean distance) between it and the j-th cluster is preserved from input space to latent space, then the grid in the i-th row and j-th column is marked as blue, otherwise yellow. As shown in the figure, only our method can fully preserve the global rank between different clusters, while all other methods fail.
Finally, we perform statistical analysis for the local isometry property of each algorithm. For each sample xi in the dataset, it forms a number of point pairs with its neighborhood samples {(xi, xj)|i = 1, 2, ..., N ;xj ∈ NXi }. We compute the difference in the distance of these point pairs from the input space to the latent space {dZ(xi, xj) − dX(xi, xj)|i = 1, 2, ..., N ;xj ∈ Ni}, and plot it as a histogram. As shown in Fig A5 (g), the curves of DCRL are distributed on both sides of the 0 value, with maximum peak height and minimum peak-bottom width, respectively, which indicates that DCRL achieves the best local isometry. Although IDEC claims that they can preserve the local structure well, there is still a big gap between their results and ours.
(a) DEC (b) IDEC (c) JULE (d) DSC
(e) N2D (f) DCRL (g) Local Isometry
Figure A5: Statistical analysis of different algorithms to compare the capability of global and local structure preservation from the input space to the latent space.

heading: A.10 QUANTITATIVE EVALUATION OF DOWNSTREAM TASKS
text: Numerous deep clustering algorithms have recently claimed to obtain meaningful representations, however, they do not analyze and experiment with the so-called ”meaningful” ones. Therefore, we are interested in whether these proposed methods can indeed learn representations that are useful for downstream tasks. Four different classifiers, including a linear classifier (Logistic Regression; LR), two nonlinear classifiers (MLP, SVM), and a tree-based classifier (Random Forest Classifier; RFC) are used as downstream tasks, all of which use default parameters and default implementations in sklearn (Pedregosa et al., 2011) for a fair comparison. The learned representations are frozen and used as input for training. The classification accuracy evaluated on the test set serves as a metric to evaluate the effectiveness of learned representations. In Tab A3, DCRL outperformed the other methods overall on all six datasets, with MLP, RFC, and LR as downstream tasks. Additionally, we surprisingly find that with MLP and RFC as downstream tasks, all methods other than DCRL do not even match the accuracy of AE on the MNIST-full dataset. Notably, DEC and IDEC show a sharp deterioration in performance on downstream tasks, falling short of even the simplest AEs, again showing that clustering-oriented loss can disrupt the geometry of the data.
Table A3: Performance of different algorithms in downstream tasks. Datasets Algorithms MLP RFC SVM LR
MNIST-full
AE 0.9746 0.9652 0.9859 0.9565 DEC 0.8647 0.8706 0.8707 0.8566 IDEC 0.9797 0.9737 0.9852 0.9650 JULE 0.9802 0.9825 0.9787 0.9743 DSC 0.9622 0.9501 0.9837 0.9752 N2D 0.9796 0.9803 0.9799 0.9792 DCRL 0.9851 0.9874 0.9869 0.9841
MNIST-test
AE 0.9415 0.9420 0.9745 0.9495 DEC 0.8525 0.8605 0.8725 0.8685 IDEC 0.9740 0.9725 0.9845 0.9655 JULE 0.9775 0.9845 0.9800 0.9825 DSC 0.9535 0.9740 0.9825 0.9795 N2D 0.9715 0.9760 0.9725 0.9725 DCRL 0.9855 0.9875 0.9865 0.9855
USPS
AE 0.9421 0.9469 0.9677 0.9073 DEC 0.8289 0.8668 0.8289 0.8294 IDEC 0.9482 0.9556 0.9656 0.9125 JULE 0.9576 0.9617 0.9703 0.9476 DSC 0.9351 0.9572 0.9612 0.9342 N2D 0.9569 0.9569 0.9569 0.9541 DCRL 0.9656 0.9651 0.9604 0.9551
Fasion-MNIST
AE 0.8613 0.9932 0.8314 0.7588 DEC 0.6268 0.9853 0.6377 0.6245 IDEC 0.8367 0.9918 0.8607 0.7514 JULE 0.8541 0.9892 0.8566 0.7723 DSC 0.8084 0.9823 0.8618 0.7676 N2D 0.8412 0.9493 0.8230 0.7753 DCRL 0.8642 0.9942 0.8468 0.7768
REUTERS-10K
AE 0.9325 0.9170 0.9375 0.8205 DEC 0.7985 0.7880 0.8105 0.7450 IDEC 0.9225 0.8930 0.9280 0.7705 JULE - - - - DSC - - - - N2D 0.9205 0.9080 0.9240 0.8335 DCRL (ours) 0.9360 0.9185 0.9390 0.8475
HAR
AE 0.9181 0.9139 0.9201 0.8849 DEC 0.7696 0.7847 0.7628 0.7634 IDEC 0.8973 0.9031 0.9041 0.8822 JULE - - - - DSC - - - - N2D 0.9138 0.9083 0.9174 0.8799 DCRL (ours) 0.9235 0.9193 0.9293 0.8996

heading: A.11 MORE ABLATION EXPERIMENTS
text: The results of the ablation experiments on the MNIST-full dataset have been presented in Tab 4 in Sec 4.3. Here, we provide four more sets of ablation experiments on the other four datasets. The conclusion is similar (note that the clustering performance of the model without clustering-oriented losses is very poorly, so the “best” metric numbers are not meaningful and are shown in gray color): (1) CL is very important for obtaining good clustering. (2) SL is beneficial for both clustering and representation learning. (3) Our training strategies (WC and AT) are very superior in improving metrics such as ACC, RRE, Trust, Cont, and CRA.
Table A4: Ablation study of loss items and training strategies used in DCRL.
Datasets Methods ACC/NMI RRE Trust Cont d-RMSE LGD CRA w/o SL 0.976/0.939 0.0093 0.9967 0.9816 24.589 1.6747 0.32 w/o CL 0.814/0.736 0.0004 0.9998 0.9990 7.458 0.0487 1.00 w/o WC 0.977/0.943 0.0065 0.9987 0.9860 5.576 0.6968 0.98 w/o AT 0.978/0.944 0.0069 0.9986 0.9851 5.617 0.7037 0.96 MNIST-full
full model 0.980/0.946 0.0056 0.9997 0.9871 5.498 0.6916 1.00 w/o SL 0.973/0.932 0.0146 0.9928 0.9727 7.701 1.0578 0.31 w/o CL 0.773/0.747 0.0020 0.9994 0.9954 7.229 0.0809 1.00 w/o WC 0.956/0.904 0.0132 0.9955 0.9735 5.470 0.9364 1.00 w/o AT 0.970/0.929 0.0118 0.9974 0.9747 5.567 0.9404 1.00 MNIST-test
full model 0.972/0.930 0.0109 0.9981 0.9761 5.800 0.9339 1.00 w/o SL 0.958/0.902 0.0095 0.9967 0.9812 14.609 0.9847 0.29 w/o CL 0.664/0.658 0.0020 0.9996 0.9952 2.934 0.0687 1.0 w/o WC 0.956/0.896 0.0060 0.9991 0.9868 6.572 0.5335 1.00 w/o AT 0.947/0.885 0.0080 0.9979 0.9833 5.960 0.4967 1.00 USPS
full model 0.960/0.902 0.0057 0.9997 0.9870 6.498 0.5318 1.00 w/o SL 0.706/0.682 0.0108 0.9964 0.9781 25.954 1.8936 0.30 w/o CL 0.576/0.569 0.0004 0.9994 0.9995 7.654 0.0523 1.00 w/o WC 0.702/0.695 0.0084 0.9972 0.9814 13.238 1.3474 1.00 w/o AT 0.708/0.694 0.0097 0.9975 0.9798 13.354 1.3611 1.00 Fasion-MNIST
full model 0.710/0.685 0.0083 0.9986 0.9820 13.378 1.3389 1.00 w/o SL 0.819/0.564 0.0529 0.9610 0.9185 44.481 1.9090 0.38 w/o CL 0.542/0.279 0.0277 0.9868 0.9456 37.018 2.2294 1.00 w/o WC 0.830/0.583 0.0420 0.9667 0.9361 35.302 2.8286 1.00 w/o AT 0.825/0.563 0.0440 0.9650 0.9330 39.275 2.9146 1.00 REUTERS-10K
full model 0.836/0.590 0.0320 0.9838 0.9380 34.547 2.7209 1.00 w/o SL 0.835/0.746 0.0116 0.9944 0.9792 8.168 0.8882 0.33 w/o CL 0.744/0.615 0.0024 0.9986 0.9948 15.060 0.2193 1.00 w/o WC 0.786/0.701 0.0130 0.9950 0.9756 15.398 0.6171 1.00 w/o AT 0.834/0.745 0.0089 0.9965 0.9835 15.726 0.4734 1.00 HAR
full model 0.845/0.758 0.0066 0.9989 0.9863 15.287 0.4618 1.00

heading: A.12 PARAMETER SENSITIVITY
text: We also evaluated the sensitivity of parameters k and κ on the MNIST-test dataset and the results are shown in Tab A5. The parameters k and κ are found to have little effect on the clustering performance (ACC/NMI), and some combinations of k and κ even produce better clustering performance than the metrics reported in the main paper. However, the effect of k and κ on representation learning is more pronounced, and different combinations of k and κ may increase or decrease performance. In general, this paper focuses on the design of the algorithm itself and has not performed the parameter search to find the best performance.
Table A5: Parameter Sensitivity with different parameters k and κ on the MNIST-test dataset. Parameters ACC/NMI RRE Trust Cont d-RMSE LGD CRA k=1, κ=3 0.975/0.936 0.0125 0.9944 0.9756 5.757 0.8868 1.00 k=3, κ=3 0.973/0.931 0.0114 0.9970 0.9757 5.805 0.9207 1.00 k=5, κ=3 0.972/0.930 0.0109 0.9981 0.9761 5.800 0.9339 1.00 k=8, κ=3 0.972/0.929 0.0104 0.9989 0.9765 5.810 0.9476 1.00 k=10, κ=3 0.972/0.929 0.0105 0.9990 0.9764 5.704 0.9487 1.00 k=5, κ=1 0.967/0.912 0.0068 0.9993 0.9845 5.409 0.2524 1.00 k=5, κ=3 0.972/0.930 0.0109 0.9981 0.9761 5.800 0.9339 1.00 k=5, κ=5 0.972/0.929 0.0146 0.9964 0.9691 15.0653 1.5719 1.00 k=5, κ=8 0.972/0.929 0.0190 0.9943 0.9615 29.4607 2.5410 1.00 k=5, κ=10 0.972/0.929 0.0195 0.9951 0.9597 37.7661 3.1434 1.00

Review summaries:
In this paper, the authors proposes a deep clustering model to enable the clustering and representation learning to favor each other via preserving the geometric structure of data. The proposed DCRL framework integrates an isometric loss for local intra-manifold structure and a ranking loss for global inter-manifold structure. The authors evaluate the proposed framework on five datasets and the experimental results show that the proposed framework brings certain improvements over the baseline approaches.