Neural Manifold Clustering and Embedding
Authors: ZENGYI LI
Yubei Chen
Yann LeCun
Friedrich Sommer
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-supervised Learning
Clustering
Subspace Clustering
Manifold Learning
Deep Subspace Clustering

Abstract: Given a union of non-linear manifolds, non-linear subspace clustering or manifold clustering aims to cluster data points based on manifold structures and also learn to parameterize each manifold as a linear subspace in a feature space. Deep neural networks have the potential to achieve this goal under highly non-linear settings given their large capacity and flexibility. We argue that achieving manifold clustering with neural networks requires two essential ingredients: a domain-specific constraint that ensures the identification of the manifolds, and a learning algorithm for embedding each manifold to a linear subspace in the feature space. This work shows that many constraints can be implemented by data augmentation. For subspace feature learning, Maximum Coding Rate Reduction (MCR$^2$) objective can be used. Putting them together yields Neural Manifold Clustering and Embedding (NMCE), a novel method for general purpose manifold clustering, which significantly outperforms autoencoder-based deep subspace clustering and achieve state-of-the-art performance on several important benchmarks. Further, on more challenging natural image datasets, NMCE can also outperform other algorithms specifically designed for clustering. Qualitatively, we demonstrate that NMCE learns a meaningful and interpretable feature space. As the formulation of NMCE is closely related to several important Self-supervised learning (SSL) methods, we believe this work can help us build a deep understanding on SSL representation learning.

Under review as a conference paper at ICLR 2022
NEURAL MANIFOLD CLUSTERING AND EMBEDDING
Anonymous authors
Paper under double-blind review
ABSTRACT
Given a union of non-linear manifolds, non-linear subspace clustering or manifold clustering aims to cluster data points based on manifold structures and also learn parameterize each manifold as linear subspace in feature space. Deep neural networks have the potential to achieve this goal under highly non-linear settings given their large capacity and flexibility. We argue that achieving manifold clus- tering with neural networks requires two essential ingredients: domain-specific constraint that ensures the identification of the manifolds, and learning algo- rithm for embedding each manifold to linear subspace in the feature space. This work shows that many constraints can be implemented by data augmentation. For subspace feature learning, Maximum Coding Rate Reduction (MCR²) objective can be used. Putting them together yields Neural Manifold Clustering and Em- bedding (NMCH), a novel method for general purpose manifold clustering, which significantly outperforms autoencoder-bet ofate-of-the-ar performance on several important benchmarks. Further, more challenging natural image datasets, NMCE can also outperform other algorithms specifically designed for clustering. Qualitatively, learns a meaningful and interpretable feature space. As the formulation of NMCE is closely related several important Self-supervised learning (SSL) methods, we believe this work can help us build a deeper understanding on SSL representation learning.
deep subspace clustering and achieve
demonstrate that NMCE
INTRODUCTION
Here we investigate unsupervised representation learning, which aims to learn structures (features) from data without the use of any label. If the data lie in a linear subspace, the linear subspace can be extracted by Principal Component Analysis (PCA) (Jolliffe 1986). one of the most basic forms of unsupervised learning. When the data occupy union of several linear subspaces, subspace clustering (SC) (Vidal et al., 2016) is needed to cluster each data point to subspace well as estimating the parameters of each subspace. Here we are concerned with even more challenging scenarios, when data points come from a union of several non-linear low-dimensional manifolds. In such scenarios, the clustering problem can be formulated as follows (Elhamifar Vidal 20IT):
Task 1. Manifoll Clustering and Embedding: Given that the data points come from a union of low- dimensional manifolds, we shall segment the data points based on their corresponding manifolds, and obtain a low-dimensional embedding for each manifold.
Various methods have been developed to solve this problem (Abdolali & Gillis 2021). ). but it still an open question how to use neural networks effectively in manifold clustering problems (Hacffelc al.) 2020). this paper, we propose neural manifold clustering and embedding (NMCE) that follows three principles: 1) The clustering and representation should respect domain-specific con- straint, e.g. local neighborhoods, local linear interpolation or data augmentation invariances. 2) The embedding of particular manifold shall not collapse. 3) The embedding of identified manifolds shall be linearized and separated, i.e. they occupy different linear subspaces. We achieve 1) using data augmentations, and achieve 2) and 3) with the subspace feature learning algorithm maximal coding rate reduction (MCR?) objective (Yu et al. 2020). This work make the following specific contributions:

---

Under review as a conference paper at ICLR 2022
1. We combine data augmentation with MCR to yield novel algorithm for general purpose manifold clustering and embedding (NMCE). We also discuss connections between the algorithm and self-supervised contrastive learning.
2. We demonstrate that NMCE achieves state-of-the-art performance on standard subspace clustering benchmarks, and can outperform the best clustering algorithms on more chal- lenging high-dimensional image datasets like CIFAR-10 and CIFAR-20. Further, empirical evaluation suggests that our algorithm also learns a meaningful feature space.
RELATED WORK
Manifold Learning. In classical manifold learning, the goal is to map the manifold-structured data points to low-dimensional representation space such that the manifold structure is preserved. There are two key ingredients: 1) Choosing a geometric property from the original data space to be preserved. For example, the local euclidean neighborhood (Belkin & Niyogil (003), or linear interpolation by neighboring data points (Roweis & Saul] 2000). 2) The embedding should not collapse to a trivial solution. To avoid the trivial solution, the variance of the embedding space is typically constrained in spectral-based manifold learning methods.
Manifold Clustering and Embedding. When the data should be modeled as a union of several manifolds, manifold clustering needed in addition to manifold learning. When these manifolds are linear, subspace clustering algorithms (Ma et al. 2007 Elhamifar & Vidal 2013 Vidal et al. 2016) can be used. When they are non-linear manifold clustering and embedding methods were proposed. They generally divide into 3 categories (Abdolali & Gillis 2021): 1. Locality preserving. 2. Kernel based. 3. Neural Network based. Locality preserving techniques implicitly make the assumption that the manifolds are smooth, and are sampled densely (Souvenir & Pless 2005 Elhamifar & Vidal 2011: et al. 2018). Additionally, smoothness assumption can be employed (Gong et al., 2012). Our method generalizes those techniques by realizing them with geometrical constraints. The success of kernel based techniques depends strongly on the suitability of the underlying kernel, and generally requires representation of the data in space with higher dimension than the data space (Patel & Vidal 2014). Deep subspace clustering methods, such as as et al. (2017); Zhang et al. 2019); Zhou al. 2018) jointly perform linear subspace clustering and representation of the data, and has the potential to handle high dimensional data effectively. However, it has been shown that most performance gains obtained by those methods should be attributed to an ad-hoc post-processing step applied to the self-expression matrix. Using neural networks only provide a very marginal gain compared to clustering the raw data directly using linear SC (Haeffele et al.] 2020). Our work differs from those techniques mainly in two aspects: i) While most of the previous methods were generative (autoencoders, GANs), our loss function is defined in the latent embedding space and is best understood as a contrastive method. ii) While previous methods use self-expression based SC to guide feature learning, ours uses MCR to learn the subspace features. Recently, some deep SC techniques also applied data augmentation (Sun et al.) 2019 Abavisani al. (020). However, in those works, data augmentation played a complementary role of improving the performance. In our method, data augmentation plays central role for enabling the identification of the clusters.
Self-Supervised Representation Learning. Recently, self-supervised representation learning achieved tremendous success with deep neural networks. Similar to manifold clustering and em- bedding, there are also two essential ingredients: 1) Data augmentations are used to define the domain-specific invariance. 2) The latent representation should not collapse. The second require- ment can be achieved either by contrastive learning (Chen et al., 2020). momentum encoder (He al. 20200 Grill et al. 2020) or siamese network structure (Chen & He 202I). More directly re- lated to our work is Tao et al. (202T). which proposed feature orthogonalization and decorrelation, alongside contrastive learning. Recently, variance regularization along was also successfully used to achieve principle 2) (Zbontar et al., 2021 Bardes et al.) 2021), attaining state-of-the-an SSL repre- sentation performance. Part of our method, the total coding rate (TCR) objective achieves a similar effect, see discussion in Appendix A.6 However, beyond self-supervised features, our algorithm additionally show strong clustering performance, and directly learns a meaningful latent space. The simultaneous manifold clustering and embedding in NMCE is also related to online deep clustering (Caron et al., 2018; 2020) method. Also notable is Deshmukh et al. (202I), where the concept of population consistency is closely related to the constraint functional discussed.
2

---

Under review as a conference paper at ICLR 2022
Clustering with Data Augmentation Our method use data augmentation to ensure correct cluster- ing of the training data. Although not explicitly pointed out, this is also the case for other clustering techniques (Shen al. 2021 Tsai et al. 22000] Li al. 202T). Our understanding of data augmen- tation also consistent to works that specifically study the success of data augmentations (HaoChen
et al. 2021
von Kügelgen al. 202T).
NEURAL MANIFOLD CLUSTERING AND EMBEDDING
3.1
PROBLEM SETUP
Let's assume data points x; E Rd are sampled from a union U"_ X; of manifolds X1, X2 .... Xn assign each data point the
As stated in Task the goal of manifold clustering and embedding corresponding manifold (clustering), as well as learning a coordinate for each manifold (manifold learning). To achieve this goal, we use neural network f. which learns to map a data point æ to the feature embedding E Remm and the cluster assignment E [1. n) ie. 2,C f(r). The cluster assignment shall be equal the ground-truth manifold assignment should parameterize the coordinates of the corresponding manifold X
To make the feature space easier to work with, one can enforce the additional separability require- ment that for any Xj, Xı: and + k: the feature vectors are perpendicular, Z; Ze. Here Zj denotes the embedding feature vectors of data points in Xj. and we define perpendicular between two sets in the following fashion: If 2; C Zj. 2u C Zv such that Vzj E Z zn E Zn. we have z4 o 0, then either Z, or 2 has zero measure.
In the following, we first argue that to make clustering possible with a neural network, one should define the additional geometric constraint that makes the manifold clusters identifiable. Second, we discuss how to implement the required geometric constraints and combine them with recently proposed joint clustering and subspace learning algorithm MCR² Yu et al. 2020 to achieve neural
manifold clustering and embeddng (NMCE).
X
Z
Clustering with neural network does not respect the manifold structure.
Addional constraint implemented via data augmentation allows clustering to respect the manifold.
Learned feature is orthogonal for different clusters.
Figure I: Adding small Gaussian noise data augmentation and requiring the augmented samples of the same point to be assigned to the same cluster and have similar embedding allows neural network to find the desired manifolds.
3.2 CLUSTERING ALWAYS INVOLVES IMPLICIT ASSUMPTIONS
Even the simplest clustering algorithms rely on implicit assumptions. For example, in k-means clustering, the implicit assumption is that the clusters in the original data space are continuous in terms of L2 distance. For Linear SC, the assumption is that data points are co-linear within each cluster. If a neural network is trained on example data to learn a cluster assignment function
not consider topological issues here and assume that all them are homeomorphic to Rd for some
d: UU to permutation since the training is unsupervised.

---

Under review as a conference paper at ICLR 2022
f(x), the resulting clustering will be arbitrary and not resemble the solution of k-means or linear SC clustering. This is because neural networks are flexible and no constraint is imposed on the clustering function force the result to respect the geometry of the original data. One example of this is shown in Figure | left panel, where a deep clustering method outputs a rather arbitrary clustering for the double spiral data.
To make the clusters learnable for a neural network, one needs to introduce constraints explicitly. In the example in Figure ! one can easily reason that, be able to separate the two spirals from each other, the clustering function needs to ensure that all close neighbors of a point from the data distribution are assigned to the same cluster, essentially the same assumption implicitly made in locality-basee manifold clustering algorithms (Abdolali & Gillis 2021). We formalize the notion of constraints to a constraint functional DJ)) (for a specific data distribution) that has the following property: All cluster assignment functions f that makes 1) attain its minimum value (assumed to be 0) and also cluster data points to the ground truth number of clusters will correctly cluster all data points. For example, we can construct functional D that takes value D 0 for all clustering functions that satisfy the locality constraint on the dataset, and > 0 otherwise. This notion of constraint function is completely general. For example, linear subspace clustering is recovered if the constraint function take value 0 and only if the found clusters are linear subspace.
In practice, one usually cannot optimize the neural network clustering function f subject to D 0, since the correct solution would have to be found at initialization. A more practical way to use the constraint function is to use the relaxed objective with weighting X:
L(f) Leckt (f) + A D(f)
(1)
where Lest is some objective function that will force f to cluster the dataset. With a suitable A, optimizing this objective leads learning of the correct clusters, by the assumption above. To achieve manifold clustering, one just needs to find the appropriate constraint functional.
3.3 SUBSPACE FEATURE LEARNING WITH MAXIMUM CODING RATE REDUCTION
Having introduced explicit constraints for learning the manifold clustering with neural networks, we still need a concrete algorithm for learning linear subspace-structtreed representation given the clustering (manifold learning). Fortunately, the recently proposed principle of Maximum Coding Rate Reduction (MCR²) Yuu et al. 2020) provides principled learning objective for this purpose. We denote the dataset (union of all manifolds) by X. and random variable that represent distribution on each manifold X; by X. For a certain encoder 2 = s(X) (without clustering output), E Tedem the Gaussian coding rate function defined to be:
R(z,e) = logdet(1 Cembcov((2 (cov(2))
demb 22
where cov denotes the covariance matrix function for a vector random variable: cov(Z) [=(=][=>"]] This function is approximately the Shannon coding rate of a multivariate Gaussian distribution given average distortion (Cover 1999), and can be motivated by ball- ball-packing argu- ment. However, we focus on its geometric implication and do not discuss the inoommation-hhorente aspect further. Readers interested the full introduction of this objective can refer to the original papers (Yu et al. 2020 Ma et al. 2007).
Suppose for that we also given the cluster assignment function c(x) that outputs the mani- fold index e(x) = i for E X,. We can then calculate the average coding rate for particular cluster i as R(Z,,e), e), where Z, = f(x)). The MCR" principle states that, to learn a subspace-structuree representation, one needs to optimize f by maximizing the difference between the coding rate of all of Z and the sum of coding rate for each cluster Z₁:
n1
AR(Z,c,e) R(2,e) Σ R(Z,,e)
It has been shown that MCR" guarantees the perpendicularity requirement in the problem setup above. Theorem A.6 in Yu et al. 2020) states that under the assumption that rank(Z,) is bounded, and the coding error is sufficietly low, maximizing the MCR² objective guarantees that Z; 1 Z; for any i + j, and the rank of each Z; is as large as possible. For full details and proof, see Yu et al. 2020)
4

---

Under review as a conference paper at ICLR 2022
3.4 NEURAL MANIFOLD CLUSTERING AND EMBEDDING
The MCR principle can be extended to the case where the labeling function c(z) is not given, but instead learned jointly by optimizing the MCR² objective. In this case, we fuse the clustering function into f: 2, f(x)), note that now 2; depends on I implicitly. However, as discussed carlier, one will not be able to find the correct clusters by using MCR alone, as is unconstrained in the data space. An additional constraint term D(f) has to be included to make the clusters identifiable. This leads to the Neural Manifold Clustering and Embedding (NMCE) objective:
DxMcR(f,c)<D -
())
is possible to replace the MCR² objective in by any other objective for subspace feature learning, however, this generalization is left for future work. Given suitable constraint functional D, the NMCE objective will guide the neural network to cluster the manifold correctly and also learn the subspace- struccuud feature, thus achieving manifold clustering and embedding. We explain in the next section how D can be practically implemented.
3.5 IMPLEMENTING CONSTRAINTS VIA DATA AUGMENTATION
Here we propose simple method to implement various very useful constraints. The proposed method uses data augmentations, a very common technique in machine learning, and enforce two conditions the clustering and embedding function f: 1. Augmented data points generated from the same point should belong to the same cluster. 2. The learnt feature embedding of augmented data points should be similar if they are generated from the same point. We use T(x) represent random augmentation of x: C fx)(x)), ć, = f(T (z)), then D(f) = (ppt))immz,, z'). Here, sim is function that measures similarity between latent representations, for example cosine similarity, or L2 distance. The requirements e = e' can be enforced by using average of e and c' to assign and x' to clusters during training.
For the double spiral example, the augmentation simply to add a small amount of Gaussian noise to data points. This will have the effect of forcing neighboring points in the data manifold to also be neighbors in the feature space and be assigned to the same cluster. The result of using this constraint in the NMCE objective can be seen in Figure | middle and right panel. For more difficult datasets like images, one needs to add augmentations to constrain the clustering in the desired way. The quality of clustering is typically measured by how well it correspond to the underlying content information (object class). Therefore, we need augmentations that perturbs style information (color, textures and background) but preserves content information, so that the clustering will be based on content but not style. Fortunately, extensive research in self-supervised contrastive learning has empirically found augmentations that achieves that very effectively (Chen et al. 2020 von Kügelgen et al. 2021).
3.6
MULTISTAGE TRAINING AND RELATIONSHIP TO SELF-SUPERVISED CONTRASTIVE LEARNING
In practice, we find that for all but the simplest toy experiments it is difficult to optimize the full NMCE objective (Eq. 4 from scratch. This is because the clusters are incorrectly assigned at the start of training. and the sum in Eq efffectccv cancels out the total coding rate term R(2.c), c). To achieve good performance, is critical to achieve high total coding rate, and high similarity between augmented samples in the feature space. We thus resort to a multistage training procedure, with the first stage always being optimizing the following objective:
LTcR(sçe) = - R(2,,) XD(5)
(5)
We call this the Total Coding Rate (TCR) objective, which is novel self-supervised learning objec- tive by itself. Through simple arguments (Appendix (A.6) one can see that this objective encourages the covariance matrix of 2 to be diagonal. Along with a similarity constraint between augmented
samples, this objective turns out to asymptotically achieve the same goal as VICReg (Bardes et al. (02T) and BarlowTwiii Twins (hontar al. 202T). We discuss this further in Appendix A.6
5

---

Under review as a conference paper at ICLR 2022
After training with the TCR objective, found that usually the feature 2 already possess approxi- mate subspace structure. For simple tasks, the features can be directly clustered with standard linear SC techniques such as EnSC (You al. 2016). For more difficult tasks, we found that the full NMCE objective performs much hetter.
4
RESULTS
We provide code to reproduce the double spiral toy example in the supplementary material, the full code will be released upon publication.
For the synthetic experiments, MLP backbone is used as backbone and two linear last layers are used to produce feature and cluster logits output, ELU is used the activation function. For all image datasets, standard ResNet with ReLU activation and various sizes used as the backbone. After the global average pooling layer, linear layer with 4096 output units, batch normalization and ReI.U activation used. After this layer, two linear heads are used to produce feature and cluster logits outputs. The number of clusters used is always equal the ground truth. For all experiments, two augmented samples, or "views", are used for each data sample in batch. Gumbel-Softmax (Jang et al. 2016) is used when the cluster assignment learned. Similar to Yu al. (2020), feature vectors are always normalized to the unit sphere. The constraint term D is always cosine similarity between two augmented samples. Cluster assignment probabilities and feature vectors are averaged between augmentations before sending into the NMCE loss. The regularization strength À is always determined empirically. Hyper-parameters and further details are available the Appendix
Table 1:
Clustering performance comparison on COIL20 and COIL100. Listed are error rates, NMCE is our method, see text for references for other methods.
Dataset
SSC
KSSC
AE+EDSC
DSC
S2ConvSCN
MIRDSC-DA
NMCE (Ours)
COIL20
14.83
24.65 47.18
14.79
5.42 30.96
2.14
1.79
0.0 11.53
COIL100
44.90
38.88
26.67
20.67
4.1 SYNTHETIC AND IMAGE DATASETS
For the toy example in Figure I data augmentation is simply adding small amount of noise, and full NMCE objective is directly used, the clustering output is jointly Iearned.
To verify the NMCE objective, manifold-structured data generated by passing Gaussian noise through two randomly initialized MLPs. Small Gaussian noise data augmentation is used to enforce the locality constraint. A two stage training process is used, the first stage uses TCR objective while the second stage uses full NMCE objective. Much like the toy experiment, the result is consistent with interpretation, Appendix A.I for results and details.
perform synthetic experiment by clustering mixture of
To compare NMCE with other subspace clustering methods, we apply it to COIL20 (Nene et al., 1996a) and COIL100 (Nene et al. 1996b), both are standard datasets commonly used in SC liter- ature. They consist of images of objects taken a rotating stage in degree intervals. COIL20 has 20 objects with total of 1440 images, and COIL100 has 100 objects with total of 7200 images. Images are gray-scaled and down-sample by 2x. For COIL20, a 18- 18-layer ResNet with 32 filters are used as the backbone, the feature dimension is 40. For COIL100, and 10-layer ResNet with 32 filters used, and feature dimension is 200. We determined the best augmentation policy for this experi- COIL20, and the same policy is applied COIL100. Details
ment by manual experimentation on the searched policy can be found in Appendix B For this experiment, we use EnSC (You et al.) 2016) cluster features learned with TCR objective. We found this procedure already performs strongly, and full NMCE objective is not necessary.
We compare with classical baseline such as Sparse Subspace Clustering (SSC) (Elhamifar & Vidal 2013) as well as recent deep SC techniques including (SSC(PPtel & Vidal 2014), AE+EDSC() al. 2014 DSC(ji et al. 2017). S2ConvSCN(Zhang al. 2019) and MI RDSC-DAC (DSCCDDAAAavaiisaaaa Abavisani

---

Under review as a conference paper at ICLR 2022
al. 2020). As can be in Table T1 our method achieves perfect result of 0.0 error rate for C01L20, and error rate 11.53 for COIL100, substantially outperforming previous state of the art of 1.79 and 20.67 emrrr rate. This indicates that NMCE can truly leverage the non-linear processing capability of deep networks, unlike previous deep SC methods, which only marginally outperform linear SC on the pixel space (Haeffele et al. 2020
4.2 SELF-SUPERVISED LEARNING AND CLUSTERING OF NATURAL IMAGES
Recently, unsupervised clustering has been extended to more challenging image datasets such as CIFAR-10, CIFAR-20 (Krizhevsky et al. 2009) and STL-10 (Coates al. 2011). The original MCR² paper (Yu et al. 2020) also performed those experiments, but they used augmentation in a very different way (See Appendix A.2 for a discussion). The CIFAR-IO experiment used 50000 images from the training set, CIFAR-20 used 50000 images from training set of CIFAR-10 100 with 20 coarse labels. The STL-10 experiment used 13000 labeled images from original train and test set. ImageNet-100 10 and ImagetNet--ogge used 13000 and 19500 images subset from ImageNet, re- spectively (Li al. 2021). We use ResNet-18 as the backbone to compare with MCR and use ResNet-34 as the backhone to compare with other clustering methods. In both cases, the feature dimension is 128. Standard image augmentations for self-supervised learning used (Chen et al.) 2020
Table 2: Supervised evaluation performance for different feature types and evaluation algorithms.
See text for details.
Model
Proj
Pre-fcature
Pool
Proj (16 avg)
Pre-feature (16 avg)
Pool (16 avg)
SVM kNN NearSub
0.911
0.889
0.895
0.922
0.905
0.929
0.904
0.851
0.105
0.910
0.800
0.103
0.898
0.902
0.903
0.903
0.909
0.911
4.2.1 TRAINING STRATEGY, SUPERVISED EVALUATION
Here we use a three stage training procedure: 1, Train with TCR objective. 2. Reinitialize the last linear projection layer and add cluster assignment layer. Then, freeze parameters in the backbone network and train the two new layers with full NMCE objective. 3. Fine tune the entire network with NMCE objective.
As discussed earlier, the first stage very similar to self-supervised contrastive learning. Therefore, we study features learned at this stage with supervised evaluation, the result with ResNet-34 is listed in Table 2 To evaluate the learned features, we use SVM, kNN and a Nearest Subspace classifier (NearSub). We SVM instead of linear evaluation training because we find that it more stable and insensitive to particular parameter settings. Here we compare three types of features, Proj: 128d output of the projector head. Pre-feature: 4096d output of the linear layer after backbone. Pool: the 2048d feature from the last average pooling layer. We find that averaging features obtained from images processed by training augmentations improves the result. Therefore we also compare performance in this setting and denote it by 16avg, as 16 augmentations are used per image. Images from both training and test are augmented. As can be seen from the table. without augmentation, SVM using the projector head feature gives the best accuracy. This is surprising, as most self-supervised learning techniques effectively use Pool features because the performance obtained with Proj features is poor. When averaging is used, SVM with Pool feature becomes the best performer, however, SVM and kNN with Proj feature also achieve very high accuracy. In Appendix, we further compared Proj and Pool features using SVM in cases where limited labeled data are available. In such cases, Proj feature significantly outperform Pool feature, showing that the TCR objective encourages learning of meaningful feature in the latent space unlike other algorithms, where the gap between performance of Pool and Proj features can be very large (Chen) et al. 2020

---

Under review as a conference paper at ICLR 2022
Table 3: Clustering performance comparisons. Clustering: clustering specific methods. SC: sub- space clustering methods. See text for details.
Models
CIFAR-10 NMI
CIFAR-20 NMI
STL-10
ImugeNet-110
ImageNet-Dugs NMI
ACC
ARI
ACC
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ACC
ARE
Clastering
k-meuns
0.525
0.589
0.276
0.130
0.084
0.028
0.192
0.125
0.061
0.241
0.119
0.057
0.105
0.055
0.020
Spectral
0.455
0.574
0.256
0.136
0.090
0.022
0.159
0.098
0.048
0.271
0.151
0.076
0.111
0,038
0.013
0.790
0.266
0.859
0.429
0.455
CRLC
0,799
0.679
0.634
0.425
0.416
0.263
0.818
0.729
0.682
MoCo Baseline
0.776
0.669
0.008
0.397
0.524
0:338
0.347
MICE
0.835
0.737
0.698
0.440
0.436
0.280
0.752
0.635
0.575
0.439
0.428
0,286
TCC
0.906
0.790
0.733
0.491
0.479
0.312
0.814
0.732
0.689
0.897
0.848
0.825
0.546
0.512
0.409
ConCURL
IDID
0.815
0.711
0.663
0.425
0.426
0.264
0.756
0.643
0.575
0.954
0.898
0.901
0.591
0.546
0.413
SC
MCRR-EnSC
0.347
0.491
0.290
MCR-EESC
0.653
0.629
0.438
MCR3-SENer
0.765
0.655
0.573
NMCE-Res
0.830
0.761
0.710
0.437
0.488
0.322
0.725
0.614
0.552
0.906
0.819
0.808
0.398
0.393
0.227
SMCE-Rest.
0.891
0.812
0.795
0.531
0.524
0.375
0.711
0.600
0.530
Figure 2: Visualization of principal components from subspace representation learned from CIFAR-
10. Rows in each panel are training images that has the largest cosine similarities with different principle components of that subspace. Similarity of images within each component is apparent. For more complete visualization, see Figure A.6
4.2.2 CLUSTERING PERFORMANCE
We compare clustering performance of NMCE after all 3 stages of training to other techniques. Due to space limits, we only compare with important baseline methods well highest perform- ing deep learning based techniques, rather than being fully comprehensive. For a more complete list of methods, use clustering Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Score (ARI) as metrics. For definitions, see Yu et al. (2020). for example. We list results in Table 3 K- means(MacQQuen et al. (967) and Spectral clustering et al. (002) results were adopted from Zhang et al. (2021) and Shen et al. (202T). we always selected the best result. For deep learning based techniques, we compare our ResNet-34 results with CC(Li et al. 2021). CRLC(Do et al.) 2021 MiCE et al. 2020), (She et al. 2021), ConCURL(Deshnuki et al. 2021), and IDo et al.) 2021). We did not include Park al. (2021) or Kinakh et al. (2021) since the former is a post processing step, and the later uses a special network architecture. We also sepa- rately compare ResNet- 18 result with features learned by MCR IMCR3-CTLL to be precise, Appendix A.2) and clustered by EnSC You et al.) 2016), ESC et al. 2018) or SENet(Zhang
for example [Shen et al. (2021). We follow the convention in the field and
al. 2021 b. They denoted by =CSC-ENSC, MCR and MCR- respectively. For
8

---

Under review as a conference paper at ICLR 2022
MCR2-EnSC. we adopt the result from Yu et al. (2020). since the result is higher and include all datasets, MCR2-ESC and MCR? SENet results are from Zhang et al. (021))
Compared with the result from MCR², method achieved substantial gain in all metries sidered. For example, the accuracy is improved by more than 6% on CIFAR-I0, and by 9% on CIFAR-20. This shows that used data augmentation in much more effective way than original MCR2. Our method can also outperform previous techniques which were specifically optimized for clustering, this happens on most CIFAR-I0 10 and CIFAR-20 metrics, for example.
In the experiments above, stage 1 learns pre-features that are approximately subspace-structureed. and stage 2 can be seen as performing subspace clustering on the pre-feature output, MCR2 was originally a subspace clustering objective (Ma et al. 2007). For CIFAR and STL-10 experiments, fine tuning the entire network with the NMCE objective (stage 3) improves performance by a small but significant amount, see Appendix A.3 Features learned from stage 1 of the training can also be directly clustered by methods like EnSC, but the results were rather poor for the image dataset. seems that NMCE is more robust way to cluster the learned features for the image experiments.
After all training stages, one can visualize the structure of each subspace by performing PCA and displaying the training examples that have the highest cosine similarity to each of the principle components. In Figure we show this for out of 10 clusters of CIFAR-10. 10. As can be seen, principle components in each cluster correspond to semantically meaningful sub-clusters. For example, the first row in "dog" cluster are mostly close-ups of white dogs. This shows that after training with the full NMCE objective, samples are clustered based content similarity in the latent space, instead of being distributed uniformly on the hyper-sphers as training with just the TCR objective or other self- supervised learning methods would encourage |Wang & Isola (2020); Durrant & Leontidis) (2021) Zbontar et al. 2021
Bardes et al. (2021): Zimmermann et al. 2021)
5 DISCUSSION
In this paper we proposed general method for manifold clustering and embedding with neural net- works. consists of adding geometric constraint to make the clusters identifiable, then performing clustering and subspace feature learning. In the special case where the constraint is implemented by data-aaggmentatinn, we discussed its relationship to self-supervised learning and demonstrated its competitive performance on several important benchmarks.
In the future, is possible to extend this method beyond data augmentation to other type of constraint functions, improves its performance using stronger subspace feature learning algorithm.
REFERENCES
Mahdi Abavisani, Alireza Naghizadeh, Dimitris N Metaxas, and Vishal M Patel. Deep subspace clustering with data augmentation. In NeurIPS, 2020.
Maryam Abdolali and Nicolas Gillis. Beyond linear subspace clustering: A comparative study of nonlinear manifold clustering algorithms. arXiv preprint arXiv:2103. 10656, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varinne-invariacce-ovvvaaa reg.aaaa for self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1333-1366 2003.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Computer Vision ECCV 2018 15th European Confer- ence, Munich, Germany, September 8- 14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notex in Computer Science, pp. 139-156. Springer, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.

---

Under review as a conference paper at ICLR 2022
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Yubei Chen, Dylan M. Paiton, and Bruno A. Olshausen. The sparse manifold transform. In Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10534- 10545, 2018
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 215-223. JMLR Workshop and Conference Proceedings, 2011.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Victor G Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. Solo-leam: A library of self-supervised methods for visual representation learning. arXiv preprini arXiv:2108.01775. 2021.
Aniket Anand Deshmukh, Jayanth Reddy Regatti, Eren Manavoglu, and Urun Dogan. Representa- tion learning for clustering via building consensus. arXiv preprint arXiv:2105.01289, 2021.
Kien Do, Truyen Tran, and Svetha Venkatesh. Clustering by maximizing mutual information across
views. arXiv preprint arXiv:2107.11635 2021.
Aiden Durrant and Georgios Leontidis. Hyperspherically regularized networks for byol improves feature uniformity and separability. arXiv preprint arXhv:2105.00923 2021.
Ehsan Elhamifar and René Vidal. Sparse manifold clustering and embedding. Advances in neural information processing systems, 24:55-63. 2011.
Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications.
IEEE transactions on pattern analysis and machine intelligence, 35(11):275--281,, 2013.
Dian Gong, Xuemei Zhao, and Gérard Medioni. Robust multiple manifolds structure learning. In International Conference on Machine Learning. PMLR, 2012.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Benjamin D Haeffele, Chong You, and René Vidal. A critique of self-expressive deep subspace clustering. arXiv preprint arXiv:2010.03997, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self self-supervised deep learning with spectral contrastive loss. arXiv preprint WrXiv:2106.04156 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterizzaton with gumbel-softmax. arXiv preprint arXiv:1611.01 2016.
Pan Ji, Mathieu Salzmann, and Hongdong Li. Efficient dense subspace clustering. In IEEE Winter Conference on Applications of Computer Vision, pp. 461-468. IEEE, 2014.
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep subspace clustering networks. arXiv preprint arXiv: (0X900709000508 2017.
Ian T. Jolliffe. Principal Component Analysis. Springer, 1986.
10

---

Under review as a conference paper at ICLR 2022
Zhao Kang, Chong Peng, Jie Cheng, and Qiang Cheng. Logdet rank minimization with application to subspace clustering. Computational intelligence and neuroscience, 2015, 2015.
Vitaliy Kinakh, Slava Voloshynovskiy, and Olga Taran. Scatsimelr: self-supervised contrastive learning with pretext task regularization for small-scale datasets. In 2nd Visual Inductive Priors for Data-Efficient Deep Learning Workshop, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In 202/ AAAI Conference on Artificial Intelligence (AAAT), 2021.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. JEEE transactions on pattern analysis and machine intelligence, 1466)1154--156, 2007.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium mathematical statistics and probability, volume 1.14, pp. 281-297. Oakland, CA, USA, 1967.
SA Nene, SK Nayar, and H Murase, Columbia object image library (coil-20). Technical Report, 005-96, 1996a.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-100). Technical Report, 006-96, 1996b.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In Advances neural information processing systems, pp. 849-856, 2002.
Sungwon Park, Sungwon Han, Sundong Kim, Danu Kim, Sungkyu Park, Seunghoon Hong, and Meeyoung Cha. Improving unsupervised image clustering with robust learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12278-12287, 2021.
Vishal M Patel and René Vidal. Kernel sparse subspace clustering. In 2014 ieee international conference on image processing (icip). pp. 3849-2853. IEEE, 2014.
Sam T Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2333-2555 2000.
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip HS Torr, and Ling Shao. You never cluster alone. arXiv preprint arXiv:2106.01908, 2021.
R. Souvenir and R. Pless. Manifold clustering. In International Conference Computer Vision (ICCV), pp. 648-653, 2005.
Xiukun Sun, Miaomiao Cheng, Chen Min, and Liping Jing. Self-supervised deep multi- sub- space clustering. In Asian Conference on Machine Learning, pp- 1001-1016. PMLR, 2019.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. clustering-friendyy representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Rep- resentations, 2021.
Tsung Wei Tsai, Chongxuan Li. and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations, 2020.
René Vidal, Yi Ma, and S Shankar Sastry. Generalized principal component analysis, volume 5. Springer, 2016.
Julius Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. arXiv preprint arXiv:2106.04611 2021.

---

Under review as a conference paper at ICLR 2022
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9939. PMLR, 2020.
Chong You, Chun-Guang Li, Daniel Robinson, and René Vidal. Oracle based active algorithm for scalable elastic net subspace clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3928-30222 0978-3997221 2016.
Chong You, Chi Li, Daniel P Robinson, and René Vidal. Scalable exemplar-bar based subspace clus- tering class-imbalanced data. In Proceedings of the European Conference Computer Vision (ECCV), pp. 67-83, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint ar 1708. 038888. 2017.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.0322. 032300, 2021.
Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang. Jun Guo, and Zhouchen Lin. Self-supervised convolutional subspace clustering network. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5473-5482, 2019.
Shangzhi Zhang, Chong You, René Vidal, and Chun-Guang Li. Learning self-expressive network for subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12393-12403) 2021.
Pan Zhou, Yunqing Hou, and Jiashi Feng. Deep adversarial subspace clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1596-1604, 2018.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint 102.08850 2021.
12

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL RESULTS AND DISCUSSIONS
A.1 SYNTHETIC EXPERIMENT
We verify the proposed manifold clustering and embedding algorithm by simple synthetic ex- periment. Ground truth data from a union two manifolds with dimension and 6 is generated by passing 3d and 6d iid. Gaussian noise through two randomly initialized neural networks with Leaky-ReLLL activation function (negative part slop 0.2). Data augmentation is Gaussian noise with manitude 0.1. Training used two stages, the first stage used only TCR objective, the second stage with full NMCE objective.
Two situations examined. One where the random neural network also has randomly initialized biases, which will cause the two manifold to be far apart, making locality constraint implemented by noise augmentation sufficient for identifying the manifolds. Another situation is when bias is not used, the two manifolds then intersect at O, where the density is rather high. This makes the manifolds not identifiable with only locality constraint.
Results are listed in Table A.I As can be seen, when the clusters are identifiable, NMCE is able correctly cluster the data points, as well as learn latent features that perpendicular between different clusters. At the same time, the feature is not collapsed within each cluster, since if so the average cosine similarity within cluster would be 1. When the clusters are not identifiable, they cannot be learned correctly.
Table A.I: Result from synthetic experiment. Accuracy is in (chance level is 50%), z-sim is the average absolute value of cosine similarity between feature vectors for different pairs of z. True Cluster: pairs of z are from different ground truth clusters. Found Clusters: pairs of z are from two different found clusters. Within Cluster: pairs of are randomly picked from the same found cluster, averaged between two found clusters.
Dataset
Accuracy
z-sim: True Cluster
z-sim: Found Clusters
z-sim: Within Cluster
Identifiable 100.0 69.8
0.017
0.017
0.503
Not-identifiable
0.717
0.287
0.770
A.2 RELATIONSHIP TO SELF-SUPERVISED LEARNING WITH MCR?
The original paper YYu et al. 2020) performed self-supervised learning using MCR² objective with- out any additional term. Their method treats different augmentations of the same image a self- labeled subspace. They used large number of augmentations (50) of each image, with only 20 images in each batch. The performance of this method is rather poor, which is expected based on our understanding. In this case, augmented images will form a subspace with certain dimension in the feature space, thus large amount of information about augmentations will be preserved in the latent space. Clustering can then utilize style information and not respect class information.
To improve performance, variant called MCR*-CTRI is developed, where the total coding rate term is down-scaled. This variant performs significantly better, and is also used in our comparison. This result is also expected, since decreasing the subspace term effectively contract different aug- mentations of the same image in the fcature space, making the feature better respect the constraint needed for correct clustering. However, since the total coding rate is not high in this case, the feature is not diverse enough to achieve good performance.
A.3 FINE-TUNING BACKBONE WITH NMCE OBJECTIVE
For CIFAR-10, CIFAR-20 and STL-10 10 experimenss, the first training stage already learns very strong self-supervised features, which is then clustered into subspaces in the second stage with backbone network frozen. The clustering performance is already quite good after this stage. In the third stage, the backbone is fine-tuned, which further improves clustering performance. In Table A.2 we show
13

---

Under review as a conference paper at ICLR 2022
the effect of fine-tuning backbone network CIFAR-10 and CIFAR-22 experiment with ResNet- 18. As can be seen, fine-tuning produces a small but noticeable gain in clustering performance for all metrics and both datasets. This indicates that using the full NMCE objective improve performance. If the optimization issue can he resolved, and the entire network is trained with the NMCE objective from scratch, the performance may be further improved.
Table A.2: Fine tuning backhone with NMCE objective. Results shown are from ResNet- 18. Fine tuning backbone improves result slightly but notably.
Model
ACC
NMI
ARI
CIFAR-10 before
0.819
0.743
0.690
CIFAR- 10 after
0.830
0.761
0.710
CIFAR-20 before
0.422
0.471
0.300
CIFAR-20 after
0.437
0.488
0.322
A.4 COMPARE POOL AND PROJ FEATURE ON LOW DATA CLASSIFICATION
Here compare SVM accuracy of Pool and Proj features from CIFAR-10 ResNet- 18 experiment. The feature averaged over 16 augmentations. The accuracy is plotted as a function of the number of labeled training samples used, see Figure A.
As can be seen, Proj feature clearly outperforms Pool feature when few labeled examples are avail- able. This makes Proj feature much more useful than Pool feature, since labeled examples are often scarce real applications.
Note that what shown here is obviously not the optimal way to labeled example, if further leverage the clustering information, accuracy should reach 90 with only 10 labeled examples.
Frw d2ta sib siediitttinn with and fonuuna on
0.2
0.0
252
922
520
010
400
950
Figure A.I: CIFAR-10 SVM accuracy plotted against number of labeled examples used for Pool and Proj feature from ResNet-18 experiment. Features averaged over 16 augmentations is used. Error bar is std. over random sampling of training examples.
A.5 EFFECT OF LAMBDA PARAMETER
Here we study the effect of X parameter that balances the constraint and subspace feature learning term. CIFAR-10 self-supervised accuracy with SVM and kNN evaluation on Proj feature is listed in Table A.3 As can be seen, the accuracy is reasonable in range of À spanning more than 3x low to high, indicating that the quality of the learned feature is not very sensitive this parameter.
Table A.3: Effect parameter
A
20
30
40
50
60
70
Proj SVM acc
0.899
0,902
0,903
0,902
0.903
0.903
Proj kNN acc
0.890
0.895
0.894
0.896
0.895
0.897
14

---

Under review as a conference paper at ICLR 2022
A.6 UNDERSTANDING THE FEATURE SPACE
Here we show that the TCR objective used in the first stage in our training procedure for CIFAR and STL.110 datasets theoretically achieve the same result as the recently proposed VICReg (Bardes) al. 2021) and BattowTwins(Zontarr The optimization target of the two techniques are both making the covariance matrix of latent vector Z approach the diagonal matrix. The first stage training using TCR essentially achieves the same result. To see this, we note the following property of the coding rate function (Kang et al. 2015): For any Z Rmmdd
al., 202I), both are self-supervised learning algorithms.
min(m.a]
logdet(I (I +2²
log of)
(6)
Where is the ith singular value of Z. Additionally, we have: Em (m, of = ||
F. which
follows easily from || tr(Z" Z). Since the function log(1 + x) is concave, the optimization problem max
2i=1 log(1 + r;)) given Σ Li = C' reaches maxima when all x are equal each other. Since we normalize the row of Z, || m. optimization of Equation 6 result in solution with uniform singular value, which is equivalent to diagonal covariance.
We could not successfully reproduce VICReg in our setup due to the large amount of hyper- parameters that needs to be tuned. Therefore we resort to the open- open-source library solo-learn (da Costa et al., 202T), which provided VICReg implementation. Running the provided script for VICReg produced accuracy of 1 on CIFAR-10. 10. We also implemented TCR objective in solo- learn library. Running TCR obtained accuracy of 92. All
and augmentations the same VICReg, except batch size is 1024 instead of 256, projection dimension is 64 instead of 2048. Larger batch size or smaller projection dimension didn't work for VICReg, so we stayed with the original parameters.
The covariance matrices of learned feature for SimCLR, VICReg and TCR computed over entire training set visualized in Figure A.2] For VICReg, first 128 dimension is visualized out of 2048. As can be seen, the diagonal structure is visible in SimCLR feature space, TCR feature space is the closest to diagonal matrix. VICReg feature space is also quite close diagonal, but the off-diagonal terms seems noisier. Additionally, we plot normalized singular values for VICReg projection space in Figure A.4 This can be compared to TCR and SimCLR singular values in Figure A.3 a). As can be seen, TCR achieves flatter singular value distributions than VICReg, neither SimCLR or VICReg are close to full rank in projection space.
We demonstrate subspace structure of feature space after clustering with full NMCE objective by plotting singular values of each learned subspace Figure A.3 b). Each subspace found are around rank 10. In all other panels of Figure A.3] we display samples whose feature vector has the highest cosine similarity to the top 10 principle components of each subspace. One can see that most prin- ciple components represent interpretable sub-cluster within each class (even if the sub-cluster is of a different class than the cluster). Same as in Figure 2 feature vectors calculated by averaging 16 augmented images are used.
a)
b)
C)
Figure A.2: a). b), c) Covariance matrix of feature vector computed over training set for TCR, SimCLR and VICReg. VICReg result is slightly noisier on the diagonal than TCR. VICReg result is the first 128 dimensions out of 2048, see text for details.
15

---

Under review as a conference paper at ICLR 2022
a)
b)
5025
6410
0.0
4081
Companents
Components
Figure A.3: a) Singular values of feature vector distribution for SimCLR and TCR. b) Singular values for subspaces learned after all training stages. The rest of panels: visualizing 10 training examples most similar principal components of each clustered subspaces.
B EXPERIMENTAL DETAILS
We list hyper-parameters for each experiment in Table B.1
16

---

Under review as a conference paper at ICLR 2022
0.8
0.6
0.2
0.0
500
1000
1500
2000
Figure A.4: Singular values of feature vector distribution for VICReg using ResNet-18. VICReg
uses 2048d feature space.
B. TOY SYNTHETICC EXPERIMENTS
The toy dataset consists of double spiral with radius approximately equal to 15 and Gaussian noise magnitude 0.05, samples are generated online for each batch. Data augmentation Gaussian noise with magnitude 0.05.
B.2 COIL-20 AND COIL-100
The augmentation policy we found with manual search on COIL20 is (all are from torchvision transforms): Random Horizontal Flip with p 0.5. 2. RandomPerspective with magnitude 0.3 and = 0.6 3. Colorlitter with manitude (0.8, 0.8, 0.8, 0.2). always applied. The entire dataset is passed as a single batch.
B.3 CIFAR-10, CIFAR 20, STL-10, IMAGENET-10 10 AND IMAGENET-DOGS
For CIFAR-10 10 and -1..R-20.
standard ResNet- 18 and ResNet-34 with 64 input filters. The first layer uses 3x3 kernel, and and no max pooling is used. For other experiments, we use standard ResNet- 18 and ResNet-34 with 5x5 first layer kernel and max pooling. For COIL--20 and COIL--100 experiment, 32 input filters is used, and the ResNet-10 10 obtained by reducing number of blocks in each stage in ResNet- 18 to 1. For full details,
code release.
For CIFAR-10 10, CIFAR-20 STL-10, ImageNet- 10 and ImageNet-Dogg experiments, we used LARS optimizer (You al. 2017), with base batch size 256, for other experiments we used Adam. We note that stage 2 and in the stage training process is quite sensitive to weight decay, a careful search of this parameter is usually required for good performance.
B.4 COMPUTATIONAL COST
All experiments involving ResNet-34 requires 8 GPUs, others can be done in I GPU,
Our objective doesn't add significant computational burden compared to neural-networks involved. The covariance matrix is computed within batch. For batch of latent feature vectors with shape [B, df, where B is batch size and is latent dimension. We first compute the covariance matrix with shape [d, d). with (B2dd) cost. Then the log determinant of this matrix is calculated, which we assume has (d3)) cost. The O(B2) scaling with respect to batch size is the same as most contrastive learning method such as SimCLR, which is known to scale to very large batch size.
17

---

Under review as a conference paper at ICLR 2022
Table B.1: Hyper-parameters for all experiments. Ir: learning rate. wd: weight decay. et coding error dg: dimension of feature output. A: regularization constant. bs: batch size. epochs (steps); total epochs trained, total steps trained if the entire dataset is passed at once. S1, S2, S3 denote training stages.
Model
Ir
wd
d₂
X
bs
epochs (steps)
Double Spiral
1e-3
le-6
0.01
6
4000
4096
30000
Synthetic
le-3
le-6
0.01
12
100
4096
3000
COIL-20
le-3
le-6
0.01
40
20
1440
2000
COIL-100
le-3
lc-6
0.001
200
20
7200
10000
CIFAR-10 ResNet-18 S1
le-6
0.2
128
50
1024
600
CIFAR-11 10 ResNet-18 S2
0.5
0.005
0.2
128
50
1024
100
CIFAR-10 ResNet-18
0.003
0.005
0.2
128
50
1024
100
CIFAR-10 10 ResNer-34:SI
le-6
0.2
128
50
1024
1000
CIFAR-10 ResNet-34 S2
0.5
0.001
0.2
128
1024
100
CIFAR-10 ResNet-34 S3
0.003
0.001
0.2
128
0
1024
100
CIFAR-20 ResNet-18 S1
1c-6
0.2
128
50
1024
600
CIFAR-20 ResNet-18 $2
0.5
0.001
0.2
128
0
1024
100
CIFAR-20 ResNet-18 S3
0.003
0.001
0.2
128
1024
100
CIFAR-20 ResNet-34
1e-6
0.2
128
50
1024
1000
CIFAR-20 ResNet-34 S2
0.5
0,002
0.2
128
0
1024
100
CIFAR-20 ResNet-34 S3
0.003
0.002
0.2
128
1024
100
STL-10 ResNet-18 S1
1e-6
0.2
128
30
1024
1000
STL-10 ResNet-18
0.5
0.002
0.2
128
1024
400
STL-110 ResNet-18 S3
0.0005
0.002
0.2
128
30
1024
400
STL-10 ResNet-34
1e-6
0.2
128
30
1024
2000
STL-10 ResNet-34 $2
0.5
0.002
0.2
128
0
1024
400
STL-10 ResNet-34
0.0005
0.002
0.2
128
30
1024
400
18


Reviews:
[{"review_id": "cgPBkuw0aZ", "reviewer": "Reviewer_SD19", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths & Originality: \n\nThe idea of combing \"Maximum Coding Rate Reduction\" with data argumentation techniques is interesting and seems reasonable. \n\nIn the experimental parts, this work have results on a number of real data sets, which can be considerate a fairly enough set up to support the effectiveness of a newly proposed algorithm. \n\n### Weakness: \n\nOverall it seems NMCE is like a relative less significant changes from the work from recent proposed \"Maximum Coding Rate Reduction (MCR2) (Yu et al., 2020)\" with some data argument techniques, and there is no interesting theoretical analysis behind the proposed NMCE work too, from this paper.\n\nAlternative methods: in the experimental results & section 4, there are a number works are cited and compared to NMCE but still lack some important references. For example in Table 1, for COIL data set, results from linear methods like SSC and some deep clustering methods are listed but there is a whole field of nonlinear manifold clustering is not mentioned here. Reference 1 and 2 listed below are two methods in the field of nonlinear manifold clustering and some in-depth discussion or comparison to them are quite helpful for reviewer to evaluate the contribution from NMCE. Indeed, there is one survey paper about nonlinear manifold clustering is cited in section 2 but this is not enough. \n\nTo the toy example, double spiral, it seems NMCE can get 100% accuracy as shown in Appendix A. However, only one 100% number is kind of less clear as we can imagine that if keep increasing the scale of Gaussian noise in the data argumentation step, we should except this 100% to go down. Or maybe we can say the key is how to define \"a small amount of noise\" as stated in the paper, and should be nice to see the performance from alternative methods on toy examples too, to gave people more insights, i.e., multi-linear subspace clustering methods clearly not work well for this toy example as we know. \n\n### Reference:\n1. Souvenir, R., & Pless, R, Manifold clustering. The10th International Conference on Computer Vision (ICCV 2005).\n2. Dian Gong, Xuemei Zhao, Gérard G. Medioni, Robust Multiple Manifold Structure Learning. ICML 2012.", "summary_of_the_paper": "This work proposed a general manifold clustering algorithm called Neural Manifold Clustering and Embedding (NMCE), which utilize Maximum Coding Rate Reduction (MCR2 ) as the objective function and data augmentation to enforce constrains.  \n\nIn the implementation stage, given that even the toy experiment is difficult to optimize with the full NMCE objective, a multistage training procedure is applied with the first stage actually trying to optimize the Total Coding Rate (TCR), which is another kind of self-supervised learning objective claimed in this paper. \n\nOn synthetic and real-world datasets COIL20, COIL100, CIFAR-10, CIFAR-20 and STL-10, NMCE achieved comparable and sometimes better results, compared to baseline methods or some alternative manifold clustering methods listed in the paper.  ", "summary_of_the_review": "Overall I feel this is a reasonable submission but seems not good enough for ICLR publication, so I gave \" marginally below the acceptance threshold \". ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aqy5DuGDr-", "reviewer": "Reviewer_HTFw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Manifold clustering is a very hard problem. Authors propose a solution for non-linear manifold clustering. The aim of the paper is to learn a representation space in which the manifolds are co-linear. The problem is quite interesting and there are practical applications. However, I have several concerns:\n- I believe two highly relevant papers are missing in the paper and these papers are\n[1] Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation (ICLR 2021)\n[2] Representation Learning for Clustering via Building Consensus (arxiv, May 2021)\nI am completely aware these papers don’t consider manifold clustering however they can be used in this setting as they are. I will lay out my concerns by referring to these two missing papers. \nAuthors rightfully highlights the importance of assumptions and constraints in their paper. One of the proposed constrained is orthogonality (Section 3.1). Both [a] and [b] utilizes orthogonality. [1] directly use originality in their loss and [2] uses random projections. \nAuthors are trying to learn one representation space for different manifolds. Similar objective can be achieved by consensus loss of [2]. \n\n- The objective given in eq 3 is interesting. In my understanding, objective wants to condense each manifold as much as possible (summation term) and all manifolds should cover the largest available space (first term). I think these conditions will be satisfied if all manifolds are equidistant from each other. This can be desired property however how realistic it is? For example, let’s consider the following classes {car, bus, motorbike, bicycle, lion, mountain lion, whale, lion}. Why should each manifold be equidistant from each other? One would like to see semantic structure in the learned representation. Would you please explain how will eq 3 type loss will handle this issue? \n\n-\tThe proposed loss depends on logdet (please see Eq 2). As far as I know, to calculate such a quantity, one needs to calculate the eigen values of the cov(Z). I am assuming that the authors are calculating this quantity for each batch. Since eigen values needs to get calculated by another algorithm, e.g. Gaussian elimination, \no\tdoes this mean the proposed loss cannot generalize to large batch sizes? Some of the self-supervised learning methods are trained by using 8192 batch size. \no\thow can one compute logdet in multiple gpus? Does one need to do this computation on one gpu?\n\nExperimental results:\n-\tWould you please compare your CIFAR-10, CIFAR-20 and STL-10 results with [1] and [2]? Please note that the results shown in [1] and [2] may be using different architectures. As far as I remember [1] use Resnet-34 and [2] use Resnet-18.\n-\tBoth [1] and [2] are end-to-end training methods. Especially [2] supplies better results for Resnet-18 setting. Would you please elaborate the advantage of multi-stage training given that end-to-end are suppling similar results?\n-\tWould you please extend your empirical studt to ImageNet-10 and ImageNet-Dogs.\n-\tWould you please supply some ablation studies e.g. impact of lambda parameter? \n\nI am looking forward to author responses and I am willing to change my assesment/score if the reponse(s) is(are) satisfactory.", "summary_of_the_paper": "Authors developed a method and training procedure for manifold clustering problems. The proposed solution inspired from information theoretic methods namely maximum rate reduction. Authors also support their claims with empirical results.", "summary_of_the_review": "Although authors proposed an interesting solution to a very hard problem, I believe proposed method and experimental work needs to be improved especially taking in to account missing references and supplying a detailed discussion. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YpbnEuRNN6", "reviewer": "Reviewer_dgFk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Fundamentally the work is based on the existing Maximum Coding Rate Reduction (MCR2) objective (Yu et al., 2020). Through their analysis, to achieve the goal of manifold learning in unsupervised fashion, in principle the authors emphasize the importance and necessity of geometry-awareness constraint to be added upon the MCR2 objective. The constraint is surprisingly implemented by the simple data augmentation for example of adding small noises to the given data. And then apply the intuition that the learnt feature embedding of augmented data points should be similar if they are generated from the same point as the geometric constraint. I think this is a very interesting and useful way to enforce the local structures in the data. The authors show a toy example with their code for the performance of the proposed principle, although no code for reproducing large experiments is provided. It is hard to say what performance could be given the high complexity of data while a simple data-augmentation strategy is used, although the past contrastive learning objective works well in practice. Of course, there is much theoretical gaurantee for the introduced constraint strategy. \n\nThe paper can be further improved by increasing readability.   For example I dont think Definition 1 is a good way to introduce Manifold Clustering and Embedding task.  A definition should be used for the rigorous, concise and accurate \"mathematical\" concepts. For example, the \"perpendicular\" may be written as a definition.  \n\nCan you give some discussion on the dimensions d1,d2,...,dn?  What is their role in the algorithm?  Should they be known or can be learned from the algorithm?  Otherwise, there is no need to present them.   It seems the algorithm has nothing to do with them.\n\nYou may add how cov(Z) is handled in the algorithm?  What is the algorithm complexity given that the model parameters are inside latent variable Z?", "summary_of_the_paper": "This paper enhances the performance of Maximum Coding Rate Reduction (MCR2) objective in subspace feature learning by incorporating a strategy of adding a manifold learning inducing constraint, aiming at a novel method for general purpose manifold clustering. They claim that the new model significantly outperforms autoencoder-based deep subspace clustering and achieves\nstate-of-the-art performance on several important benchmarks. ", "summary_of_the_review": "The proposed method builds upon the existing existing Maximum Coding Rate Reduction (MCR2) objective with a data augmentation as a constraint to enforce manifold learning. It seems that the performance is satisfactory comparing with other state-of-the-art methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VkYKkA0wOf", "reviewer": "Reviewer_aRVQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides an intuitive extension of the MCR^2 embedding approach by regularizing it with a \"concordant data augmentation embedding\" constraint. The authors reason that such a constraint is needed to \"[restrict] the large flexibility of neural networks\".\n\nHowever, the paper is imprecise in many of the statements made. A reference to \"additional geometric constraints that make the manifold clusters identifiable\" essentially reduces to proximity of samples and their noisy versions in the manifold embedding. $L_{clst}$ is described as \"some objective function that will force $f$ to cluster the dataset\" but again there is no practical example of such a function. The concept of a constraint functional $D(f)$ is also never specified for a practical setting. In the appendix the authors say \"the performance of [MCR^2] is rather poor, which is expected based on our understanding\" - this does not illuminate a reason for the reader or make it clear what \"CTRL\" brings in.\n\nIt is not clear why only a partial subset of the code that implements the experiments is provided with thee rest to be held for release \"upon publication\". Also there are no descriptions of the computation cost of the different approaches compared, including the proposed one.\n\nThe fact that the augmentation/regularization used in the numerical experiments is tuned by hand raises further questions about the robustness of the proposed approach. The authors should provide some discussion on this aspect of the implementation - this can be done in the appendix if needed.\n\nThe training procedures used in Section 4.2 appear to be ad-hoc - can the authors comment on their reasoning? Additionally, some exclusions in the numerical comparison seem arbitrary as well - if a method in the literature addressed the same problem considered here, why not compare against it? If there are differences in approaches they can still be mentioned to provide a contrast.\n\nWhy is there no numerical comparison in the experiments shown on the appendix? \n\nThere is a repeated typo (e.g., in Fig. 2 and Fig. A.3) captions \"principle components\" -> \"principal components\".", "summary_of_the_paper": "The paper considers the problem of clustering a dataset corresponding to a sampling of a union of nonlinear manifolds where each cluster shall correspond to an individual manifold. The proposed method leverages previous results on subspace clustering via neural networks and a penalty function based on concordant classification of samples and their augmented versions.", "summary_of_the_review": "While the extension of a supervised multi-manifold embedding to the manifold clustering problem is intuitive and interesting, there are several imprecise concepts underlying its description, and the carefully crafted implementation muddles whether the proposed approach can have wider applicability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VcrU_2jWvw", "reviewer": "Reviewer_zSx8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Paper strengths #\nThe paper suggests a loss function for learning manifold clustering and embedding by combining two ideas from the literature. First, building on a technical paper introducing the “maximal coding-rate reduction” principle which suggests that to learn a subspace-structured representation, one needs to maximize the difference between the coding rate of all clusters (pooled) and the sum of coding rates for each cluster. Second, building on the insight (shared with related works detailed in the “Clustering with Data Augmentation” section) that data-augmentation provides an important signal for manifold embedding, as preservation of class membership across augmented samples replaces preservation of local geometry from classic manifold learning. While both ideas are not new per se, it is interesting that the current paper outperforms both lines of work by using a combined loss function (named NMCE, as the paper initials), suggesting a fruitful synergy between those two ideas.\nThe optimization target allows using any deep network as a manifold embedding function in either an unsupervised or supervised manner. The unsupervised method (using a simpler loss function, named TCE) nicely resembles recent self-supervised methods, while the supervised method is used as fine-tuning of the unsupervised method and seems to work well.\n\n# Paper weaknesses #\nThe paper structure does not follow the logical structure presented in my summary: while the combined loss function is well described and justified, the two algorithms are presented as an afterthought. It is not clear why the unsupervised method is needed, while the supervised method is not presented at all. Restate the paper contribution to make the distinction between the loss function, the unsupervised, and the supervised methods.\nPresent the unsupervised method by itself (rather than as an empirical note in section 3.6) and elaborate on the comparison to self-supervised methods (currently in “additional results” appendix). Explain why the unsupervised method is needed (my guess is this is due to zeroing out of the NMCE objective if class assignments are randomly initialized)\nThe supervised method is not adequately described, which seems like a major omission. If I correctly filled in the blanks, a training set was used for minimizing the NMCE objective function using the ground-truth class assignment (i.e. only the embedding is learned), then clustering success is measured on a test-set with some unknown initialization of class assignment. This initialization may be important for the results if it is not random (e.g., by using the training set and finding the nearest neighbor). \nNo motivation is given for the role of such work in the ML landscape. The paper starts with a technical assumption (“a union of low-dimensional nonlinear manifolds”) rather than from an intuitive statement that (1) each sample comes from some class; (2) classes representation has a meaningful structure, conceptualized as a low-dimensional nonlinear geometry; (3) it is natural to ask if class-membership and a low-dimensional geometry can be learned simultaneously.\nImportant principles are hidden in technical definitions and the authors should aim to provide more intuition. Most notable are the mysterious references to MCR2. If “Gaussian coding rate” is a quantification of variance, “Maximum Coding Rate Reduction (MCR2)” supports maximizing the total variance while minimizing  “variance within clusters”. Similarly the “constraint functional D(f)” is presented in section 3.2 as a “completely generic” method and only in section 3.5 do we understand how to implement it by enforcing similar clustering of data-augmented samples. I find the generic presentation unhelpful while the use of data-augmentation to uncover manifold structure central to the paper.\nThe main method presented in the paper does not seem to work well on synthetic data sets, and the authors do not share the details. Summarize the difference between EnSC and the full loss functions and explain why “For simple tasks, the features can be directly clustered with standard linear SC techniques such as EnSC”.  Also, provide the (poor) result of the full algorithm.\n\n## Smaller issues ##\n (1) The vogue statement “This work shows that many constraints can be implemented by data augmentation” should be re-stated through the lens of self-supervised learning, e.g., “manifold local structure can be learned by enforcing constant class-assignment on data-augmented samples”\n\n (2) Highlight the difference from the related work in the “Clustering with Data Augmentation” section and state in advance that current work outperforms them.\n\n (3) The statement “we need augmentations that perturbs style information (...) but preserves content information, so that the clustering will be based on content but not style” seems weird as this is almost the definition of data augmentation.\n\n (4) The statement “This indicates that NMCE can truly leverage the non-linear processing capability of deep networks” seems false in the context of section 4.1, where the TCR objective is used and not the full NMCE objective.\n\n (5) In section 4.2.1 the role of stage 2 (“Reinitialize the last linear projection layer and add a cluster assignment layer … freeze parameters in the backbone network and train the two new layers with full NMCE objective”) is not clear. If I understand correctly, you don’t show results of stage 2 without stage 3 and you don’t justify that stage 2 is needed. You should provide some evidence stage 2 is helpful or if not skip it for the sake of simplicity.\n\n (6) I don’t understand the discussion which makes most of section 4.2.1, about using representations at different levels of the embedding network, and with or without averaging. It seems the authors had several options and made their choice, so this section and table 2 can go to the “additional results” appendix to justify the choice.\n", "summary_of_the_paper": "The paper “Neural Manifold Clustering and Embedding” proposes methods for utilizing a deep neural network for simultaneously learning the clustering of samples and a non-linear embedding into a low-dimensional “latent space” where each cluster representation is simple.\nThe main contributions of this paper are:\n\n (1) Proposing a novel loss function that combines the “maximal coding-rate reduction” principle and self-supervised view of how to use data-augmentation in clustering;\n (2) Proposing an unsupervised method for solving the problem from scratch using a deep neural network as embedding operation, with parameters learned to maximize “coding-rate reduction” while preserving class membership across augmented samples;\n (3) Proposing a supervised method for fine-tuning the results of the above method by minimizing the “coding-rate reduction” of each class while maximizing the overhaul  “coding-rate reduction” and preserving class membership across augmented samples.\n", "summary_of_the_review": "The paper combines two previous lines of work into a novel loss function which is then used in both supervised and unsupervised manners and achieves state-of-the-art results. It is currently below the acceptance threshold because the presented algorithms’ description and justification are lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received a majority voting of rejection, although the author response successfully convinced one reviewer to increase his/her score from 5 to 6. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Presentation**

The presentation of this paper needs huge efforts to further improve. Several reviewers and I suffered from difficulties to understand the motivation and challenges of this paper. It seems that Section 3.5 is the novelty part of this paper, but I failed to catch their points. 

**Contribution**

Two contributions points were claimed in this paper. (1) The combination of data augmentation and MCR$^2$. Without knowing the challenges in this paper, it is difficult to evaluate this point. Based on my current understanding (The presentation heavily affects my understanding), this point is very incremental. (2) The proposed method achieved state-of-the-art performance. This point is problematic. I will explain below.

**Related Work**

The authors failed to notice a huge body of manifold learning work and contrastive clustering work. Some state-of-the-art methods are not included for comparisons.

**Experimental Evaluation**

(1) Lack of state-of-the-art methods; (2) No standard deviation; (3) The experimental results are incomplete; and (4) It seems that the proposed method only achieved high performance on CIFAR-10 and CIFAR-20. I am not the person who requests the authors achieve the best performance on all the datasets. Everyone knows no algorithm always wins. But the authors should provide some analyses on the inferior performance for better understanding the model.

No objection from reviewers was raised to again this recommendation.
Decision: Reject