Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors developed a method and training procedure for manifold clustering problems. The proposed solution inspired from information theoretic methods namely maximum rate reduction. Authors also support their claims with empirical results.
main_review: Manifold clustering is a very hard problem. Authors propose a solution for non-linear manifold clustering. The aim of the paper is to learn a representation space in which the manifolds are co-linear. The problem is quite interesting and there are practical applications. However, I have several concerns:
- I believe two highly relevant papers are missing in the paper and these papers are
[1] Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation (ICLR 2021)
[2] Representation Learning for Clustering via Building Consensus (arxiv, May 2021)
I am completely aware these papers don’t consider manifold clustering however they can be used in this setting as they are. I will lay out my concerns by referring to these two missing papers. 
Authors rightfully highlights the importance of assumptions and constraints in their paper. One of the proposed constrained is orthogonality (Section 3.1). Both [a] and [b] utilizes orthogonality. [1] directly use originality in their loss and [2] uses random projections. 
Authors are trying to learn one representation space for different manifolds. Similar objective can be achieved by consensus loss of [2]. 

- The objective given in eq 3 is interesting. In my understanding, objective wants to condense each manifold as much as possible (summation term) and all manifolds should cover the largest available space (first term). I think these conditions will be satisfied if all manifolds are equidistant from each other. This can be desired property however how realistic it is? For example, let’s consider the following classes {car, bus, motorbike, bicycle, lion, mountain lion, whale, lion}. Why should each manifold be equidistant from each other? One would like to see semantic structure in the learned representation. Would you please explain how will eq 3 type loss will handle this issue? 

-	The proposed loss depends on logdet (please see Eq 2). As far as I know, to calculate such a quantity, one needs to calculate the eigen values of the cov(Z). I am assuming that the authors are calculating this quantity for each batch. Since eigen values needs to get calculated by another algorithm, e.g. Gaussian elimination, 
o	does this mean the proposed loss cannot generalize to large batch sizes? Some of the self-supervised learning methods are trained by using 8192 batch size. 
o	how can one compute logdet in multiple gpus? Does one need to do this computation on one gpu?

Experimental results:
-	Would you please compare your CIFAR-10, CIFAR-20 and STL-10 results with [1] and [2]? Please note that the results shown in [1] and [2] may be using different architectures. As far as I remember [1] use Resnet-34 and [2] use Resnet-18.
-	Both [1] and [2] are end-to-end training methods. Especially [2] supplies better results for Resnet-18 setting. Would you please elaborate the advantage of multi-stage training given that end-to-end are suppling similar results?
-	Would you please extend your empirical studt to ImageNet-10 and ImageNet-Dogs.
-	Would you please supply some ablation studies e.g. impact of lambda parameter? 

I am looking forward to author responses and I am willing to change my assesment/score if the reponse(s) is(are) satisfactory.
summary_of_the_review: Although authors proposed an interesting solution to a very hard problem, I believe proposed method and experimental work needs to be improved especially taking in to account missing references and supplying a detailed discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed a general manifold clustering algorithm called Neural Manifold Clustering and Embedding (NMCE), which utilize Maximum Coding Rate Reduction (MCR2 ) as the objective function and data augmentation to enforce constrains.  

In the implementation stage, given that even the toy experiment is difficult to optimize with the full NMCE objective, a multistage training procedure is applied with the first stage actually trying to optimize the Total Coding Rate (TCR), which is another kind of self-supervised learning objective claimed in this paper. 

On synthetic and real-world datasets COIL20, COIL100, CIFAR-10, CIFAR-20 and STL-10, NMCE achieved comparable and sometimes better results, compared to baseline methods or some alternative manifold clustering methods listed in the paper.  
main_review: ### Strengths & Originality: 

The idea of combing "Maximum Coding Rate Reduction" with data argumentation techniques is interesting and seems reasonable. 

In the experimental parts, this work have results on a number of real data sets, which can be considerate a fairly enough set up to support the effectiveness of a newly proposed algorithm. 

### Weakness: 

Overall it seems NMCE is like a relative less significant changes from the work from recent proposed "Maximum Coding Rate Reduction (MCR2) (Yu et al., 2020)" with some data argument techniques, and there is no interesting theoretical analysis behind the proposed NMCE work too, from this paper.

Alternative methods: in the experimental results & section 4, there are a number works are cited and compared to NMCE but still lack some important references. For example in Table 1, for COIL data set, results from linear methods like SSC and some deep clustering methods are listed but there is a whole field of nonlinear manifold clustering is not mentioned here. Reference 1 and 2 listed below are two methods in the field of nonlinear manifold clustering and some in-depth discussion or comparison to them are quite helpful for reviewer to evaluate the contribution from NMCE. Indeed, there is one survey paper about nonlinear manifold clustering is cited in section 2 but this is not enough. 

To the toy example, double spiral, it seems NMCE can get 100% accuracy as shown in Appendix A. However, only one 100% number is kind of less clear as we can imagine that if keep increasing the scale of Gaussian noise in the data argumentation step, we should except this 100% to go down. Or maybe we can say the key is how to define "a small amount of noise" as stated in the paper, and should be nice to see the performance from alternative methods on toy examples too, to gave people more insights, i.e., multi-linear subspace clustering methods clearly not work well for this toy example as we know. 

### Reference:
1. Souvenir, R., & Pless, R, Manifold clustering. The10th International Conference on Computer Vision (ICCV 2005).
2. Dian Gong, Xuemei Zhao, Gérard G. Medioni, Robust Multiple Manifold Structure Learning. ICML 2012.
summary_of_the_review: Overall I feel this is a reasonable submission but seems not good enough for ICLR publication, so I gave " marginally below the acceptance threshold ".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of clustering a dataset corresponding to a sampling of a union of nonlinear manifolds where each cluster shall correspond to an individual manifold. The proposed method leverages previous results on subspace clustering via neural networks and a penalty function based on concordant classification of samples and their augmented versions.
main_review: The paper provides an intuitive extension of the MCR^2 embedding approach by regularizing it with a "concordant data augmentation embedding" constraint. The authors reason that such a constraint is needed to "[restrict] the large flexibility of neural networks".

However, the paper is imprecise in many of the statements made. A reference to "additional geometric constraints that make the manifold clusters identifiable" essentially reduces to proximity of samples and their noisy versions in the manifold embedding. $L_{clst}$ is described as "some objective function that will force $f$ to cluster the dataset" but again there is no practical example of such a function. The concept of a constraint functional $D(f)$ is also never specified for a practical setting. In the appendix the authors say "the performance of [MCR^2] is rather poor, which is expected based on our understanding" - this does not illuminate a reason for the reader or make it clear what "CTRL" brings in.

It is not clear why only a partial subset of the code that implements the experiments is provided with thee rest to be held for release "upon publication". Also there are no descriptions of the computation cost of the different approaches compared, including the proposed one.

The fact that the augmentation/regularization used in the numerical experiments is tuned by hand raises further questions about the robustness of the proposed approach. The authors should provide some discussion on this aspect of the implementation - this can be done in the appendix if needed.

The training procedures used in Section 4.2 appear to be ad-hoc - can the authors comment on their reasoning? Additionally, some exclusions in the numerical comparison seem arbitrary as well - if a method in the literature addressed the same problem considered here, why not compare against it? If there are differences in approaches they can still be mentioned to provide a contrast.

Why is there no numerical comparison in the experiments shown on the appendix? 

There is a repeated typo (e.g., in Fig. 2 and Fig. A.3) captions "principle components" -> "principal components".
summary_of_the_review: While the extension of a supervised multi-manifold embedding to the manifold clustering problem is intuitive and interesting, there are several imprecise concepts underlying its description, and the carefully crafted implementation muddles whether the proposed approach can have wider applicability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper “Neural Manifold Clustering and Embedding” proposes methods for utilizing a deep neural network for simultaneously learning the clustering of samples and a non-linear embedding into a low-dimensional “latent space” where each cluster representation is simple.
The main contributions of this paper are:

 (1) Proposing a novel loss function that combines the “maximal coding-rate reduction” principle and self-supervised view of how to use data-augmentation in clustering;
 (2) Proposing an unsupervised method for solving the problem from scratch using a deep neural network as embedding operation, with parameters learned to maximize “coding-rate reduction” while preserving class membership across augmented samples;
 (3) Proposing a supervised method for fine-tuning the results of the above method by minimizing the “coding-rate reduction” of each class while maximizing the overhaul  “coding-rate reduction” and preserving class membership across augmented samples.

main_review: # Paper strengths #
The paper suggests a loss function for learning manifold clustering and embedding by combining two ideas from the literature. First, building on a technical paper introducing the “maximal coding-rate reduction” principle which suggests that to learn a subspace-structured representation, one needs to maximize the difference between the coding rate of all clusters (pooled) and the sum of coding rates for each cluster. Second, building on the insight (shared with related works detailed in the “Clustering with Data Augmentation” section) that data-augmentation provides an important signal for manifold embedding, as preservation of class membership across augmented samples replaces preservation of local geometry from classic manifold learning. While both ideas are not new per se, it is interesting that the current paper outperforms both lines of work by using a combined loss function (named NMCE, as the paper initials), suggesting a fruitful synergy between those two ideas.
The optimization target allows using any deep network as a manifold embedding function in either an unsupervised or supervised manner. The unsupervised method (using a simpler loss function, named TCE) nicely resembles recent self-supervised methods, while the supervised method is used as fine-tuning of the unsupervised method and seems to work well.

# Paper weaknesses #
The paper structure does not follow the logical structure presented in my summary: while the combined loss function is well described and justified, the two algorithms are presented as an afterthought. It is not clear why the unsupervised method is needed, while the supervised method is not presented at all. Restate the paper contribution to make the distinction between the loss function, the unsupervised, and the supervised methods.
Present the unsupervised method by itself (rather than as an empirical note in section 3.6) and elaborate on the comparison to self-supervised methods (currently in “additional results” appendix). Explain why the unsupervised method is needed (my guess is this is due to zeroing out of the NMCE objective if class assignments are randomly initialized)
The supervised method is not adequately described, which seems like a major omission. If I correctly filled in the blanks, a training set was used for minimizing the NMCE objective function using the ground-truth class assignment (i.e. only the embedding is learned), then clustering success is measured on a test-set with some unknown initialization of class assignment. This initialization may be important for the results if it is not random (e.g., by using the training set and finding the nearest neighbor). 
No motivation is given for the role of such work in the ML landscape. The paper starts with a technical assumption (“a union of low-dimensional nonlinear manifolds”) rather than from an intuitive statement that (1) each sample comes from some class; (2) classes representation has a meaningful structure, conceptualized as a low-dimensional nonlinear geometry; (3) it is natural to ask if class-membership and a low-dimensional geometry can be learned simultaneously.
Important principles are hidden in technical definitions and the authors should aim to provide more intuition. Most notable are the mysterious references to MCR2. If “Gaussian coding rate” is a quantification of variance, “Maximum Coding Rate Reduction (MCR2)” supports maximizing the total variance while minimizing  “variance within clusters”. Similarly the “constraint functional D(f)” is presented in section 3.2 as a “completely generic” method and only in section 3.5 do we understand how to implement it by enforcing similar clustering of data-augmented samples. I find the generic presentation unhelpful while the use of data-augmentation to uncover manifold structure central to the paper.
The main method presented in the paper does not seem to work well on synthetic data sets, and the authors do not share the details. Summarize the difference between EnSC and the full loss functions and explain why “For simple tasks, the features can be directly clustered with standard linear SC techniques such as EnSC”.  Also, provide the (poor) result of the full algorithm.

## Smaller issues ##
 (1) The vogue statement “This work shows that many constraints can be implemented by data augmentation” should be re-stated through the lens of self-supervised learning, e.g., “manifold local structure can be learned by enforcing constant class-assignment on data-augmented samples”

 (2) Highlight the difference from the related work in the “Clustering with Data Augmentation” section and state in advance that current work outperforms them.

 (3) The statement “we need augmentations that perturbs style information (...) but preserves content information, so that the clustering will be based on content but not style” seems weird as this is almost the definition of data augmentation.

 (4) The statement “This indicates that NMCE can truly leverage the non-linear processing capability of deep networks” seems false in the context of section 4.1, where the TCR objective is used and not the full NMCE objective.

 (5) In section 4.2.1 the role of stage 2 (“Reinitialize the last linear projection layer and add a cluster assignment layer … freeze parameters in the backbone network and train the two new layers with full NMCE objective”) is not clear. If I understand correctly, you don’t show results of stage 2 without stage 3 and you don’t justify that stage 2 is needed. You should provide some evidence stage 2 is helpful or if not skip it for the sake of simplicity.

 (6) I don’t understand the discussion which makes most of section 4.2.1, about using representations at different levels of the embedding network, and with or without averaging. It seems the authors had several options and made their choice, so this section and table 2 can go to the “additional results” appendix to justify the choice.

summary_of_the_review: The paper combines two previous lines of work into a novel loss function which is then used in both supervised and unsupervised manners and achieves state-of-the-art results. It is currently below the acceptance threshold because the presented algorithms’ description and justification are lacking.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper enhances the performance of Maximum Coding Rate Reduction (MCR2) objective in subspace feature learning by incorporating a strategy of adding a manifold learning inducing constraint, aiming at a novel method for general purpose manifold clustering. They claim that the new model significantly outperforms autoencoder-based deep subspace clustering and achieves
state-of-the-art performance on several important benchmarks. 
main_review: Fundamentally the work is based on the existing Maximum Coding Rate Reduction (MCR2) objective (Yu et al., 2020). Through their analysis, to achieve the goal of manifold learning in unsupervised fashion, in principle the authors emphasize the importance and necessity of geometry-awareness constraint to be added upon the MCR2 objective. The constraint is surprisingly implemented by the simple data augmentation for example of adding small noises to the given data. And then apply the intuition that the learnt feature embedding of augmented data points should be similar if they are generated from the same point as the geometric constraint. I think this is a very interesting and useful way to enforce the local structures in the data. The authors show a toy example with their code for the performance of the proposed principle, although no code for reproducing large experiments is provided. It is hard to say what performance could be given the high complexity of data while a simple data-augmentation strategy is used, although the past contrastive learning objective works well in practice. Of course, there is much theoretical gaurantee for the introduced constraint strategy. 

The paper can be further improved by increasing readability.   For example I dont think Definition 1 is a good way to introduce Manifold Clustering and Embedding task.  A definition should be used for the rigorous, concise and accurate "mathematical" concepts. For example, the "perpendicular" may be written as a definition.  

Can you give some discussion on the dimensions d1,d2,...,dn?  What is their role in the algorithm?  Should they be known or can be learned from the algorithm?  Otherwise, there is no need to present them.   It seems the algorithm has nothing to do with them.

You may add how cov(Z) is handled in the algorithm?  What is the algorithm complexity given that the model parameters are inside latent variable Z?
summary_of_the_review: The proposed method builds upon the existing existing Maximum Coding Rate Reduction (MCR2) objective with a data augmentation as a constraint to enforce manifold learning. It seems that the performance is satisfactory comparing with other state-of-the-art methods.
review_title: null

Meta-review: The paper received a majority voting of rejection, although the author response successfully convinced one reviewer to increase his/her score from 5 to 6. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Presentation**

The presentation of this paper needs huge efforts to further improve. Several reviewers and I suffered from difficulties to understand the motivation and challenges of this paper. It seems that Section 3.5 is the novelty part of this paper, but I failed to catch their points. 

**Contribution**

Two contributions points were claimed in this paper. (1) The combination of data augmentation and MCR$^2$. Without knowing the challenges in this paper, it is difficult to evaluate this point. Based on my current understanding (The presentation heavily affects my understanding), this point is very incremental. (2) The proposed method achieved state-of-the-art performance. This point is problematic. I will explain below.

**Related Work**

The authors failed to notice a huge body of manifold learning work and contrastive clustering work. Some state-of-the-art methods are not included for comparisons.

**Experimental Evaluation**

(1) Lack of state-of-the-art methods; (2) No standard deviation; (3) The experimental results are incomplete; and (4) It seems that the proposed method only achieved high performance on CIFAR-10 and CIFAR-20. I am not the person who requests the authors achieve the best performance on all the datasets. Everyone knows no algorithm always wins. But the authors should provide some analyses on the inferior performance for better understanding the model.

No objection from reviewers was raised to again this recommendation.
Decision: Reject