Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors verify the union of manifolds hypothesis (proposed in earlier literature) i.e, data lies on a disjoint union of manifolds of varying intrinsic dimensions. The authors provide empirical results to support the hypothesis. 
strength_and_weaknesses: Here are some strengths associated with the current work :

1) The authors work towards verifying the union of manifolds hypothesis.

Here are some weaknesses associated with the current work :

1) The authors in this work make many statements without quantifying them thus treating them as obvious facts. 
" ... Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. ..." 

The reviewer does not agree with this hypothesis from the author. Data can be lying on a single manifold with multiple basis vectors. Different classes can choose to use appropriate basis vectors and not use the rest or use them marginally. This does not imply that different sub-regions of space cannot have different factors of variation. 

"... We first prove that these deep generative models (DGMs) are incapable of modeling disconnected supports. We then argue that the class labels provided in our considered datasets identify connected components (i.e. different classes are disconnected from each other), and show that training a push forward model on each class outperforms training a single such model on the entire dataset, even when using the same computational budget: this improvement is a firm indicator that the support is truly disconnected,
thus strongly supporting our thesis."

The reviewer does not agree with the author's statement above. It is much more easier to train a model once the data has been adequately clustered as such. Training a unique model for each class as compared to a single model for different classes, the hypothesis classes associated are completely different. Thus the argument is fallacious. 

2) In case the classes are separable and are non-intersecting and it is possible to separate the classes adequately (as the authors discuss in Section 3.1) then it is straightforward to train models on each separate class. However this is an unrealistic expectation in the real world since training separate models can be futile unless we have a mechanism to perform perfect classification of any unseen instance which is never true. Training models for all classes allows us to use the common structure and basis to mutually help learning for different classes.

3) The authors work is specific to image datasets and not applicable in general, which the authors are kind enough to mention in their discussion. Similarly the authors claim to use agglomerative clustering in this current work. However as mentioned in the no free lunch theorem, there is no one clustering algorithm which can perfectly cluster all instances for all types of data. The authors could have made the work more useful and valuable by adding a discussion on this.

4) The authors in this work assume that the number of clusters is known which is never true in the real world setting. Also the authors' empirical results demonstrate very marginal gains. The authors do not discuss many intricate cases i.e., the case of intersecting manifolds and how to cluster them adequately. 

5) The novelty/contribution of the current work is pretty limited.
clarity,_quality,_novelty_and_reproducibility: The novelty of the work is pretty limited. The manuscript in its current form requires more work.
summary_of_the_review: The current work is pretty limited in novelty and the authors make many statements/assumptions which do not hold true in general. The work is focussed on image datasets and thus the results and conclusions might not generally applicable. The draft in its current form needs more work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studies the "union of manifold" hypothesis, a refinement of the manifold hypothesis (MH) wherein data is assumed to lie on a disjoint union of manifolds of varying intrinsic dimensionality. To test this hypothesis an empirical study on common image datasets  is carried out. An empirical study of clustered generative architectures, chosen to test the disjoint manifold scenario, is also performed.


strength_and_weaknesses: [+] Interesting and timely topic. MH has been very influential but it is perhaps time to consider more realistic variants.

[+] Empirical investigation seems well-thought out and carefully performed.

[+] Relevant theory (Prop. 1) on the inability of push-forward generative models to model disconnected supports of data distributions is included.

[+] Appreciate the inclusion of Section 4.2.2, where tension between this work's empirical results and prior theory is addressed.

[-] Empirical results appear marginal (though significant). For example in Table 1, the proposed clustered-VAE (C-VAE) outperforms on MNIST, FMNIST, and SVHN, but not CIFAR-10 and CIFAR-100. Arguably CIFAR-10/100 are more important, as they are more relevant to real data. I would like to see a fuller discussion of this point.

[-] Claims about empirical results sometimes appear overstated and not fully spelled out. For instance in Section 4.2.1 the authors write "we can see that our C-VAE (classes) performs better on most cases...". It would be better to explicitly spell out which cases perform better and which do not. Same for Appendix C.1 "We can see that, modulo a few exceptions..." Which exceptions? Are they important or not?


clarity,_quality,_novelty_and_reproducibility: The message of this work is clear and appears to me to novel. The writing about the empirical claims can be improved. The results appear reproducible after a quick look at the submitted code.

summary_of_the_review: The manifold hypothesis is a fundamental idea in machine learning. Given the enormous influential of MH, it is natural to investigate extensions which have greater relevance to real data. This work makes a valuable contribution to the literature by discussing a disjoint union variant of MH. This paper is an interesting topic and well-argued, although the strength of the evidence support appears small. Nevertheless I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims at the union of manifolds assumption in deep learning, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. The authors empirically verify this hypothesis on commonly-used image datasets.
strength_and_weaknesses: Strength:
1. The union of manifolds assumption is useful in deep learning.
2. The authors provide a lot of numerical results.

Weaknesses:
1. The paper neglects a lot of important related works. First, there have been numerous works on union of subspaces and manifolds but the author failed to acknowledge them. Union of subspaces or manifolds have been studied in many works of clustering and matrix completion. See the following: [1] Vidal. Subspace clustering. 2011. [2] Elhamifar. Sparse Manifold Clustering and Embedding. 2011. [3] Fan et al. Polynomial matrix completion for missing data imputation and transductive learning. 2020.
On the other hand, deep learning based clustering methods are also closely related to union of manifolds. See the following: [4] Zhang et al. Neural Collaborative Subspace Clustering. 2019. [5] Cai et al. Efficient Deep Embedded Subspace Clustering. 2022. There are even some works proposing to train multiple autoencoders to cluster data.

2. The evidence of union of manifolds has also been observed when using t-SNE or UMAP to visualize high-dimensional data.

3. In Section 4.2, Algorithm 1 is not clearly explained. For instance, how to do the initialization and form $\mathbb{P}^{(\ell)}$?

4. The title of the paper is too broad.

clarity,_quality,_novelty_and_reproducibility: The clarity is fine but the novelty is not sufficient.
summary_of_the_review: My major concern is that the novelty and contribution of the paper are not significant or at least not well demonstrated. It is not clear what challenges or open problems the paper solved. Comparing with existing work on union of subspaces and manifolds, the quality of the paper is below the bar of ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper's main goal is to empirically verify the Union of Manifold Hypotheses (UMH). UMH is the relaxed version of the more standard Manifold Hypothesis (MH) and has been previously used in the literature, including in applications of clustering. UMH differs from MH essentially in two main ways: (1) UMH assumes that the support is disconnected, and (2) UMH allows different connected components of the support to have different intrinsic dimensionalities. This allows different parts of the input space to have different (number of) factors of variation. Importantly, the paper focuses on the natural image datasets.

The authors start by providing empirical evidence (Section 3.1), supporting the fact that common natural image datasets have disconnected supports (in the pixel space). They also demonstrate that the class label provides a good proxy for identifying (some of) the connected components. Then they estimate the intrinsic dimension (similarly to the way done in [1]) for separate classes and confirm that they all have different values (Section 3.2). These empirical results provide the main body of evidence supporting UMH.

The authors go beyond verifying UMH and also make several attempts (although not very impressive) of exploiting UMH in the context of deep learning in the natural image domain. In Section 4.1 they consider classification for CIFAR100 and present a simple idea, inspired by UMH, that leads to *very modest* (but statistically significant) improvements in the test accuracy of Resnet-18 model. In Section 4.2 the authors consider deep generative models and demonstrate non-trivial improvements (in terms of FID scores on smaller datasets, such as MNIST, CIFAR10, CIFAR100, etc). 

All the reviewers agree that UMH is an important concept in the field of machine learning. They also agree that the paper is clearly written, exposition is very easy to follow, and that the empirical evidence is well structured and presented. The main concern of the reviewers was related to the significance of the contribution. In particular, most of the reviewers focused their attention on Section 4 and found the results presented there rather weak. 

After detailed discussions (including the video conference with all of the reviewers), I decided to support the acceptance of this paper. I believe that verifying (and empirically probing into) UMH, which is a notion actively used in some of the subfields of machine learning, is a significant enough contribution. As another argument in support of this decision, I want to mention that paper [1] undertook a line of research very similar in spirit (focusing instead on MH), was published, and is gaining quite a few citations. I think the current paper is a natural continuation of the line, started in [1]. More generally, I personally believe that as a field we should focus more on studying (empirically) the properties of our datasets (not only in the natural image domain). 

My positive recommendation is conditional. I ask the authors to perform a minor revision of their paper that will include the following points (apart from all of the other points discussed during the review and rebuttal phase):

(1) Make the title more specific. All the reviewers noticed (and I agree with them) that the authors focus exclusively on the domain of natural images. Even though it is quite plausible that UMH holds for many other natural domains, the only evidence presented in this paper addresses the natural images. One of the options proposed by the reviewers is "Verifying UMH for natural images”. Personally I like this option and think that it is a fair point.
(2) I will ask the authors to read their paper once again and tone down the claims, whenever necessary. For example, it has been noticed by the reviewers that UMH was already known in the field, and yet the authors say “we propose the union of manifolds hypothesis” in the introduction. Please, make sure not to overstate your contribution.

I trust the authors to make these modifications to the manuscript.
justification_for_why_not_higher_score: While I do feel that the paper deserves being accepted and do hope that machine learning community will focus more on works like this, at this time I feel that poster is the most suitable form of presentation for this paper. I think the topic is very interesting and will provoke many debates, questions, and skepticisms. All of those can be addressed directly during the poster presentation.
justification_for_why_not_lower_score: Considering everything written above, I made a decision that the paper deserves being accepted.
Decision: Accept: poster