Verifying the Union of Manifolds Hypothesis for Image Data
Authors: Bradley CA Brown
Anthony L. Caterini
Brendan Leigh Ross
Jesse C Cresswell
Gabriel Loaiza-Ganem
Venue: ICLR 2023 poster (2023)
Keywords: manifold hypothesis
geometry
generative models

Abstract: Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data. This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data. Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. To address this deficiency, we consider the union of manifolds hypothesis, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant. We also provide insights into the implications of the union of manifolds hypothesis in deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. Our code is available at https://github.com/layer6ai-labs/UoMH.

Published as a conference paper at ICLR 2023
VERIFYING THE UNION OF MANIFOLDS HYPOTHESIS FOR IMAGE DATA
Bradley C.A. Brown"
Anthony L. Caterini
Brendan Leigh Ross
University of Waterloo
Layer 6 AI
Layer 6 AI
beabrown&uwat www.uuuwwwwtrro.oooomom
anthony@layers..a
brendan
Jesse C. Cresswell
Gabriel Loaiza-Ganem
Layer 6 Al
Layer 6 Al
praseelayee6..o
gabriel@layer6.an
ABSTRACT
Deep learning has had tremendous success at learning low-dimensional representa- tions of high-dimensional data. This success would be impossible if there was no hidden dimenssiiin structure in data of interest: this existence posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the now-dimessional structure typically present in image data. Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space have different number of factors of variation. To address this deficiency, we consider the union of manifolds hypothesis, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. We empirically verify this hypothesis on commonly- used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant. We also provide insights into the impli- cations of the union of manifolds hypothesis deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this struc- ture improves performance across classification and generative modelling tasks. Our code is available at htttp::/gggtii com/ Late//aayerraaa wataaaeeraaaabb//aaaaaaaaaaaaaea UOMH
INTRODUCTION
The manifold hypothesis (Bengio et al., 2013) states that high-ddmensiioal data of interest often lives in unknown lower-dimensional manifold embedded in ambient space, and there is strong evidence supporting this hypothesis. From a theoretical perspective, it is known that both manifold learning and density estimation scale exponentially with the (low) intrinsic dimension when such structure exists (Ozakin & Gray/ 2009 Narayanan Mitter, 2010). while scaling exponentially with the (high) ambient dimension otherwise (Cacoullos) 1966). Thus, the most plausible explanation for the success of machine learning methods on
data is the existence of far lower intrinsic dimension, which facilitates learning datasets of fairly reasonable size. This is verified empirically by Pope et al. (202I). in which a comprehensive study estimating the intrinsic dimension of commonly-un image datasets is performed, clearly finding low-dimensional structure.
However, thinking of observed data as lying on a single unknown low-dimensional manifold is quite limiting, as this implies that the intrinsic dimension throughout the dataset is constant. we consider the intrinsic dimensionality to be the number of factors of variation generating the data, we can see that this formulation prevents distinct regions of the data's 's support from having differing quantities of factors of variation. Yet this seems to be unrealistic: for example, we should not expect the number of factors needed to describe Ss and is in the MNIST dataset (LeCun et al. 1998) to be equal.
To accommodate this intuition, in this paper we consider the union of manifolds hypothesis: that high-dimensional image data often lies not on a single manifold, but on disjoint union of manifolds
Work done during intersship Layer 6 AT.

---

Published as a conference paper at ICLR 2023
of different intrinsic dimensions While this hypothesis has motivated work in the clustering literature [Vidal 2011 Elhamifar & Vidal 2011 2012 2013; Zhang et al. 2019; Abdolali & Gillis, 2021] Cai et al. 2022), to the best of our knowledge it has never been empirically explored analogously to the way that Pope et al. ( probe the manifold hypothesis. In this work we carry out this verification on commonly-used image datasets, first by confirming their supports are disconnected, and then by estimating the intrinsic dimension on each component, finding that there is indeed variation in these estimates. In order to verify that the support of the data disconnected, we leverage pushforward deep generative models (Salmona et al. [022 Ross et al., 2022), which generate samples by transforming noise through a neural network G. We first prove that these deep generative models (DGMs) are incapable of modelling disconnected supports. We then argue that the class labels provided in our considered datasets approximately identify connected components (i.e. different classes are mostly disconnected from each other), and show that training pushforward model on each class outperforms training a single such model on the entire dataset, even when using the same computational budget: this improvement is firm indicator that the support is truly disconnected.
After empirically verifying the union of manifolds hypothesis, we turn our attention to some of its implications in deep learning. We establish that classes with higher intrinsic dimension are harder to classify, and guided by this insight, we also show that classification accuracy can be improved by more heavily weighting the terms corresponding classes of higher intrinsic dimension in the cross entropy loss. We also show that the same DGMs we used to confirm the disconnectedness of the data support, which we call disconnected DGMs, provide performant class of models which are competitive with their non-disconnected basclines across a wide range of datasets and models, and thus provide a promising direction towards improving generative models.
2 BACKGROUND AND RELATED WORK
Throughout our paper, we consider the setup where we have access to dataset D generated i..d. from some distribution P in a high dimensional ambient space X' R."
{zr [
Pushforward DGMs As mentioned in the introduction, we leverage DGMs in particular pushfor- ward DGMs - in order to verify the disconnectedness of the support of PP We call pushforward model any DGM whose samples X are given by:
Z Pz and X G(Z),
where Pz is a (potentially trainable) base distribution in some latent space Z, and G: Z X is a neural network. We highlight many popular DGMs fall into this category, including (Gaussian) variational autoencoders (VAEs) (Kingma & Welling [014 RRezende et al. 2014). normalizing flows (NFs) (Dinh et al.) 2017: Kingma & Dhariwal 201 8 Behrmann et al.||2019 Chen et al. 2019 Durkan ct al. 2019), generative adversarial networks (GANs) (Goodfellow et al. 2014), and Wasserstein
autoencoders (WAEs) (Tolstikhin et al. 2018).
Intrinsic dimension estimation If we assume that IP* is supported on the closure of a d-dimensional manifold embedded in X' for some unknown d < D, natural question is how estimate d from D We follow Pope et al. (2021) and use the Levina & Bickel (2004) estimator with the MacKay & Ghabramani (005) extension, given by:
T Te( log T, (ax4)
de = n(k -
(2)
where (z) is the Euclidean distance from x its j thh neighbour in D {x), and k is hyperparameter specifying the maximum number of nearest neighbours to consider. While other estimators have been recently proposed (Block et al. 2021 Lim et al., 2021 Tempezyk et al. 2022). we stick with 22) throughout this work as it is wel estabiidbdd in the literature. As we will see, popular image datasets exhibit different intrinsic dimensions in separate regions of data space.
'The disjoint union of d-dimensional manifolds d-dimensional manifold (L.c 2013): the possibility of having different intrinsic dimensions separates the union of manifolds hypothesis from the manifold hypothesis.
Requiring the support to be the closure of manifold (or union thereof) rather than the manifold itself is merely technicality due the formal definition of support, which is always a closed set. See Appendix A
2

---

Published as a conference paper at ICLR 2023
On the relevance of understanding low-dimensional structure As mentioned in the introduction, the manifold hypothesis provides intuition as to why deep learning has been so successful. One might naïvely believe its usefulness is merely conceptual, yet uncovering and exploiting the structure underlying data of interest is a highly relevant problem (Bronstein et al. (021) which has recently started receiving increased attention. (Pope al. (202T) use (2) to estimate the intrinsic dimension of image datasets, not only finding strong evidence of low-dimensional structure, but also uncovering that datasets of higher intrinsic dimension are harder to classify. Similarly, (Ansuini al. (2019) link the intrinsic dimension of the representations of deep classifiers to their classification accuracy. Furthermore, the low-dimensional structure of data not only affects supervised methods: for example, training high-dimensional density to model data lying on low-dimensional manifold through maximum-likelihoue can result in manifold overfitting (Dai & Wipf 2019) Loaiza-Ganem et al. 2022). a phenomenon resulting in IP* not being learned, even if an infinite amount of data was observed. Indeed, there is a rapidly growing body of work aiming to account for the manifold hypothesis within DGMs (Gemici al.] [2016] Rezende al. 2020; Brehmer & Cranmer (2020]
Caterini et al., [202]] Ross & Cresswell [021] Cunningham al., 2022 De Bortoli al. 2022 Ross et al., (0022). All these works
Mathieu & Nickel| 2020 Arbel et al.] 2021; Kothari et al.
highlight the relevance of properly understanding and accounting for the low-dimensional structure
of data, an understanding we improve upon in this work.
3
THE UNION OF MANIFOLDS HYPOTHESIS
We will assume that the support of IP, sP*)((P)), ) can be written as:
L supp P*) = L (MM
where Li denotes disjoint union, each M₁ is a connected manifold of dimension d(e) and cf(-) denotes closure in x We refer this assumption the union of manifolds hypothesis. Note that we make the assumption that each Mr is connected merely for notational simplicity, as we could collapse each union of manifolds of matching dimension into a single disconnected manifold, but this notation allows us to reason about different (MMM)) as connected components of interest. Note also that the standard manifold hypothesis is equivalent to adding the assumption that all dd()s are identical. In the rest of this section we empirically verify this hypothesis on various commonly-use image datasets in two parts: first by establishing the disconnectedness of sP" and then by estimating the intrinsic dimension on each of its (approximate) connected components.
3.1 VERIFYING THE HYPOTHESIS: DISCONNECTEEDNESS
Before empirically verifying that the support of commonly-used image datasets is disconnected, let us develop intuition as to why this might be the case. For example, on MNIST we can likely continuously transform any 2 into any other 2 without leaving the manifold of 2s. whereas attempting to similarly transform a 2 into an 8 would likely result in intermediate images that are neither 2s nor 8s. We conjecture that class labels, which are often available for our considered datasets, provide a sensible proxy for the connected components cl(M of supp(PP", precisely because one can conceive of continuously transforming any one member of any class into any other member of the same class without ever leaving the class.
In order to verify this conjecture, we leverage shortcoming of pushforward models, namely their inability to model disconnected supports, which we formalize in the following proposition. We use measure- theooreet notation, where the model from (I is given by the pushforward measure P2
Let Z and X' be topological spaces and G Z + X' be continuous. Considering Z and X' as measurable spaces with their respective Borel alalgebras, let PPz be a probability measure on 2 such that supp(IPz) is connected and Pz (supp((PP)) = =1 1 Then supp ( # Pz) is connected.
Proposition 1:
Proof sketch: From the formal definition of support, supp(G+PPz) need not be equal to G(supp((Pz)) so the proof does not immediately follow from the fact that continuous functions map
'The condition Pz P2z2ssuppppp (supp((22) (P))) 1 is highly technical and holds in all settings of practical interest, but can he violated in contrived counterexamples. See Appendix for discussion.
3

---

Published as a conference paper at ICLR 2023
VAE (Baseline)
D-VAE (Ours)
V- (Qurs)
10-
05-
06-
-05
-10
00
-0.5
1.0
-1.0
00
05
-0.5
10
Figure 2: Samples from various VAE models on synthetic dataset.
connected sets to connected sets. We show that supp (G F2) = (c(((s))))(P2))) and the result follows from the closure of connected sets being connected. See Appendix [A the formal proof.
The implication of Proposition is simple: if the support of the data is not connected (i.e. has multiple connected components), then pushforward models are insufficient to properly model the data, since in prac- tice supp(I'z) 2) is always connected. Intu- itively. this result rules out the possibility of G assigning probability to regions of G(supp(PPz) which connect different disconnected components the data sup- port supp(P"), ). as illustrated in Figure 1 We highlight that topological limitations of pushforward models have been studied before (Cornish et al., 2020 Salmona et al.. 2022, Ross al., 2022), although to the best of our knowledge not to the gener- ality of Proposition 1. Indeed, there are many methods aiming to better model dis- connected supports with pushforward mod-
G
Z
X'
@up
@upp(GGPP)
(upp(P2) (P')
Depiction of the inability of pushforward models to represent multiple connected components. If supp( (P*) given by the two blue connected compo- nents on X, then by continuity, G' cannot map supp(IP2 z) to this set, and so its image must contain some region (shown in green) connecting them. Proposition I shows that the model G2IP cannot learn to assign 0 probabil- ity this green region.
Figure 1:
els (Arora et al. 2017 Banijamali et al.] 2017: Locatello al. 2018]] Khayatkhoei et al. 2018 Dinh et al. 2019 Tanielian et al.) 2020 Luzi et al. 2020 Pires & Figueiredo (2200) Duan [222]] Ye & Bors] 2021). Figure 2] which we will use as a running example throughout this section, further visualizes this result, with the first panel showing a synthetic dataset obeying the union of manifolds hypothesis (two connected components of respective intrinsic dimensions and 2), along with a Gaussian VAE trained on this dataset on the second panel. The VAE fails to recover a disconnected support, further confirming this limitation of pushforward models. We include all experimental details pertaining to this section in Appendix D.I
As means empirically verify that sPP is disconnected, we introduce disconnected DGMs: rather than train a single pushforward model (P2z, G) on D as standard DGMs, we first partition the data as D - De. where contains only the datapoints belonging the 8th class, and we then train a pushforward DGM (1o Gz) on each cluster D,. In order to obtain a sample X from a trained disconnected DGM, we simply sample l with probability proportional the size of De, [De], and then sample from the corresponding DGM, which can be performed in memory-effciient way (i.e. without having load L models in memory) as described in Appendix [.22]
We see disconnected DGMs as the most straightforward modification that can be done to pushforward models so as to enable them to model disconnected supports. Disconnected DGMs also allow to straightforwardly compare against standard pushforward models in FLOP-equivalent manner that uses the same architectures (see Appendix B. for details). If we observe any improvement of disconnected DGMs over their standard counterparts, the simplest explanation for this behaviour would be that (P") is disconnected. The third panel of Figure 2 shows a disconnected VAE (denoted D-VAE), which correctly recovers two connected components (albeit not their intrinsic dimensions, which we will address later with the fourth panel), and was given the same computational budget as the VAE on the second panel: demonstrating that disconnected DGMs show improvements
4

---

Published as a conference paper at ICLR 2023
Table 1: FID scores. We show means and standard emoos across runs.
Model
MNIST
FMNIST
SVHN
CIFAR-10
CIFAR- 100
WAE
19.1 + 2.1
51.2 2.4
89.8 12.2
146.7
145.0 = 0.7
D-WAE (random)
23.7
3.6
77.6 1.7
154.7
146.5 = 3.2
D-WAE (classes)
0.2
36.8 1.4
83.7 15.2
133.3 1 0.5
209.9 = 3.5
WAF (GMM)
152.7 + 19.1
97.6 + 20.9
276.1 + 8.6
253.6 + 17.1
250.3 - 5.0
Conditional WAE (classes)
9.9 + 0.2
36.7 0.4
172.9 + 122.2
142.9
206.0 = 36.2
Conditional WAE (clusters)
10.0 0.3
3.7
54.1 + 1.6
140.7 11
259.5 = 2.8
D-WAE (clusters)
11.4 0.3
35.3
97
139.2
134.3 = 1.8
over their standard counterparts when data disconnected, thus providing more evidence for the validity of our test of disconnectedness.
The top half of Table I shows analogous results the more realistic setting of image datasets. We use the FID score (Heusel et al., (017)) (lower is better), a commonly-used sample quality metric, to measure performance on the MNIST, FMNIST (Xiao et al. 2017). SVHN (Netzer et al. 201T). CIFAR-10, 10, and CIFAR- 100 (Krizhevsky et al. 2009) datasets. We can see that disconnected WAES (indicated as "D-WAF WAE (classes)") "") consistently outperform WAE strongly supporting our hypothesis that supp (P*) is indeed not connected in these datasets. We highlight once again that disconnected WAFs were given the exact same computational budget as WAFs (both in terms of floating point operations and memory usage) ensure fair comparisons. Nonetheless, disconnected WAEs do have more parameters. In order to ensure that the added parameters are not the reason that disconnected WAEs perform better, we perform an ablation, where we train disconnected WAE but use random labels instead of class labels (indicated as "D-WAE (random)"). We can see that disconnected WAEs using labels are still the best performing option, once again providing strong empirical evidence to the claim that one can indeed think of classes of these datasets as approximating different connected components of (P*), Finally, we point out that these results are not exclusive to WAFs: we carried out the same comparisons using VAEs, and found the conclusions to be the same, further backing our claims. Due to space constraints, we present these results in Table in Appendix C.
3.2 VERIFYING THE HYPOTHESIS: VARYING INTRINSIC DIMENSIONS
So far we have empirically verified that commonly-use image datasets have disconnected supports and that class labels provide a good proxy for the corresponding connected components. We now confirm that these disconnected components have varying intrinsic dimensions. We achieve this by applying (2) to each Dd (the subset of the data corresponding to the 8th class) separately: observing similar estimates across different values of would support the manifold hypothesis, whereas observing different values would instead support the union of manifolds hypothesis.
Figure 3 shows, for different values of the hyperparameter k. boxplots of the values d.0 for class labels ( 1, ... L in our considered datasets: MNIST, FMNIST, SVHN CIFAR-10, CIFAR-100, and ImageNet (Russakovsky et al. 2015). Two relevant patterns emerge. First, within each dataset, results are mostly consistent across different choices of k so that any conclusions we draw not caused by a specific choice of this hyperparameter. Second, for all datasets except SVHN, we can observe wide range of estimated intrinsic dimensions. These results empirically verify that the union of manifolds hypothesis is more appropriate way to think about images than the manifold hypothesis. Additionally, we estimate the variance of these estimators through subsampling in Appendix C.2] ensuring that the observed variability of estimated intrinsic dimensions is not due to chance. We highlight that (2) is known to underestimate the true intrinsic dimension when the dataset is not large enough, which might in principle affect our estimates on CIFAR- 100 and ImageNet, since these datasets have fewer datapoints per class. Strong consistency across & on C00 however suggests this is not concern for this dataset; and even if the estimates obtained on ImageNet do exhibit slightly more variation across k; the overall range estimated intrinsic dimensions remains consistent across k. In other words, these results still strongly support
'Except on CIFAR- 100, which we hypothesize is due to each class having too few datapoints. We provide further evidence that CIFAR-100 also has disconnected support in suuuscccctionn 4.2
5

---

Published as a conference paper at ICLR 2023
Intrinsic dimension range over classes
==
k=
k 20
35
Full dataset estimate
30
20
MNIST
FMNIST
SVHN
CIFAR-10
CIFAR-100
ImageNet
Figure Boxplots showing variability of intrinsic dimension estimates across classes.
the union of manifolds hypothesis. We take granular look into these estimates in Appendix where we show further choices of l: as well as the intrinsic dimensions of individual classes.
EXPLOITING THE UNION OF MANIFOLDS HYPOTHESIS
Now that we have empirically verified the union manifolds hypothesis on images-both the disconnectedness the support of the data and its varying intrinsic dimensions-we turn attention to the benefits brought to deep learning by being aware of the union of manifolds structure present in observed data. We see the results this section as sensible and promising first step towards realizing these benefits, and hope to encourage the community to further explore them.
4.1 SUPERVISED LEARNING: CLASSIFICATION ACCURACY AND INTRINSIC DIMENSION
Pope et al. 2021) showed that datasets of high intrinsic dimension are harder to classify. Here we provide a more granular look, by showing that classes of higher intrinsic dimension are harder to classify. We train 3 classifiers on CIFAR- 100 (see Appendix D.2 for details): VGG-19 19 (Simonyan Zisserman 2015), ResNet-18, and a ResNet-34 (He et al.) 2016). We focused on CIFAR- here as other datasets considered this work were either too simple to classify (MNIST, FMNIST, SVHN, CIFAR-10), or produced less dessrellable intrinsic dimension estimates (ImageNet). For each class, compute its classification accuracy and plot this against its estimated intrinsic dimension in Figure We see that, consistently across classifiers, there intrinsic dimension and classification accuracy. We also quantitatively compare intrinsic dimension and classification accuracy by computing their correlation, along with p-value for independence with f-test: we find that the negative correlation is significant (see figure caption). In other words, the higher the intrinsic dimension of class, the harder it is classify. While clearly intrinsic dimension does not fully explain accuracy, this correlation suggests the following intuition: learning useful representations for classes with higher intrinsic dimension requires learning more factors of variation and is thus more challenging.
inverse relationship between estimated
To check if this insight can help improve classifiers, we train the ResNet- 18 in two different ways (see Appendix D.2 for ex- perimental setup): in the first, we use the standard cross entropy loss, and the sec- ond, weight the terms corresponding each class in the cross entropy loss propor-
Table 2: Means and standard errors of ResNet- 18 accu- racy CIFAR- 100 across 5 runs. Test accuracy
Weights
Standard
61.38% + 0.17%
Proportional to intrinsic dimension
61. 71.77% + 0.20%
tionally their intrinsic dimension. This reweighting of the loss focuses classes of higher

---

Published as a conference paper at ICLR 2023
VGG-19
ResNet-18
ResNet-
CIFAR 100 test accuracy
0.8
0.6
0.4
10.0
12.5
15.0
17.5
20.0
10.0
12.5
15.0
17.5
20.0
10.0
12.5
15.0
17.5
20.0
Intrinsic dimension
Intrinsic dimension
Intrinsic dimension
Figure 4: Class intrinsic dimension versus test accuracy on CIFAR-100, along with a least-squares regression line. Correlation coefficients for each model are -0.243 + 0.015, -0.269 + 0.012, and -0.274 + 0.007, respectively (means and standard errors over 5 runs). Corresponding p- p-values, 0.008 = 0.002, 0.019 = 0.008, and 0.006 + 0.001, show the relationship is significant.
intrinsic dimension, as we have just shown these are harder to classify. Results are shown in Table 2 where we can see that this very simple change to the cross entropy loss marginally (but significantly, as error bars do not overlap) improves the accuracy of the network, providing an essentially "free" improvement, given the low computational overhead of estimating intrinsic dimension. Note that we did not perform any data augmentation scheme so as to not affect the intrinsic dimension of the data.
4.2 UNSUPERVISED LEARNING: DISCONNECTED DGMS THROUGH CLUSTERING
In subsection 3.1 we used disconnected DGMs to show that the support of image datasets is dis- connected. Here we show that despite their simplicity, disconnected DGMs are performant models which can provide improvements over competing alternatives. We highlight that the aim of these experiments is not to achieve statc-of-the-e performance, but rather to show that disconnected DGMs outperform comparable dinddisconeeetd models, emphasizing the relevance of properly accounting for the union of manifolds structure present in data. To this end, we introduce fully-unsupervised version of disconnected DGMs, where we run a clustering algorithm to partition the dataset instead of using class labels. The idea behind this modification is simply that if class labels are unavailable, clustering algorithm might manage to recover connected components of the data support as clusters, at least approximately. Our training and clustering procedures are detailed in Appendix B.1
We highlight once again that we use the same computational budget for training disconnected DGMs as for their standard counterparts. The only computational overhead is running the clustering algorithm itself and more disk space is required to store the model - but the number of floating point operations and memory usage throughout training is identical (see Appendix B.I All experimental details relevant to this section are included in Appendix D.I
4.2. BETTER MODELLING OF DISCONNECTED SUPPORTS
As already showed in subsection 3.1 D-WAEs outperform standard WAEs. The comparisons have discussed so far were however meant to confirm the union of manifolds hypothesis, and remain unfair as comparisons of generative performance since the D-WAE (classes) model has access class labels. The bottom half of Table 3 includes additional comparisons to show that D-WAE: are not just useful as way of confirming the disconnectedness of the data support, but that they are also an effective way of accounting for this structure. We include comparisons against standard conditional WAEs, which also have access to class labels (indicated as "Conditional WAE (classes)"), but as additional inputs to their neural networks (Sohn et al. 2015), rather than having separate models for each class like D- D-WAEs. We can see that conditional WAES (classes) outperform our D- D-WAEs (classes) only on MNIST, highlighting that D-WAEs are performant DGMs, despite their simplicity. Importanly, while we believe that D-DGMs are better than conditional models for verifying the disconnectedness of the data support (as they allow for FLOP and memory-equivalent comparisons), conditional WAEs outperforming WAEs provides further evidence supporting the union of manifolds hypothesis. Note that conditional WAES outperform WAEs on CIFAR- 100, which also supports previous conjecture that the poor performance of D- WAES on this dataset is due to having too few datapoints per class, rather than the union of manifolds hypothesis not holding.
7

---

Published as a conference paper at ICLR 2023
We also show results for the unsuperisdd version of the D-WAE (indicated as "D-WAE (clusters)"), and we can see that it not only outperforms the standard WAE, but that it remains competitive with the same conditional WAE, now conditioned on cluster membership rather than class labels for a fully-unsupervisedd baseline (indicated as "Conditional WAF (clusters)"). Finally, we also include a comparison against WAE whose prior P2 is given by mixture of T. Gaussians (indicated as "WAE (GMM)") instead of a standard Gaussian as in the other models. Having a multimodal I is a popular strategy in pushforward models to better model multimodal target distributions (Nalisnick et al. 2016:; Dilokthanakul et al. 2017: 7: Jiang et al. 2017: |en- Yosef & Weinshall 2018 Izmailov et al. 2020 Cao et al. 2020 Smieja et al. 2020; Potapezynski et al. 2020). We can see that our D- D-WAE models outperform WAE (GMM) in all cases: this is not too surprising in light of Proposition 1, since the support of Pz remains connected when using mixtures, even if is now multimodal. Once again, we omit comparisons with other pushforward models here due to space constraints, but highlight that Appendix [. includes analogous comparisons with VAES in T where we see the exact same behaviour disconnected DGMs outperforming or remaining competitive with naive conditioning and using mixtures as base distributions.
4.2.2 BETTER MODELLING OF VARYING INTRINSIC DIMENSIONS
While so far we have only focused on the ability of D-DGMs to model disconnected supports, our method also easily allows modelling varying intrinsic dimensions, thus fully accounting for the union of manifolds hypothesis. In order to do this, we simply set different dimensions for the latent spaces Ze of each of the L DGMs. In particular, we use (2) to obtain estimate di of the intrinsic dimension of M2 using De, and then set 2g- = Rd= Any observed improvement of disconnected DGMs with varying d= s over disconnected DGMs with fixed de's s can thus be attributed to having accounted for varying intrinsic dimensions. For each of the L DGMs (I%) Gr). we use two-step models (Dai& & Wipf 2019 Loaiza-Ganem al. 2022), which were specifically designed to properly model suppoold-supporte data by avoiding manifold overfitting: we first train an autoencoder-type model as first step, whose decoder is G and then train a DGM p/) on the -in-dmnesio representations obtained by running Dj through the encoder. In other words, we use a disconnected version of two-step models as a way to account for varying intrinsic dimensions. The fourth panel of Figure 2llluustrttes the benefits of this approach by training a disconnected two-step VAE (indicated as "D-VAE+VAE"). This model is trained by clustering the data to obtain its connected components, estimating the respective intrinsic dimensions 2 and 1, and then training VAE+VAE model each of these clusters. In the first cluster (of intrinsic dimension 2), the first VAE obtains 2-dimensional representations, and the second VAE learns the distribution of these representations. The same is done for the second cluster, except the first VAE obtains 1-dimensional representations.
In Table in Appendix C.I carry out analogous comparison image datasets using VAE+NF models. Surprisingly. we found that while both versions of disconnected DGMs (i.e. with fixed or varying intrinsic dimensions) outperformed non-disconnected two-step models once again confirming the conclusions about better modelling of disconnected supports setting intrinsic dimensions differently on every cluster did not improve performance over keeping intrinsic dimension constant across clusters. At a first glance this might appear to imply that there is no benefit to accounting for varying intrinsic dimensions, seemingly contradicting the theoretical result of Loaiza Ganem et al. 2022 Theorem 1), which states that manifold overfitting will occur whenever intrinsic dimension is overspecified. However, upon closer inspection of the proof of Theorem 1 in Loaiza Ganem et al.) (2022), we can see that the rate at which manifold overfitting happens depends on the difference between ambient and intrinsic dimension] This observation explains both the good empirical results Loaiza Sariiaa--neemmmemm et al. (2022) and our aforementioned results: the former are driven by the difference between ambient and intrinsic dimension (on the order of hundreds/thousaands)) while the latter by the error between the true and estimated intrinsic dimension (on the order of ones/tens). While a partially negative result, we provide further insight into the connection between the low-dimensional structure of the data and DGMs.
In order to verify that properly setting varying intrinsic dimensions in DGMs is advantageous, provided the true intrinsic dimensions generate a synthetic dataset using a pretrained BigGAN
different enough, we (Brock et al. 2019).
For positive + Loaaiza-Gaaem et al. (2022) show that the likelihood of D-dimensional model can go infinity rate of a while not recovering F if the data supported on d-dimensional manifold.
8

---

Published as a conference paper at ICLR 2023
We use this model, which has Z R 120, generate 64,000 samples from the golden retriever class. These samples have true intrinsic dimension at most 120, and their estimated intrinsic dimension through (2) is 22 (we rounded the real-valued estimate up). We also generate another $4,000 sam- ples from this model, except we zero out all but m latent coordinates before pass- ing them through the generator, resulting in samples of true intrinsic dimension of at most m, and estimated intrinsic dimension
D-WAE+NF
D-WAE -NF Iconstant di
1.4
1.3
1.0
Number free latent dimensions (m)
3. We then combine these two types of sam- ples into single dataset of size 128, 2,,,00, with each type of sample corresponding to class. This dataset has the property that its classes have a large difference in intrinsic dimension, and estimating the intrinsic dimension of the entire dataset while ignoring classes gives value of 5. We then train two models on this dataset: disconnected two-step model, where the first models are WAEs with fixed intrinsic dimension set to 5 (i.e. the single estimate of intrinsic dimension over the entire dataset) and the second models NFs (in- dicated as "D- WAE+NF (constant d)"); and a disconnected version of the same model except clusters have their latent dimensions set to 22 and (i.e. class intrinsic dimension estimates), respectively (indicated as "D- WAE+NF"). We then repeat this entire process for m = 3 and m 5, and show the resulting FID scores in Figure 5 for runs. We can see that setting the latent dimension of each cluster to its estimated intrinsic dimension results in a large performance improvement versus setting it without accounting for varying intrinsic dimensions. We can also see that this gap in performance tightens as the difference in true intrinsic dimensions decreases. These results show that considering varying intrinsic dimensions is practically relevant in DGMs, particularly when the difference in true intrinsic dimensions is large.
Figure 5; data, the shaded areas show standard errors.
Mean FID scores for runs on golden retriever
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
In this work, empirically verified the union of manifolds hypothesis by showing that image datasets have multiple connected components whose intrinsic dimensions vary widely. We also establish uses of the union of manifolds hypothesis, showing that classes of higher intrinsic dimension are harder to classify and how this insight can be used to improve classification accuracy. We anticipate that the broader deep learning community will further unlock the potential of the union of manifolds hypothesis for both understanding natural data and discovering empirical improvements.
Although we confirmed the union of manifolds hypothesis on commonly-used datasets, have not tested on anything besides image data; verification of the union of manifolds hypothesis and the efficacy of disconnected DGMs on other types of data remain be proven. We have reason believe our improvements will generalize though, as has often been the case with other DGM innovations developed on image-like like data.
We also highlight that, while our experiments show that classes are good approximation for connected components, they do not imply an exact match: our results are consistent with some classes overlapping, e.g. 1s and 7s on MNIST might belong to the same connected component. Additionally, while we improve upon the manifold hypothesis, the realistic scenario of intrinsic dimension varying within a connected component is not covered by the union of manifolds hypothesis, and we believe that further probing data for this structure to be an interesting direction for future work.
Another point to note is that, while we did obtain classification accuracy improvements through the union of manifolds hypothesis, these were marginal. Finally, even though our main goal with disconnected DGMs was to verify the union of manifolds hypothesis, they can likely be improved upon as DGMs in several ways, for example through parameter sharing, or by learning the number of clusters (Salvador & Chan 2004 Kulis & Jordan 2012 Villecroze et al. 2022).

---

Published as a conference paper at ICLR 2023
REFERENCES
Maryam Abdolali and Nicolas Gillis. Beyond linear subspace clustering: A comparative study of nonlinear manifold clustering algorithms. Computer Science Review, 42: 100435, nov 2021. doi: cosrev. 2021
https:sss///o..
101 co..sse.....rre
100436
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. ICLR, 2021.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference Machine Learning, pp. 224-232. PMLR, 2017.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pp. 1027-1035, 2007.
Ershad Banijamali, Ali Ghodsi, and Pascal Popuart. Generative mixture of networks. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 3753-3760, IEEE, 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. International Conference on Machine Learning, pp. 572-582. PMLR, 2019.
Matan Yosef and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. arXiv preprint arXiv: arXiv:1808 1808. 10356, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013.
Adam Block, Zeyu Jia, Yury Polyanskiy, and Alexander Rakhlin. Intrinsic dimension estimation. arXiv preprint arXiv:2106.04011 2021.
Vladimir Igorevich Bogachev. Measure theory, volume 2. Springer, 2007.
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33:442-453. 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICLR, 2019.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
Theophilos Cacoullos. Estimation of multivariate density. Annals of the Institute of Staristical Mathematics, 18(1):179-189 1966.
Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1-10, 2022.
Lele Cao, Sahar Asadi, Wenfei Zhu, Christian Schmidli, and Michael Sjöberg. Simple, scalable, and stable variational deep clustering. In Join! European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 108-124. Springer, 2020.
Anthony L Caterini, Gabriel Loaiza-Ganem Ganem, Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, volume 34, 2021.
10

---

Published as a conference paper at ICLR 2023
Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and Joem-Henrie Jacobsen, Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, volume 32, 2019.
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International conference machine learning, pp. 2133-2143. PMLR, 2020.
Edmond Cunningham, Adam Cobb, and Susmit Jha. Principal manifold flows. arXiv preprint arXiv:2202.07033, 2022.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. ICLR, 2019.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornoon, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint arXiv:2202.0276. 2022.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. ICLR, 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. JCLR, 2017.
Laurent Dinh, Jascha Sohl-Dickstein, Hugo Larochelle, and Razvan Pascanu. RAD approach to deep mixture models. arXiv preprint
2019.
Leo L Duan. Transport Monte Carlo: High-accuracy posterior approximation via random transport. Journal of the American Statistical Association, (just-accepted): 2021.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural Spline Flows. In Advances in Neural Information Processing Systems, volume 32, 2019.
Ehsan Elhamifar and René Vidal. Sparse manifold clustering and embedding. In Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL htttpss:/procceedinn ietpe/eeeddinn.nunieeeeeee neurips. cc/ ccc/aaper/211//11e b
Ehsan Elhamifar and Rene Vidal. Sparse subspace clustering: Algorithm, theory, and applications, 2012. URLLsttps://arxiv.oooo
Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications.
IEEE transactions on pattern analysis and machine intelligence, 35(11):2765-2781 2013.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows riemannian
manifolds. arXiv preprint arXiv: /61 1,02304, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. IEFF, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 pmlr, 2015.
11

---

Published as a conference paper at ICLR 2023
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In International Conference on Machine Learning, pp. 4615- 4630. PMLR, 2020.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joini Conference on Artificial Intelligence, pp. 1965-1972, 2017.
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. Advances in Neural Information Processing Systems, 31, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible I xx 1 Convolutions. In Advances in Neural Information Processing Systems, volume 31, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. ICLR, 2014.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Proceedings the Thirty-Sventth Conference on Uncertainty in Artificial Intelligence, volume 161, pp. 1269-1278, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brian Kulis and Michael Jordan. Revisiting k-means: new algorithms via bayesian nonparametrics. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1 1131-1138 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2224, 1998.
John M Lec. Introduction to smooth manifolds. pp. 1-31. Springer, 2013.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Uzu Lim, Vidit Nanda, and Harald Oberhauser. Tangent space and dimension estimation with the wasserstein distance. arXiv preprint arXiv:2110.0633) 2021.
Gabriel Ganiz, Brendan Leigh Ross, Jesse C Cresswell, and Anthony L. Caterini. Diagnosing and fixing manifold overfitting deep generative models. Transactions on Machine Learning Research, 2022. URL httpl:hts::///c http:///eeeeeeeeeeeeeeeee / net
Francesco Locatello, Damien Vincent, Ilya Tolstikhin, Gunnar Rätsch, Sylvain Gelly, and Bernhard Schölkopf. Competitive training of mixtures of independent deep generative models. arXiv preprint arXiv: 11130, 2018
Lorenzo Luzi, Randall Balestriero, and Richard G Baraniuk. Ensembles of generative adversarial networks for disconnected data. arXiv preprint arXiv:2006. /4600, 2020.
David JC MacKay and Zoubin Ghahramani. Comments on' maximum likelihood estimation of intrinsic dimension' by levina and p. bickel (2004). The Inference Group Website, Cavendish Laboratory, Cambridge University, 2005.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. In Advances in Neural Information Processing Systems, volume 33, 2020.
Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent gaussian mixtures. In NIPS Workshop on Bayesian Deep Learning, volume 2, pp. 131, 2016.
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems, 23, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 201 1.
12

---

Published as a conference paper at ICLR 2023
Arkadas Ozakin and Alexander Gray. Submanifold density estimation. Advances in Neural Informa- tion Processing Systems, 22, 2009.
Guilherme GP Freitas Pires and Mário AT Figueiredo. Variational mixture of normalizing flows. In ESANN, 2020.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimen- sion of images and its impact on learning. In International Conference on Learning Representations, 2021.
Andres Potapezynski, Gabriel Loaiza-Ganem, and John P Cunningham. Invertible gaussian reparam- eterization: Revisiting the gumbel-softmax, Advances in Neural Information Processing Systems, 33:12311-12321, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pp. 1278-1286. PMLR, 2014.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kan- war, Phiala Shanahan, and Kyle Cranmer. Normalizing flows tori and spheres. In International Conference on Machine Learning, pp. 8083-8092. PMLR, 2020.
Lior Rokach and Oded Maimon. Clustering methods. In Data mining and knowledge discovery handbook, pp. 321-352. Springer, 2005.
Brendan Leigh Ross and Jesse Cresswell. Tractable density estimation on learned manifolds with conformal embedding flows. In Advances in Neural Information Processing Systems, volume 34, 2021.
Brendan Leigh Ross, Gabriel Loaiza-Ganema Anthony Caterini, and Jesse C Cresswell. Neural im- plicit manifold learning for topology-awen generative modelling. arXiv preprint arXiv:2206. 11267, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Antoine Salmona. Valentin de Bortoli, Julie Delon, and Agnès Desolneux. Can push-forward generative models fit multimodal distributions? arXiv preprint arXiv:2206. /4476, 2022.
Stan Salvador and Philip Chan. Determining the number of clusters/segments in hierarchical clus- tering/segmentatiion algorithms. In 16th IEEE international conference tools with artificial intelligence, pp. 576-584. IEEE, 2004.
René L Schilling and Franziska Kühn. Counterexamples in Measure and Integration. Cambridge University Press, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Marek Smieja, Maciej Wolezyk, Jacek Tabor, and Bernhard C Geiger. Segma: Semi-supervised gaussian mixture autoencoder. IEEE transactions on neural networks and learning systems, 32(9): 3930-3941, 2020.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: no gan's land. In International Conference on Machine Learning, pp. 9418-9421
PMLR, 2020.
Piotr Tempezyk, Rafat Michaluk, Lukasz Garcarek,, Przemyslaw Spurek, Jacek Tabor, and Adam Golinski. Lidl: Local intrinsic dimension estimation using approximate likelihood. In International Conference Machine Learning, pp. 21205-21231. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2023
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders ICLR, 2018.
René Vidal. Subspace clustering. IEEE Signal Processing Magazine, 28(2):52-68, 2011.
Valentin Villecroze, Harry J Braviner, Panteha Naderian, Chris Maddison. and Gabriel Loaiza- Ganem. Bayesian nonparametrics for offline skill discovery. arXiv preprint 4rXiv.220.006675, 2022.
Joe H Ward Jr. Hierarchical grouping to optimize objective function. Journal of the American statistical association, 58(301):23-224, 1963.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for bench-
marking machine learning algorithms. arXiv preprint arXiv: 2017.
Fei Ye and Adrian G Bors. Lifelong mixture of variational autoencoders. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Tong Zhang, Pan Ji, Mehrtash Harandi, Wenbing Huang, and Hongdong Li. Neural collaborative subspace clustering. International Conference on Machine Learning, pp. 7384-7393, PMLR, 2019.
14

---

Published as a conference paper at ICLR 2023
A DEFINITION OF SUPPORT AND PROOF OF PROPOSITION 1
Before giving the standard definition of support, we motivate it. Intuitively, the support of dis- tribution I' is "the smallest set" to which IP assigns probability 1. The idea of "the smallest set" satisfying property is often formalized by taking the intersection of all sets satisfying the property. Naive application of this principle however results in a "definition" " of support with undesirable properties. Consider for example the case where IP is standard Gaussian distribution any definition of support that we use should obey that the support of TP is IR. However, if we simply follow the "definition" of support as the intersection of all sets of probability 1, we would not obtain this: for any I E R. clearly P(R) \ {x}) = 1. yet MEER R. \ {æ} = 0. where 0 denotes the empty set. In other words, arbitrary intersections of sets of probability 1 need not have probability 1, and in fact can have probability O. More abstractly, sets of probability 1 are not closed under arbitrary intersections, which is desirable when defining "the smallest set" satisfying a given property as the intersection of all such sets. In order to solve this problem, the support is defined as the intersection of all closed sets having probability 1 (closed sets are indeed closed under intersections).
R. Clearly
Definition 1 (Support of probability distributions): Let 2 be a topological space. Equipping 2 with its Borel r-algebra to make it measurable space, let IP be a probability distribution on Z. Let C(IP) be the collection of all closed sets C' in Z such that P(C) = 1. The support of IP, denoted as supp(P), is defined as:
(P) = n C. CEC(P)
Before continuing, point out that there exist counterexamples showing that the support of distribution need not have probability/ 1 (Schilling & Kühn 202T, Examples 6.2 and 6.3). In other words, even though closed sets are closed under intersections, closed sets of probability 1 need not share this property. These examples are however much more contrived than the one above and of no practical interest (if 2 is a separable metric space, as is the case of IRd, then any Borel measure P on E satisfies P(supp(P)) = 1, see [Boaachey| (2007 Proposition 7.2.9)), and thus support is commonly defined as above. Note that the assumption that 12 (supp(F'z) could be replaced by the requirement that Z be a separable metric space and I' Borel measure on it (which covers all settings of practical interest), although this would yield a slightly less general result. We restate Proposition 1 below for convenience:
in Proposition I
Let Z and X' be topological spaces and G Z - X' be continuous. Considering Z and X' as measurable spaces with their respective Borel o-algebras, let Pzz be probability measure on 2 such that supp((Fz) is connected and Fz (supp((P))) )) = Then supp(G
Proposition
is connected.
Proof: As mentioned in the proof sketch in the main manuscript, showing that supp G+ = el (G(supp(Pz)) where cl(.) denotes closure in X, is enough, since by continuity of G, G G(supp(IPz) is connected, and the closure of connected sets is itself connected.
We begin by verifying that el (((pp()(())) 2)) C supp(G y PP)). Let C E ((G'P)) By definition of C(G Pz), C' is closed in X' and is such that (CP)(C) = 1. By definition of pushforward measure, it follows that:
(5) and by continuity of G, it also follows that G (C" is closed in Z. Then, GC) e C(P2). and by definition of support, supp(Pz) z) C (C). It then follows that G(supp(Pz)) C GIG C C. Since this holds for every C E C(G # Pz), we have that:
Pz = G&PP((C) 1,
((supp(Pz)) C
N
C= (=supp(GyPP)
(6)
P2)
Applying cl(-) on both sides, and using the fact that supp G P2) is closed in X' (since it is an intersection of closed sets), yields ((upp(((P))) C suppl G PPz).
It now only remains to show that supp( G PPZ) C clGG(supp (Pz))(ppp22)) P2)). By definition of pushforward measure, we have that:
G+ IP (G(sup 2))) PG P2
2 (z(spp((P2)) 1,
15

---

Published as a conference paper at ICLR 2023
where the inequality follows from supp(Pz) C G (- (((((spp(FZ)) then GzP2 ( (F)))) = 1. It follows that ((G(ssu))(P)))
Since ((supp(Fz)) S
c(G ((GPPz) Pz). and thus s _ c((((supp(Pp))) z))) by definition of support.
B DISCONNECTED DGMs: DETAILS
B.1 TRAINING DISCONNECTED DGMs
Algorithm 1: Training of disconnected DGMs
Input: cluster wwwsteringgllelr ww..iigglorrrriim....
Output: {(12) Z Ge)le= DI..... Dz www.strring.lloorittmmf
2 for == 1 to L do
Potentially initialize p and Gt
Train G₃ and potentially PlZ on Dr
5 end
Training We summarize the training procedure for disconnected DGMs that we mentioned in the main manuscript in Algorithm| I Note that use the exact same computational budget to train a disconnected DGM {(Pl) Ge) as we do to train an equivalent non-disconnected baseline (Ipz, G)- We begin with the observation that due to the serial nature of Algorithm [ all I. models need not be in memory when training disconnected DGM: once the eth model has been trained, it can be moved disk before loading the (6 + 1)th model to memory. In other words, even though disconnected DGMs do have more parameters than their non-disconnectee require more disk space be stored, it not needed at any point during training to load more than single model in memory. We also make the key observation that, if using the same architecture throughout (i.e. G and every Ge have the same architecture, as do Pz and P-Z if they are trainable), then training the non-disconnected DGM (P2z, G)) for N epochs on D requires the same amount of compute as training each (P Ge) for N epochs, since every (IP dataset D,. To this in more detail, assume training (P'z z, G) for single epoch requires T gradient steps, so the total training cost is TN gradient steps. In order to train a single (P()
counterparts and thus
2 G.)) is trained on a smaller
(t) G&) model for a single epoch, since Dj is (D) Del times smaller than D, only /Del/|D|T gradient steps are required. Thus, training single (P z G model for N epochs requires |Del/[D TN gradient steps, and
follows that training all the (IP Z Gt) for l I,..., L requires
Σ
4
D D TN
TN L D
TN D
=
De
D = TN
(8)
E=1
gradient steps. Finally, since exactly the same number of steps are required to train disconnected and dindonsconeettd models, and these models share architectures, it follows that the FLOP and memory costs of training them are indeed equivalent under our setup.
Clustering algorithm In order to generate the dataset clusters to train each component in our disconnected DGMs (with the goal of approximating the connected components of the full dataset) without labels, we perform agglomerative clustering (Rokach & Maimon 2005) on the dataset. At the start of the algorithm, each datapoint with the smallest linkage distance are merged, decreasing the number of clusters by 1. This occurs until a pre-specified number of clusters, L., is reached. For all of our datasets, we set the number of classes for the MNIST, FMNIST, SVHN, and CIFAR-10 10 datasets. We experimented with ranges L between 7 and 15 and did not find a large variation in performance. Automatically inferring L from dataset is the subject of future work and would make our clustering models completely unsupervised. For all of our experiments involving agglomerative clustering, we used Ward's linkage criterion (Ward Jr. 1963). The distance between two clusters with this criterion is the variance of the Euclidean distance between all datapoints in the clusters being merged. Therefore, at each step, the two clusters with the smallest variance will be combined. We experimented with using
its own cluster. Then, at every step. the two clusters
which is
16

---

Published as a conference paper at ICLR 2023
single linkage, which defines the distance between two clusters the minimum distance between datapoint both clusters, but found that this led to very unbalanced cluster sizes and had very poor performance when training our DGMs. We also tried custom linkage metric that merges the two clusters at each step to maximize the inter-cluster intrinsic dimension estimate variance, with the intuition that building clusters with large variance in intrinsic dimension would better model the submanifolds of the dataset. In practice, we found this method very sensitive to initialization and saw the same unbalanced cluster failure mode as the single linkage criterion.
For all of our experiments, we also tried using the k x-means++ (Arthur & Vassilvitskii 2007) clustering algorithm. This also worked well but gave slightly worse results than agglomerative clustering with Ward's 's criterion across the board.
B.2 SAMPLING FROM DISCONNECTED DGMS
Algorithm 2: Sampling disconnected DGMs
Input: m, trained disconnected DGM ((T2) Ge) and corresponding cluster sizes
(Dil..... [Dz]-
Output:
S <-0
(Dil
1D2
(m1 ,,,,,.,,, mL)
Multinomial
m,
-t'=1 [Del
ΣE= Dre
for l = to L do
for to my do
Z ~ p(f)
(Z)
S - S U (X}
end end
As mentioned in the main manuscript, sampling from disconnected DGM is achieved by first sampling with probability proportional to [De]. and then sampling from the corresponding DGM:
Drel
l p(e) Ee
2IE~Pl P/2 and x= GZ)
Sampling from ( can be made memory-efficient, and we now describe how to do so. Assume we want generate m samples from trained disconnected DGM model {(Pl)) (=) The idea is to make sure that each of the L DGMs are only loaded into memory once and one at time, which requires first knowing exactly how many samples will required from each model. Note that, if denote the number of samples coming from the eth model as my then clearly (m.,..., m,) follows multinomial distribution with parameters m and ((Pi)...., 1D,l)) Et [Drrl. We can thus first sample from this multinomial, and then sample the appropriate number of times from each model, as described Algorithm [ Note that, similarly Algorithm I the outer for loop Algorithm [ can be trivially made memory-efficien by loading the required model (P/) Ge) into memory and then removing from memory once it has been sampled from my times (and the inner for loop can be trivially parallelized).
C ADDITIONAL RESULTS
C.1 DISCONNECTED DGMS
Table 3 shows analogous results to those of Table for VAEs. We can see that D-VAEs (classes) consistently outperform VAEs and D-VAEs (random), except on CIFAR-100, further confirming the disconmectedness of the support of the data. Also, conditional VAEs (classes) convincingly outperform D-VAEs (classes) only on FMNIST and SVHN, and conditional VAEs (clusters) convincingly outperform D- VAEs (clusters) only on these same datasets, showing once again that D-DGMs provide sensible way account for disconnected supports.
17

---

Published as a conference paper at ICLR 2023
Table 4 shows results of running a VAE+NF two-step model mentioned in subsubbscction 4.2.2| We can see that the best performing disconnected versions still outperform the non con-disconnected version. This is consistent with our previous results and highlights once again the relevance of accounting for multiple connected components, As mentioned in the main manuscript though, there is no significant improvement hetween fixing the latent dimension of every cluster to its estimated intrinsic dimension, and simply fixing the intrinsic dimension of every cluster to the same value (the estimated intrinsic dimension of the entire dataset).
Table 3: FID scores. We show means and standard errors across runs.
Model
MNIST
FMNIST
SVHN
CIFAR-10
CIFAR-100
VAE
110.7 +: 1.7
100.1 +: 0.6
38.2..0..
213.8 1.3
198.4 0.6
D-VAE (random)
155.4 1.3
125.6 0.6
1.7.6
232.3 0.8
220.9 0.8
D-VAB (classes)
81.5
1.0
86.4 2.0
202.4 0.6
203.3 =
VAE (GMM)
1.6
0.3
95.0
2.3.7
200.3 + 1.6
Conditional VAE (classes)
82.0 0.3
84.7+0.9
71.9
1.4
211.6 1.5
203.0 + 1.7
Conditional VAF (clusters)
92.3 + 1.5
100.2 1.7 105.2+1.0
218.8 + 0.3
260.9 + 55.7
D-VAB (clusters)
84.9 = 1.0
106.9 0.1
173.7 4 15.4
220.8 + 0.9
217.8 +
Table FID scores. We show means and standard errors across runs.
Model
MNIST
FMNIST
SVHN
CIFAR-I0
CIFAR-100
VAE+NF
119.7 2.6
140.8 1 3.2
0.4
216.4 1.3
204.0 0.7
D-VAE+NF (constant d, classes)
76.9 - 1.1
1.2.5 86.6
87.7 0.8
1988
189.5
DA (constant d,clusters)
75.1
12.1
243.6
207.2 -
D DA (classes)
74.5 0.3
8.2
9.2
2.1.4
188. + 2.2
D VAE+NF (clusters)
73.0 0.9
9.8
18.66188.6
229.0
207.8 + 1.9
C.2
INTRINSIC DIMENSION ESTIMATES
Variance Estimated Intrinsic Dimension
Inter-class
Intra-class
2
MNIST
FMNIST
SVHN
CIFAR-10
CIFAR-100
ImageNet
Dataset
Figure 6: Variability of estimated class intrinsic dimension (with k 10) across classes in blue (i.e. 11 dots, one per subsample), and across subsamples in green (i.e. one dot per class).
18

---

Published as a conference paper at ICLR 2023
Variability of intrinsic dimension estimates In order to ensure that the variability of estimated in- trinsic dimensions across classes observed in Figure caused by the true intrinsic dimensions being different rather than the variance of the estimator used, carry out the following experiment: For each dataset and each of their classes, we randomly subsample (without replacement) 1000 datapoints (except for CIFAR-100 where estimate the corresponding class intrinsic dimension using only this reduced dataset. We repeat this process an additional 10 times, and compute the variance of estimated intrinsic dimension across classes (i.e. claer- and across subsamples (i.e. intra-class). Observing much lower variability across subsamples- as we
300 instead, there only 500 datapoints per class), and
Figure for all datasets except shows that the variability observed across classes cannot be explained by the variance of the estimator, once again supporting the union of manifolds hypothesis. Note that we sample without replacement to avoid duplicates, as these would result in distance of 0 to their nearest neighbour.
Intrinsic dimension of specific classes We show a more granular breakdown Figure 3 with intrinsic dimension estimate values for each class in Figure 7 for MNIST, Figure for FMNIST, Figure D for SVHN, and Figure for for CIFAR-10. 10. Since CIFAR- 100 and ImageNet have too many classes to show, we only include the top highest and lowest intrinsic dimension classes in Figure] TT for CIFAR- 100, and Figure 12 for ImageNet.
20.0
k=
17.5
k= 10
Intrinsic dimension estimate
1=
15.0
==
k= 25
12.5
10.0
5.0
0.0
1
7
6
9
4
0
2
5
3
8
Class
Figure 7: Intrinsic dimension estimates for classes in the MNIST dataset.
19

---

Published as a conference paper at ICLR 2023
3=3
20.0
Intrinsic dimension estimate
k= 15
17.5
k= 20
15.0
k= 25
12.5
10.0
Trouser Pullover
Coat
T-shirt/top
Dress
Shirt Ankle
Bag
Sandal
Sneaker
Class
Figure 8: Intrinsic dimension estimates for classes in the FMINST dataset.
k=3
k=
Intrinsic dimension estimate
k= 15
k= 20
25
15
10
7
9
4
8
0
6
1
2
3
5
Class
Figure Intrinsic dimension estimates for classes in the SVHN dataset.
20

---

Published as a conference paper at ICLR 2023
35
3=3
30
Intrinsic dimension estimate
k= 15
k= 20
25
k= 25
20
15
10
5
Airplane
Ship Automobile Cat
Deer
Bird
Horse
Dog
Frog
Truck
Class
Figure 10: Intrinsic dimension estimates for classes in the CIFAR-10 dataset.
k=3
20.0
k=
Intrinsic dimension estimate
k= 15
17.5
k= 20
k= 25
15.0
12.5
10.0
Cockroach Plate
Apple
Orange
Rocket
Snake
Forest
House
Tractor Butterfly
Class
Figure 11: Intrinsic dimension estimates for classes in the CIFAR-100 dataset.
21

---

Published as a conference paper at ICLR 2023
40
3=3
Intrinsic dimension estimate
35
k=
1=
30
20
k= 25
25
20
15
Jellyfish
Wing
Hammerhead
Tiger shark
Sealyham terrier
Monarch butterfly
Butcher shop
Grocery store
Confectionery
Toyshop
Class
Figure 12: Intrinsic dimension estimates for classes the ImageNet dataset.
22

---

Published as a conference paper at ICLR 2023
D EXPERIMENTAL DETAILS
D.1 DISCONNECTED DGMS
For all models, randomly select 10% of the training dataset to be used for validation and train on the remaining 90%. We use a separate test dataset to report all metrics for our models. A batch size of 128 is used for all datasets. Unless otherwise noted, at the beginning of training, data to between 0 and 1. For all experiments, we use the ADAM optimizer (Kingma & Bal 2015), typically with learning rate 0.001 and cosine annealing for a maximum of 100 epochs. We also use gradient norm clipping with a value of 10.
scale all the
Simulated Data To generate the ground truth data, we create a uniform mixture of two components. The first is two dimensional square of width 1 centered at (0, 0.8). The distribution along the x and y axis are independent beta distributions with (a. .) set to (2. 2) for the y axis and (2. 5) for the axis. The second component is a one dimensional sinusoid (with y the input and x: as the output) that is one unit in length centred at 0.5 with an amplitude of 0.5 and a period of The distribution along this line is beta distribution with (a, 3) set to (2. We generate 10,000 samples for training, 10,000 for validation and 5,000 for testing. We train three models on this simulated data. For the baseline model we use a VAE with MLPs for the encoder and decoder, with ReLU activations. The encoder and decoder each have a single hidden layer with 25 units. The latent dimension of the model is set to 2. The learning rate is set to 0.0001. We do not early stopping and train for 200 epochs. For the second model, we train a clustering VAE with two components trained using the same setup as the baseline model. For the third model we train disconnected VAE+ VAE where each component is two step VAE+VAE model. Both VAEs have the same architecture and are each trained the same way as the bascline model. The latent dimension of the GAE is obtained by running the MLE intrinsic dimension estimator 2 on each component.
VAES For MNIST and FMNIST, we use MLPs for the encoder and decoder, with ReLU activations. The encoder and decoder have two hidden layers with 512 units. For SVHN, CIFAR- 100 and CIFAR-10, we use convolutional networks. The encoder and decoder have 4 convolutional layers with (32, 32, 16, 16) and (16, 16, 32, 32) channels, respectively, followed by a flattening operation and a fully-connected layer. The convolutional networks also use ReLU activations, and have kernel size 3 and stride 1. The latent dimension of the model is set to 20. We use early stopping validation loss with a patience of 30 epochs, up to a maximum of 100 epochs. For the GMM, we set the prior to have 10 modes with learnable mixture weights, means, and standard deviations. The mixture components of the prior were initialized with fixed means spaced 3 units apart centred at O, standard deviations of 1, and uniform mixture weights. For all models, the decoder's 's variance is not learned.
For all datasets, we use an MLP with 2 hidden layers of size 256 each for the discriminator, The encoder and decoder for all datasets are the same convolutional networks used for VAEs except we increase the channels to (64, 64, 32, 32) and (32, 32, 64, 64) respectively. The latent dimension of the model is also set to 20. We train every model for total of 300 epochs and perform early stopping on the reconstruction error with a patience of 30 epochs only for the GMM baselines. We note that when training for the full 300 epochs (without early stopping) the GMM baselines perform significantly worse and the other models perform better. We use a learning rate of 0.0005 for the discriminator and do not use any learning rate schedule for the model. We weight the discriminator loss with lambda value of 10. For the GMM, we set the prior to have 10 modes with fixed means spaced 3 units apart centred at O, fixed standard deviations of 1, and uniform mixture weights. We attempted to learn these parameters similar to the VAE but observed worse results.
WAEs
For the VAE GAE, we followed the same setup as the single VAE. For the NE, we use a rational quadratic spline flow (Durkan et al., 2019) with 64 hidden units, 4 layers, and 3 blocks per layer. For all datasets, we standardize the data (ie. the latent vectors of the VAE) by subtracting the mean and dividing by the standard deviation. We do not scale the data or apply logit transform. The latent dimension is set to the estimated intrinsic dimension of the cluster for fitted models, and set to 20 otherwise.
VAE+NF
23

---

Published as a conference paper at ICLR 2023
WAE+NF For the WAE, we followed the same setup as the single WAE except we use residual networks (He et al., 2016) to handle the larger and more complex golden retriever datasets. The encoder first applies convolution (increasing the channel dimension to 16), batch normalization, and ReI.U activation to the input before a max pool operation with stride 2 and kernel size of 3. The main body of the encoder is set of residual layers with channel dimensions (1,,32, 64, 128). Each layer consists of two basic blocks which are two consecutive convolution, BatchNorm (Ioffe & Szegedy 2015) and ReLU layers where the input of the block added the model's I activations before the final ReLU. The stride of the first convolution of each layer is set to 2 to halve the spatial resolution. The skip connection is downsampled by a convolution with a stride of 2. The activations are then average-pooled across each channel and projected to the latent space dimension by a linear layer. The decoder is identical to the encoder main body except there are no downsampling operations, the spatial resolution of the activation is doubled at each layer by bilinear upsampling and the channel dimensions are (128, 64, 16). For the GAE, we do not perform early stopping and train for 100 epochs. The latent dimension is set to 20. The NF density estimator is also identical to the VAE+NF setup except we perform early stopping on validation loss with a patience of 30 epochs, for maximum of 100 epochs.
D.2 EXPLOITING THE UNION OF MANIFOLDS HYPOTHESIS: SUPERVISED LEARNING
In order to investigate if there is a correlation between intrinsic dimension and the difficulty of classification for a given class, we train three different classifiers on CIFAR- 100 and measure the accuracy for each class. We use the same hyperparameters and training strategy for each of the three models we consider (VGG-19 (Simonyan & Zisserman, 2015), ResNet-18, and ResNet-34 (He et al. 2016)). We do not modify the model architecture except for changing the dimension of the last linear layer to project into a 100-dimensional vector to match the number of classes in CIFAR- 100. We use the CIFAR- 100 dataset test split (10,000 images) then randomly select 10% of the training dataset as our validation set (5,000 images) and use the remaining 90% as our train split. Note that this validation split is constant across all runs. The classification accuracy is reported as the accuracy on the test set on the same epoch where the best validation accuracy is achieved. We do not add any data augmentations as doing so could modify intrinsic dimension estimates and confound our results. Studying the relationship between intrinsic dimension and data augmentations an interesting direction for future work. The combination of (1) no data augmentations, and (2) the presence of a validation set causes our results to be lower than the state-of- of off-the-rt for the models, although the results remain self-consistent here. We train each model using the cross entropy loss for a total of 200 epochs starting with a learning rate of 0.1 and decrease this to 10-2,4 x 10-3, and & x 10 4 after 60, 120, and 160 epochs respectively. We use the standard SGD optimizer with momentum set to 0.9 and train with weight decay of 5 x 10 For all datasets, we normalize by the channel-wise mean and standard deviation of all training images. We use batch size of 128 for training, and batch size of 100 for both validation and test.
We now detail the exact loss we used for our experiments with the re- re-weighted cross entropy loss. A weighted version of the categorical cross entropy loss is given by:
We Vie log (x
(10)
where yi is a one-hot vector of length I corresponding to the label of x,, (a;) is the L-dimensional output of the classifier containing assigned class probabilities, and we is the scalar weight given to the hth class. The standard categorical cross entropy uses the weights Wt = 1 for & = I,..., L. Our modified weights given by:
Wi L.
&
(11)
Σ
K &
Note that when the intrinsic dimension estimates of all classes match, our proposed weights recover the standard ones, but more heavily weight classes of higher intrinsic dimension otherwise.
24


Reviews:
[{"review_id": "cXVh0sxZ7XU", "reviewer": "Reviewer_FLmA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims at the union of manifolds assumption in deep learning, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. The authors empirically verify this hypothesis on commonly-used image datasets.", "summary_of_the_review": "My major concern is that the novelty and contribution of the paper are not significant or at least not well demonstrated. It is not clear what challenges or open problems the paper solved. Comparing with existing work on union of subspaces and manifolds, the quality of the paper is below the bar of ICLR.", "strength_and_weaknesses": "Strength:\n1. The union of manifolds assumption is useful in deep learning.\n2. The authors provide a lot of numerical results.\n\nWeaknesses:\n1. The paper neglects a lot of important related works. First, there have been numerous works on union of subspaces and manifolds but the author failed to acknowledge them. Union of subspaces or manifolds have been studied in many works of clustering and matrix completion. See the following: [1] Vidal. Subspace clustering. 2011. [2] Elhamifar. Sparse Manifold Clustering and Embedding. 2011. [3] Fan et al. Polynomial matrix completion for missing data imputation and transductive learning. 2020.\nOn the other hand, deep learning based clustering methods are also closely related to union of manifolds. See the following: [4] Zhang et al. Neural Collaborative Subspace Clustering. 2019. [5] Cai et al. Efficient Deep Embedded Subspace Clustering. 2022. There are even some works proposing to train multiple autoencoders to cluster data.\n\n2. The evidence of union of manifolds has also been observed when using t-SNE or UMAP to visualize high-dimensional data.\n\n3. In Section 4.2, Algorithm 1 is not clearly explained. For instance, how to do the initialization and form $\\mathbb{P}^{(\\ell)}$?\n\n4. The title of the paper is too broad.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Zdev9D3F9Q", "reviewer": "Reviewer_GFby", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors verify the union of manifolds hypothesis (proposed in earlier literature) i.e, data lies on a disjoint union of manifolds of varying intrinsic dimensions. The authors provide empirical results to support the hypothesis. ", "summary_of_the_review": "The current work is pretty limited in novelty and the authors make many statements/assumptions which do not hold true in general. The work is focussed on image datasets and thus the results and conclusions might not generally applicable. The draft in its current form needs more work.", "strength_and_weaknesses": "Here are some strengths associated with the current work :\n\n1) The authors work towards verifying the union of manifolds hypothesis.\n\nHere are some weaknesses associated with the current work :\n\n1) The authors in this work make many statements without quantifying them thus treating them as obvious facts. \n\" ... Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. ...\" \n\nThe reviewer does not agree with this hypothesis from the author. Data can be lying on a single manifold with multiple basis vectors. Different classes can choose to use appropriate basis vectors and not use the rest or use them marginally. This does not imply that different sub-regions of space cannot have different factors of variation. \n\n\"... We first prove that these deep generative models (DGMs) are incapable of modeling disconnected supports. We then argue that the class labels provided in our considered datasets identify connected components (i.e. different classes are disconnected from each other), and show that training a push forward model on each class outperforms training a single such model on the entire dataset, even when using the same computational budget: this improvement is a firm indicator that the support is truly disconnected,\nthus strongly supporting our thesis.\"\n\nThe reviewer does not agree with the author's statement above. It is much more easier to train a model once the data has been adequately clustered as such. Training a unique model for each class as compared to a single model for different classes, the hypothesis classes associated are completely different. Thus the argument is fallacious. \n\n2) In case the classes are separable and are non-intersecting and it is possible to separate the classes adequately (as the authors discuss in Section 3.1) then it is straightforward to train models on each separate class. However this is an unrealistic expectation in the real world since training separate models can be futile unless we have a mechanism to perform perfect classification of any unseen instance which is never true. Training models for all classes allows us to use the common structure and basis to mutually help learning for different classes.\n\n3) The authors work is specific to image datasets and not applicable in general, which the authors are kind enough to mention in their discussion. Similarly the authors claim to use agglomerative clustering in this current work. However as mentioned in the no free lunch theorem, there is no one clustering algorithm which can perfectly cluster all instances for all types of data. The authors could have made the work more useful and valuable by adding a discussion on this.\n\n4) The authors in this work assume that the number of clusters is known which is never true in the real world setting. Also the authors' empirical results demonstrate very marginal gains. The authors do not discuss many intricate cases i.e., the case of intersecting manifolds and how to cluster them adequately. \n\n5) The novelty/contribution of the current work is pretty limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XmqL_lb11PE", "reviewer": "Reviewer_6dov", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the \"union of manifold\" hypothesis, a refinement of the manifold hypothesis (MH) wherein data is assumed to lie on a disjoint union of manifolds of varying intrinsic dimensionality. To test this hypothesis an empirical study on common image datasets  is carried out. An empirical study of clustered generative architectures, chosen to test the disjoint manifold scenario, is also performed.\n\n", "summary_of_the_review": "The manifold hypothesis is a fundamental idea in machine learning. Given the enormous influential of MH, it is natural to investigate extensions which have greater relevance to real data. This work makes a valuable contribution to the literature by discussing a disjoint union variant of MH. This paper is an interesting topic and well-argued, although the strength of the evidence support appears small. Nevertheless I recommend acceptance.\n", "strength_and_weaknesses": "[+] Interesting and timely topic. MH has been very influential but it is perhaps time to consider more realistic variants.\n\n[+] Empirical investigation seems well-thought out and carefully performed.\n\n[+] Relevant theory (Prop. 1) on the inability of push-forward generative models to model disconnected supports of data distributions is included.\n\n[+] Appreciate the inclusion of Section 4.2.2, where tension between this work's empirical results and prior theory is addressed.\n\n[-] Empirical results appear marginal (though significant). For example in Table 1, the proposed clustered-VAE (C-VAE) outperforms on MNIST, FMNIST, and SVHN, but not CIFAR-10 and CIFAR-100. Arguably CIFAR-10/100 are more important, as they are more relevant to real data. I would like to see a fuller discussion of this point.\n\n[-] Claims about empirical results sometimes appear overstated and not fully spelled out. For instance in Section 4.2.1 the authors write \"we can see that our C-VAE (classes) performs better on most cases...\". It would be better to explicitly spell out which cases perform better and which do not. Same for Appendix C.1 \"We can see that, modulo a few exceptions...\" Which exceptions? Are they important or not?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster