Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps
Authors: Nico Courts
Henry Kvinge
Venue: ICLR 2022 Poster (2022)
Keywords: generative models
applications of topology to deep learning
many-to-one maps
invertible neural nets

Abstract: Many-to-one maps are ubiquitous in machine learning, from the image recognition model that assigns a multitude of distinct images to the concept of “cat” to the time series forecasting model which assigns a range of distinct time-series to a single scalar regression value. While the primary use of such models is naturally to associate correct output to each input, in many problems it is also useful to be able to explore, understand, and sample from a model's fibers, which are the set of input values $x$ such that $f(x) = y,$ for fixed $y$ in the output space. In this paper we show that popular generative architectures are ill-suited to such tasks. Motivated by this, we introduce a novel generative architecture, Bundle Networks, based on the concept of a fiber bundle from (differential) topology. BundleNets exploit the idea of a local trivialization wherein a space can be locally decomposed into a product space that cleanly encodes the many-to-one nature of the map. By enforcing this decomposition in BundleNets and by utilizing state-of-the-art invertible components, investigating a network's fibers becomes natural.

Published as a conference paper at ICLR 2022
BUNDLE NETWORKS: FIBER BUNDLES, LOCAL TRIV- IALIZATIONS, AND A GENERATIVE APPROACH TO EX- PLORING TO-ONT MAPS
Nico Courts
Henry Kvinge
University of Washington
Pacific Northwest National Laboratory, University of Washington
ncourt a @math. washington. edu
wanny.kvinge@pnn....
ABSTRACT
Many-to-one maps are ubiquitous in machine learning, from the image recogni- tion model that assigns a multitude of distinct images to the concept of "cat" the time series forecasting model which assigns range of distinct time-series to a single scalar regression value. While the primary use of such models naturally associate correct output to each input, in many problems it is also useful to be able to explore, understand, and sample from a model's s fibers, which are the set input values x such that (x) y. for fixed y in the output space. In this paper we show that popular generative architectures are ill-suited to such tasks. Motivated by this introduce novel generative architecture, a Bundle Network, based on the concept of a fiber bundle from (differential) topology. BundleNets exploit the idea of a local trivialization wherein a space can be locally decomposed into a product space that cleanly encodes the many-to-ome one nature of the map. By enfore- ing this decomposition in BundleNets and by utilizing state-of-the-aa thrttr-arttt invertible components, investigating network's s fibers becomes natural.
INTRODUCTION
In the last decade, generative methods have made remarkable strides toward the goal of faithfully modeling complex data distributions. Families of model architectures such as generative adversarial networks (GANs) (Goodfellow et al.] (2014), variational autoencoders (VAEs) (Kingma & Welling, 2013), and normalizing flows (Rezende & Mohamed] 2015) have enabled range of tasks that were previously thought to be unachievable for complex (and often high-dimensional) distributions such as those representing natural images. Applications include image to image translation (Mao et al., 2019 Lee et al. 2018), music generation (Mogren) 2016), image super-resolution (Ledig et al., 2017 and density estimation (Dinh et al., 2016). In this work we introduce a new task in machine learning which revolves around the problem of modeling the structure of many-to-one maps.
The majority of classification and regression tasks in machine learning involve many-to-one maps. That is, maps that send range of distinct inputs the same output. For example, most image recognition tasks are many-to-one since they send an infinite set of natural images to a finite number of classes. Of course, there no reason that model's target space needs be discrete coossite-considrr, for instance, a sentiment analysis task where sentiment is naturally represented by a continuum. There are likely a range of distinct expressions that result the same sentiment value. This paper was inspired by a problem in materials science wherein one wishes to explore all the processing conditions that will result in a material with a specific fixed property (c.g., strength). Understanding all such processing conditions allows the materials scientist to choose among range of options that will all produce a material achieving their desired performance metrics.
While the focus of most research on classification and regression tasks is training models that give correct predictions, it can also be useful to be able to fix an output value y and explore, quantify, or sample from all those which map to y. Such an exercise allows one understand different modes of variation that a model collapses in the process of making its predictions. If function π X - Y
Work completed during an internship Pacific Northwest National Lahoratory.

---

Published as a conference paper at ICLR 2022
is the ground truth pairing between input space X and output space Y, this amounts to calculating the inverse image x (z), or fiber of π at y.
Our goal in the present work is to (1) formalize this problem and (2) describe deep learning framework which readily enables this kind of analysis. We take as our inspiration the notion of a fiber bundle from topology (Seifert [933 Whitnyy T935). Consider the projection map on X Y x Z. π. Y x Z + Y. which sends π TV. easily calculated component of X that we want to predict and Z encodes the remaining variation occurring among all that map to y. Unfortunately, in our nonlinear world data distributions can rarely be decomposed into product space on a global level like this. Instead, it is more realistic to hope that for each sufficiently small neighborhood U of Y,, we can find a data distribution preserving homeomorphism U x Z 3 (U). This informally summarizes the idea of a fiber bundle, where locally the space X is the product of a base space (some subset of Y) and a fiber Z. Following the mathematical literature, we call this task local trivialization of X. A local trivialization is useful since it allows us to understand and explore the fibers of π..
For any E Y, the inverse image π (v) {v} x Z = Z. Interpreted in terms of the machine learning task, Y is the
In our experiments we show that popular generative models including Wasserstein GANs (Wu ct al, 2018) and conditional GANs (Mirza & Osindero 2014), are unable to accurately model the fiber structure of a machine learning task. To address this, we describe a new deep learning architecture called Bundle Nenwork which is designed to produce local trivializations of an input space X and thus enable one to model fibers of a task. Bundle Networks uses a clustering algorithm to divide up the output space Y into neighborhoods {U;} and then learn a conditioned, invertible mapping that decomposes the inverse image of each of these neighborhoods π (U;) into a product U; x 2 that preserves the distributions induced by the data. We show that Bundle Networks can effectively
model fibers for a range of datasets and tasks, from synthetic datasets that represent classic examples of fiber bundles, to the Wine Quality (Cortez et al. 2009) and Airfoil Noise (Brooks et al. 1989)
datasets from Dua & Graff 2017).
In summary, our contributions in this paper include the following.
We identify and formalize the problem of learning the fibers of a machine learning task.
We describe one approach to solving this problem, using a fiber bundle framework to learn how to locally decompose the input space of the task into product of label space and fiber space that parametrizes all additional variation in the data.
We introduce new family of deep generative models called Bundle Networks and show via empirical experiments that, while other popular architectures struggle to model the fibers of machine learning task, Bundle Networks yield strong results.
RELATED WORK
Conditional and manifold-based generative models: The local trivialization task that we describe in Section can be interpreted as a special type of conditional generative task. That is, a task where data generation is conditioned on some additional variables. Common models within this framework are the various flavors of conditional variational autoencoders (VAE) (Sohn et al. (0155) and con- ditional generative adversarial networks (GAN) (Mizza & Osindero 2014). While our approach utilizes conditional components to capture differences in model behavior at different locations in input space, additional aspects of the task, such as the local decomposition of input space into fiber and label space, mean that standard generative models do not give strong results.
This paper also shares similarities to other work at the intersection of generative modeling and man- ifold theory. In Kalatzis et al.) (202T), the authors propose multi-chart flows (MCFs) for learning distributions on topologically nontrivial manifolds. Similar to the present work, the authors use coordinate charts to learn non-Euclidean distributions using otherwise Euclidean machinery. Un- like our work, MCFs are designed to learn general distributions on manifolds and do not have the specialized architectural adaptations or novel training routine designed for modeling many-to-on problems. Vector bundles have also been applied to data science problems in (Soocoola & Perea 2021 Ye & Lim 2017);
Gao et al. 20021 Knöppel & Pinkall (2016).
2

---

Published as a conference paper at ICLR 2022
Invertible neural networks: Invertible neural networks have recently become topic of interest within the deep learning community. Some the first network layers that were both invertible and learnable were suggested in (Dinh et al., 2015 2016) and are in essence affine transformations fol- lowed by orthogonal transformations, Ardizzone et al. (018)) and Sorrenson et al. (2020) are among the works that have used invertible networks to solve problems of distribution approximation with the latter introducing general incompressible-ffow networks (GINs) which are volume-press preuseeg versions of invertible neural networks. The most common instance of invertible neural networks normalizing flows, which were popularized by (Rezende & Mohamed, 2015).
Distribution disentanglement is (roughly) the task of learning latent space for distribution such that orthogonal directions capture independent and uncorrelated modes of variation the data. Many recent methods of solving this problem leverage popular generative models such as generative adversarial networks (GANs) and variational autoencoders (VAEs) (Chen al. 2018 Kim Mnih, 2018, Dupont 2018 Chen et al. 2016 Lin al. (020). Like disentan- glement, the local trivialization task described in this paper involves factoring data distribution. Unlike the problem of feature disentanglement, which generally seeks to decompose a distribution into all independent modes of variation, only seek to distinguish between those modes of varia- tion defined by the classes or targets the supervised task and all other variation. The local nature of our factorization further sets apart from work feature disentanglement.
Feature disentanglement:
3 FIBER BUNDLES AND LOCAL TRIVIALIZATIONS
(Aber [0.1])
(fber
tix (0.1)
Bundie/Toaal Space
Aralytic ontimatiion d22 + (Jberttc fiter)
Bexc Space
Localy Trictal Neighborhadd
Figure 1: Examples of fiber bundles. (Left) A Möbius band as bundle over the circle $1. typical fiber over base point (dark orange) and locally trivial neighborhood and local trivialization are included for illustration. (Right) Two other examples of fiber bundles. Above is the Torus (with circular fiber) lying over the circle. Below is the analytic continuation of the complex function which has two-point fiber.
V2 over the punctured dise fi (= E Clo |
One way understand complicated geometric spaces is decompose them into simpler compo- nents. For example, topologically torus T' can be understood to be the product two circles T 2 S x St. Such decompositions into product spaces are particularly useful in applications since many algorithms on the entire space can then be reduced to independent, component-wist calculations. However, even in situations where local neighborhoods look like product, such de- compositions may not be possible globally because of "twists" that occur in a space (for example in the Möbius band, see Figure I. First developed in the work Seifrtt (1933) and WWitney (1935), fiber bundles are a geometric structure designed to capture the locally product-like nature of such spaces.
Formally, a fiber bundle is triple (, 2) of topological spaces along with a (surjective) continuous map x E + and collection of local trivializations T {(U,, Pi)eer subject the conditions that (i) {Uihez an open cover of B and (ii) for each i
3

---

Published as a conference paper at ICLR 2022
I. (fi gives homeomorphism Pi : U, x Z =, (U;) such that the obvious pro- jection from U, x Z to U, agrees with the map a restricted from "U,) to U, This I, the following diagram commutes.
last condition is equívalent to saying that for all The space is generally called the lotal space, B is called the base space, and is called the fiber. One can think of this definition as rigor- ously capturing the idea that for small enough neighborhoods, E looks like the product of subset of B and Z. Some standard fiber bun- dles are found in figure !
(U:)
U x Z
9
proj
U,
(I)
3.1 THE LOCAL TRIVIALIZATION OF A MACHINE LEARNING TASK AND MODELING DATA ON A FIBER
Suppose that T is a supervised machine learning task that involves taking data from X and predicting correspond element of Y. Let Dx be the data distribution X and let measurable function t : supp(Dx) + Y. define a ground truth label for each x in the support of Dx. Then for any measurable open set U S Y. Dx induces a conditional distribution, Dz-1(0) on π (U). In similar fashion, via the pushforward of Dx using ग,, we get label distribution Dy on Y. Finally, each measurable subset U C Y inherits conditional distribution Dy from Dy. We call the sets (U)) with distribution (r))( the trivializable neighborhoods of task T.
A local trivialization of X with respect to π. involves:
1. A choice of open cover U (Ut) of Y such that U,U, Y
2. A choice of fiber space Z with associated fiber distribution Dz-
3. Learning an invertible, rrivializing map yi : U, x Z π - for each U3 E EU, such that
(a) (P; maps distribution Du; @ Dz D ((U,) and (b) 9i; makes Diagram commute.
Note that in the above formulation, the space X takes the role of the total space E from section and takes the role of the base space B. Though good choices of fiber Z and measure P can occasionally be guessed based the nature of the data, they generally must be arrived at through experimentation. Further, the use of local trivializations should be seen as a convenient framework that enables the input space to be locally approximated the product of two subspaces: one corre- sponding to the label and one capturing variation unrelated to the label. One should not expect that a task and its fibers are literally a fiber bundle "on the nose."
The trivializing maps {pith allow one to explore the fibers of т, which includes sampling from them. For example, if we choose some y Y then we can sample from the fiber π (v) by (1) identifying the neighborhood U; that y belongs to, (2) sampling a collection of points {21...., 2t} using Dz, and (3) applying the map yo; to (y, 2:) for each 2 E {21....,ze!
4 BUNDLE NETWORKS
We introduce new model called Bundle Network (or BundleNet for short) to address the problem described in section 3.T A BundleNet (sketched out figure combines an invertible architecture with the ability to condition on elements of R. which are in bijective correspondence with the neigh- borhoods of Y. Specifically, BundleNet is a function : X x + Y x Z, such that given a specific choice of conditioning vector E R, Φ. := ((- X +Y Z is an invertible neural network. o; and +-1 are meant to model yo and you (respectively). Moving from forward to back, the first layer of + an affine transformation conditioned (solely) on elements from R. After this layer, the model consists of alternating RNVP blocks and coordinate permutations.
As noted in section 3.1 a number of choices need to be made before a model is trained. Let D, S X X Y be the training set and px X x + X and py X x - Y be the standard projections of X x onto its components. Firstly, the collection of neighborhoods 2/ in Y needs to be chosen. In the case of regression, where py (D;) can contain arbitrary number of distinct elements, that using k-means was an efficient and effective way to cluster these points into neighborhoods. The set of cluster centroids then exactly the vectors R {ri,...,,", r.) that will be used to condition
found
4

---

Published as a conference paper at ICLR 2022
Impat: point in Dosmate lying over It (x) y in pase-
nt argmtRrex lr - yll
S
(x.2)
H
H
R
N
Conditional Affine
Transformatinn
Figure 2: A diagram of the BundleNet architecture. The upper arrows indicate the forward operation of the model while the lower arrows indicate the reverse operation of the model.
the network. The number of clusters, is hyperparameter that can be explored during training (we discuss the robustness of BundleNet to different choices of q in section A...I), In the case where the output values are finite (i.e. py (D,) can only take finite number of values C' {ci..... Cy H2y), one can simply set U; = {ci). After selecting neighborhoods U in the base space, the neighborhoods in X are defined automatically by taking inverse image with respect to The fiber Z and fiber distribution D& also need to be chosen. These choices are discussed in the first part of Section 6
We call application of the model in the direction X + Z forward operation and application of the model in the direction Y x % + X reverse operation in accordance with the ground truth association function क X Y. As depicted in Figure 2 apply & to training point (I. y) De in the "forward direction" we solve r; = arg min ||r y/I. finding the representative r; in R that closest label y. Then we apply $ (- r:) to x. To run & in reverse on training example (x, y) we invert 4; to Φ,-1, sample some z from Dz, and compute 4 z). To ease notation we write (xx) (respectively, Φ- Φ- 2)) to denote the process of first computing and then applying D to x (resp. 4.-1 to (y. 2)).
Since we are primarily interested in generating and sampling from task fibers via their local trivi- alization, at inference time we only run & in reverse, applying .,' to collection of points (y, 2) sampled from Du, Dz. We evaluate this mapping by measuring the discrepancy of the resulting image in X against the distribution D,-"(U))- One could conceivably also use & for classification, but in that case one would need to develop a strategy for identifying the proper r; E R to use to condition & in the absence of a label associated with X. One could likely use nearest neighbor approach using points that are known to belong to each π (U,), but we leave this for future work.
The loss choose for our model consists of two main parts, Ltwd and Lowa, corresponding loss from the forward and wackward directions. Since & is invertible, these can be trained simultancously and updated with a unified gradient step. Together, our final loss is then Lbundlenet Ltwd + Ltwa-
The forward loss Liwa is the simpler the pair. Given (x, v) E Dr. we find such that E U, and compute Lawa(x, y) = [|py (Ø, (x)) y/12, where py denotes the projection from Y Z to Y. This term ensures that in the forward direction accurately maps elements from X to base space Y. The backward loss is comprised of several terms: Liwd LKI. fwd + LwI.- + LMSMD (where MSMD mean squared minimum distance). Choosing some U, E U4, we sample points from Du, and pair each with point that we sample from our priors (that is, from D.) get set S. S is passed through Φ;- to generate points X. Then the MSMD, KL-fwd, and KL-bwd losses (ef. section A.2) are used to compute loss term by comparing Φ (S) and points sampled from D, *(V.). The KL-divergence terms act to ensure that the point clouds Φ; (S) and (U,) cover one another and the MSMD term ensures that no point in - is too far from some point in (U,).
5

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
We evaluate Bundle Networks on four datasets. Two of them are synthetic and represent classic examples of fiber bundles: the Möbius band and the torus, both embedded in 12 The other datasets
are Wine Quality (Cortez et al. 2009) and Airfoil Noise (Brooks et al. 1989). both of which are available in the UC Machine Learning Repository (Dua & Graff 2017). The Wine Quality task (as we have structured it) involves predicting wine quality and color (white vs. red) from 11 measured quantities. The Airfoil Noise task has 5 real-valued input parameters and one predicted output (scaled sound pressure level). Further information can be found in section A.I For the invertible layers in BundleNet we used the FrEIA package (Ardizzone et al. 2021), We chose Wine Quality and Airfoil Noise because they represent low www-dieeniiona, many- tannytto-one relationships with several continuous variables making each a good first test case for the local trivialization task.
In selecting models against which to compare our BundleNets, we chose modern yet general- purposse models that have proven effective at many generative tasks. This includes Wasserstein divergence traditional Wasserstein GANs by Wu et al.) (2018)
GANs (WGAN) developed an improvement and conditional GANS (CGANs) from Mirza & Osindero (2014). While WGAN directly learns to approximate the global distribution of data using Wasserstein distance, it doesn't include condi- tional aspect that allows a user to specify fiber to sample from so it is excluded from our fiberwise metries. We also implement a version of CGAN that we call CGAN-local that conditions on a concatenated vector consisting both of the base point and a neighborhood representative, chosen in a manner analogous to BundleNet. We developed this model to identify whether a CGAN could per- form on par or better than BundleNet when it is provided with the exact same locality information during training and testing. As we describe in section 5.2 we take CGAN-local's i failure to reach BundleNet's is level of performance evidence that the specialized architecture and loss functions BundleNet uses makes better equipped for the task of fiber modeling.
5.1 TRAINING AND EVALUATION PROCEDURE
Each model was trained on a single GPU (Nvidia Tesla p100 @ 16GB) for a period of 2000 epochs using initial learning rate of 10 which was halved every 70 epochs. The weights that were used for evaluation were the final weights that were saved after the final epoch, even if the model con- verged long before. A table of hyperparameters used for these results (including fiber distributions D2) can be found in Appendix section A.3 along with a discussion of their impact on performance.
The aim of our evaluation is to understand how well BundleNet models can reconstruct fibers in the total space X. We use a number of different metrics to do this including: MSMD, KL-divergence, Wassertein metries (Wi and W2),, and maximum mean discrepancy (MMD). We perform such mea- surements in two modes: the global and fiberwise regimes. In the global regime, the trained model asked to generate 4000 points from the global distribution Dx (using points sampled from and 2) and these are compared against 4000 points from either the true data distribution (in the syn- thetic datasets when this is known) or from a held-out test set (when we don't have access to the underlying distribution). Then these two point clouds are evaluated using each of our metrics. The process is repeated 10 times and the output values are bootstrapped to compute sample mean with 95% confidence intervals. In the fiberwise regime, 15 points are sampled from Y and for each such y, we use the model to generate 200 points lying in æ - (y)) We compare this against 200 points we have sampled from either the true fiber (when this is known) or the points lying over the base point in our combined train& & test sets. Each metric is evaluated 10 times and bootstrapped as above.
5.2 RESULTS
For the results described in this section we mostly focus on the Wasserstein-1 metric Wi as our evaluation statistic. The full suite of results (MSMD, MMD, KL-fwd, KL-bwd, Wi, and W2) can be found in Appendix section A.5 With a few exceptions, performance patterns we see for the Wasserstein-1 metric are largely repeated when we apply other metrics.
As can be seen in tables I [ with the exception of CGAN-local's performance on Airfoil Noise in the fiberwise regime, BundleNet outperforms all the other models, sometimes by very wide margin, in both the global and fiberwise data generation regimes. In fact, the only dataset where other models are competitive with BundleNet is Airfoil Noise and that is again only in the fiberwise

---

Published as a conference paper at ICLR 2022
Table 1:
The Wasserstein-1 metric for global data generation across all four datasets and models. with 95% confidence intervals.
Each metric is applied to a trained model as detailed in section 5.1
BundleNet (ours)
WGAN
CGAN
CGAN-local
Torus
0.461 = 0.020
2.191 = 0.011
0.015
1.610 = 0.081
Möbius band
0.264 1 0.004
2.048 1 0.013
1.84 +: 0.015
6.497 1 0.101
Wine Quality
1.733 1 0.011
99.22 1 0.392
4.054 +: 0.014
2.114 +: 0.011
Airfoil Noise
1.448 + 0.022
4.741 + 0.034
2.508 + 0.025
1.735 + 0.013
Table The Wasserstein-1 metric for fiberwise data generation across all four datasets and the three models that have conditional component. Intervals represent 95% CIs over 10 trials.
BundleNet (ours)
CGAN
CGAN-local
Torus
0.251 = 0.013
7.996 = 0.015
0.450 + 0.018
Möbius band
0.279 1 0.011
9.860 1 0.038
8.228 0.904
Wine Quality
1.917 1 0.172
3.666 + 0.233
2.926 1 0.169
Airfoil Noise
3.124 + 0.089
3.563 1 0.158
3.076 1 0.063
regime (table 10. bottom). Here BundleNet achieves better performance than CGAN and CGAN- local on 4 out of metrics (though on 2 of these BundleNet's 95% confidence interval intersects another model's) s). Overall, CGAN-local is the model that is most competitive with BundleNet.
Within the global data generation regime, BundleNet easily outperforms the other three models on all four datasets. CGAN-local comes in second three datasets (Torus, Wine Quality, and Airfoil Noise) while CGAN comes in second on one (Möbius band). We were surprised to find that though BundleNet specifically designed for the purpose of sampling from fibers, also achieves high accuracy when reconstructing the whole distribution. Visualizations of each model's reconstruction of the Torus and Möbius band can be found in figure oo in the appendix with further commentary.
As remarked above, we developed CGAN-local so that we would have a GAN-based benchmark that receives the exact same imput as BundleNet. We hoped to able ascertain whether or not BundleNet's specialized invertible architecture and loss function combination is critical to better model performance. The results above suggest that BundleNet indeed better adapted to solving the problem posed in this paper. On the other hand, the fact that CGAN-local achieves the second best performance on a majority of datasets and metrics suggests that even without the BundleNet framework, there is value to partitioning a distribution into neighborhoods and conditioning each. We believe our results suggest that existing generative architectures cannot satisfactorily solve the problem of learning fibers.
DISCUSSION
Topological priors on fibers: As mentioned in section 4 BundleNet requires one to choose an underlying fiber distribution Dz. The natural choice would be use Gaussian distributions since they are tractable computationally and to some degree universal in statistics. A natural question is: how well can Gaussian prior capture more topologically diverse fibers-for example, circular fibers that arise in tasks that involve periodic systems or invariance to 1-dimensional rotations?
To test this, we trained an invertible neural network map a 1-dimensional Gaussian, 2- dimensional Gaussian, and a uniform distribution on a circle to distribution on an oval in the plane. In each case the model was the same deep invertible architecture with RNVP blocks trained with (forward and backward) KL-divergence. Visualizations of the result can be found in figure As expected the circular distribution quickly converges to the target oval while neither the 1- nor 2-dimensional Gaussian converges to distribution that is visually similar to the oval. This aligns with the basic fact from topology that there is no continuous bijective map from the interval [o, 1] to the circle S' (Lee 2010). Based this observation and the fact that up to homeomorphism the circle and IR are the only two real, connected, 1 1-dimensional manifolds, we investigated which prior would give better results our synthetic datasets.
7

---

Published as a conference paper at ICLR 2022
Using different priors
1-D Goussian
Target
Gaussian
Target
Unilorm circle
Targes
Epoch
40
150
300
Figure 3: Visualization of the convergence of three different priors distribution oval: 1-dimensional Gaussian (Above), a 2-dimensional Gaussian (Middle), and uniform distribution circle (Below)).
Table 3: The Wasserstein-1 metric for fiberwise data generation for models using either a single 1-dimensional Gaussian uniform distribution the circle as prior.
Gaussian prior
Circular prior
Torus (global)
1.640 1 0.020
1.338 1 0.005
Möbius band (global)
0.327 0.006
0.240 1 0.007
Torus (fiberwise)
1.781 + 0.094
1.312 + 0.008
Möhius band (fiberwise)
0.411 + 0.029
0.231 + 0.016
One might expect that the model with Gaussian prior would perform better the Möbius band, whose fibers are intervals while the circle prior would lead to better performance on the torus whose fibers are circles. Surprisingly, the circle prior resulted in better performance on both the torus and Möbius band. A possible explanation for this is that the circle prior is uniform distribution supported on compact set. The same is true for the fibers in the Möbius band and it possible that the model has harder time concentrating the density of Gaussian distribution than collapsing the shape of a circle to (approximately) line segment. On the other hand, data collected Appendix section A.3.2 shows that in some cases the effect of prior choice is negligible. In the end, all of our experiments reinforce the conclusion that while specific prior choices may be important to achieving optimal behavior, our model is relatively robust this hyperparameter.
Fiber topology need not uniform: While the theory of fiber bundles rich and well-studied in math, is perhaps restrictive to have assume that the data that arises in the real world has this very particular and highly-structtred form. Our results on the non-synthetic datasets shown in tables D and 2 however suggest that the neighborhood-ban based approach allows BundleNet to be more flexible. To test whether BundleNets can perform well on datasets whose fibers are not all the same (specifically homeomorphic) we train and test on dataset called the sliced torus that is identical to our torus dataset except that the fiber varies continuously from a point to full circle and back as we traverse the ring.
The quantitative results of our experiments can be found in table and visualizations can be found in figure We note that WGAN and CGAN both struggle to even produce the general shape of the torus, while CGAN-local gets the torus shape mostly right but clumps points from each neighborhood together. Both BundleNet and CGAN-local have trouble with the singularity where the fiber becomes point (though BundleNet appears to handle this better). For reference, each fiber consists of 100 points. most cases we see that generative metrics are comparable to those saw the torus dataset and almost all cases (except global W/ and fiberwise MSMD) the results on the sliced torus are better than the next best model we tested on the torus dataset. This, coupled with superior performance real-worke data that is unlikely take the form of true
8

---

Published as a conference paper at ICLR 2022
OSAN
Figure 4: Model performance on a "bundle" where the fibers have different geometry and topology. (Left, Top row) Three different views of the training distribution for the sliced torus. (Left, Bot- tom row) Three different views of a 70,000 global points generated by a BundleNet after training. (Right) Reconstructions of the the sliced torus using other models.
Table 4: Torus sliced torus. Torus results are juxtaposed with those of the sliced torus to give approximate benchmark for comparison since both distributions have similar scale.
Torus (global)
Sliced (global)
Torus (fiberwise)
Sliced (fiberwise)
MSMD
0.073 + 0.001
0.051 + 0.001
0.017 0.002
0.006 + 0.001
MMD (x10
0.406 = 0.018
2.127 0.071
10.02 0.557
10.361 + 0. 764
Wi
0.461 = 0.020
1.243 0.026
0.251 0.013
0.195 + 0.008
W22
0.793 1 0.108
4.813 + 0.110
0.071 0.008
0.069 1 0.006
fiber bundle, suggest that although our architecture is formulated to deal with situations where the fibers topologically homogeneous, BundleNets retains sufficient expressivity to allow for more complicated relationships the data.
Both this section and the previous investigate topological effects model performance. It is hard to extrapolate too broadly from such small synthetic experiments, but our results do suggest that exploring latent space topology might be an interesting direction for further research.
LIMITATIONS
While the generative results shown above suggest that BundleNet is powerful way to learn fibers, has some drawbacks compared to other models we tested. The most glaring of these is (1) slow training speed, which arises from our implementation of KL-divergence, which has computational complexity O(n²) where is the size of typical neighborhood and (2) memory considerations when the data is high dimensional, since the default implementation attempts compute loss terms using the entire neighborhood. A potential solution to both of these problems, would be to subsample from a neighborhood. We also found that, its current form, BundleNet was not particularly well- suited to image data (partially because of dimensionality considerations). We believe adapting the BundleNet framework to high dimensional data should the top priority for future work.
7 CONCLUSION
Fibers are a useful lens through which to understand machine learning task. However, they have been little explored in the deep learning literature so far. In this paper we showed that several promising, existing, candidate models struggled to consistently model fibers for both real and syn- thetic datasets. Drawing on deep ideas from topology, we introduced a new model architecture, the Bundle Network, which by design can model the fibers of task by breaking the input space into neighborhoods and learning a function on each that decomposes input into a product of its label and all additional variation in the data. We hope this opens door, both for applying methods from dif- ferential topology to machine learning and for further explorations into methods that can help data scientist better model and understand the fibers of their datasets.

---

Published as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT
In the interest of making our results reproducible and able to be easily expanded upon, we make codebase available the public, including implementations of BundleNet well as the WGAN, CGAN, and CGAN-local models we used. We also include training scripts for different models with sensible defaults well evaluation script and examples. Finally, we provide the data used throughout this paper including our specific splits. All of this code and data can be found in our repository at httts:://gttttaie com/ NicoCuurts -etndlkksttwrsss/////
ACKNOWLEDGMENTS
This research was supported by the Mathematics for Artificial Reasoning in Science (MARS) initia- tive via the Laboratory Directed Research and Development (LDRD) investments at Pacific North- west National Laboratory (PNNL). PNNL is multi-program national laboratory operated for the U.S. Department of Energy (DOE) by Battelle Memorial Institute under Contract No. DE-ACO5- 76RL0-1830.
The authors would also like to thank Keerti Kappagantula and WoongJo Choi for introducing them the material science problems that inspired this work.
REFERENCES
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In International Conference on Learning Representations, 2018.
Lynton Ardizzone, Jakob Kruse, Peter Steinbach, Till Bungert, Peter Sorrenson, Alexandre René, and Johnson Yue. Framework for easily invertible architectures, 2021. URL https://
github. www.coo//vv/.o/// BEE//VVLLLLLDEE HDII--DDFFEEAA
T. Brooks, D. Pope, and M. Marcolini. Airfoil self-noise and prediction. 1989.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv:1802.049: 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp- 2180-2188, 2016.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. CORR, abs/1410.855 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. CoRR, abs/1605.08803, 2016. URL http:::/aaxl...rgg///// wwwp://www.g/a/b./b.bb.o /ttpb/////100000003333
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http: /arphiie// ics. uci edu/ml
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Proceed- ings of the 32nd International Conference on Neural Information Processing Systems, pp. 708- 718, 2018.
Jean Feydy. Thibault Séjourné, François- Xaavie Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2681-2690, 2019.
Tingran Gao, Jacek Brodzki, and Sayan Mukherjee. The geometry of synchronization problems and learning group actions. Discrete & Computational Geometry, 6501)2150-21,, 2021.
10

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(25))723-773, 2012. URL
http://jm F.. ww..r/ppprrrs.... /v 3/grttton 2a. html
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp- 6629 6640, Red Hook, NY, USA. 2017. Curran Associates Inc. ISBN 9781510860964.
Dimitris Kalatzis, Johan Ziruo Ye, Jesper Wohlert, and Søren Hauberg. Multi-chart flows. arXiv preprint ar 0tt.::220060350000 2021.
Leonid Vitalevich Kantorovich. On one effective method of solving certain classes of extremal problems. In Dokd. Akad. Nauk. USSR, volume 28, pp. 212-215 1940.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018. URL
http::s//proceedingg. wiigggs.....ppp..... m...press 1y80/kimmb.htmm
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 1rXiv:1312.61 2013.
arXiv preprint
Felix Knöppel and UIrich Pinkall. Complex line bundles over simplicial complexes and their appli- cations. In Advances Discrete Differential Geometry, pp. 197-239, Springer, Berlin, Heidel- berg, 2016.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 - 86, 1951. doi: 00.1114/amm/111777996 URL httph::t/////oo /rrg 10.1214/aa /omms 11177729641
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-46899 2017.
Hsin-Ying Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Mancesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pp. 35 35-51, 2018.
John Lee. Introduction to topological manifolds, volume 202. Springer Science & Business Media, 2010.
Zinan Lin, Kiran Koshy Thekumparampil, Giulia C Fanti, and Sewoong Oh. Infogan-er: Disentan- gling generative adversarial networks with contrastive regularizers. In ICML, 2020.
Qi Mao, Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking gener- ative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 1429-1437, 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 2014.
Olof Mogren. C-mn-gan: Continuous recurrent neural networks with adversarial training. arXiv preprint 1=Xiv:6611.09004 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML215. pp. 1530-1538. JMI.. 2015.

---

Published as a conference paper at ICLR 2022
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2234 2242, Red Hook, NY, USA, 2016.
Curran Associates Inc. ISBN 9781510838819.
Luis Scoccola and Jose A Perea. Approximate and discrete euclidean vector bundles. arXiv preprint arXiv:2104.07563 2021.
H. Seifert. Topologie Dreidimensionaler Gefaserter Räume. Acta Mathematica, 60(nonc):147
238, 1933. doi: 10.10007/BF02398271.. URL
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe, Disentanglement by nonlinear ICA with general incompressiible-flov networks (GIN). CoRR, 0oss2001.044872 2020. URL /ttps::::////x
2xc//ab/22001.04/ 04872
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1-9, 2015. doi: 10.1 10.11999CVR2222557988444
Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 1969.
Qing Wang, Sanjeev R. Kulkarni, and Sergio Verdu. Divergence estimation for multidimensional densities via k-nearest-neighhor distances. IEEE Transactions on Information Theory, 55(5): 2392-2405, 2009. doi: 10.1 (009/TT1200992016//
Hassler Whitney. spacess. Proceedings of the National Academy of Sciences, 21(7):464- 468, 1935. ISSN 0027-8424. doi: 10. 11.73/pa...21.7.664 WRR.https::///t www. pnas org/
content/21/7/455
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein diver- gence for gans. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 653-668, 2018.
Ke Ye and Lek-Heng Lim. Cohomology of cryo-electron microscopy. SIAM Journal on Applied Algebra and Geometry, 1(1):507-535, 2017.
A APPENDIX
A.1 DATASETS
In this section we document our method for either generating or modifying datasets used in this work.
Torus: This is purely synthetic dataset that uses an embedding of a torus (a 2-dimensional sur- face/manifold) in RR Given two radii and R, the torus parameterization is given by:
T(r, R) {(( + r cos 0) cos Φ, (R + r cos 0) sin di r rsin 0) 0 S Φ, < 2m).
Our data uses R = 8 and = 2 and consists of 1000 points generated by uniformly sampling parameters and 0 from [0, 2T). Notice that this does not represent uniform sampling from the surface itself since it more heavily represents the inner ring of the torus. When we evaluate, we use rejection sampling sample uniformly from the surface. The base space Y is given by a circle of radius 4.5.
This is our second synthetic data and another surface in R This one is given by the parameterization:
Möbius band:
((, R) ((Rcos 0, Rsin 0,0) Ao A (- cos 0, sin 0) 0 < 0 2n, -r 5
12

---

Published as a conference paper at ICLR 2022
where Ag is the rotation matrix around the 2-axis through the angle and By is the rotation matrix around the x axis through the angle 8. Our data (both for training and evaluation) consists of 1000 points with space Y is radius 4.5 circle.
2 and R = 8 which are sampled uniformly over the parameters 0 and The base
The two real-world datasets, Wine Quality (Cortez et al. 2009) and Airfoil Noise (Brooks al. 1989), are described more completely on their pages in the UCLML repository (Dua & Graft (017)) The Wine Quality dataset provided two files (for red and white wine) with the intention of predicting quality from 11 measured quantities. We combine these two sets into one so that we have 1l imputs and two (both categorical) outputs: color and quality. The Airfoil Noise dataset has input parameters and one output parameter (scaled sound pressure level), which we left unaltered. As a pre-processing step we normalized all parameters in each dataset to span the range 0 to 10 by applying x >> 10
M (x: m)) where and M are the minimum and maximum values of each parameter. We made an 80/20% train/test split for each.
A.2 LOSS FUNCTIONS AND EVALUATION METRICS
There has been much work in the world of GANs to establish metrics that can quantify the quality of a generative model. This includes the widely-used Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al. 2017). Many of these metrics, however, rely on the as- sumption that the data has the structure of an image. For instance, in both the methods mentioned above one computes these metrics by using a pretrained Inception v.3 model (Szegedy et al.) 2015) to extract latent features from generated images. One then uses methods from statistics/optimall transport like the Wasserstein distance or Kullback-Leibler divergence to evaluate similarity under the assumption that data in these latent spaces follow Gaussian distributions.
Since the data we are using are not images, and since the assumption that our data follows some Gaussian distribution is catastrophically restrictive, we choose some standard two-sample tests to quantitatively compare the performance of different models. One could make the argument that with the blessing of low-dimensionality we are able to more directly access the metrics that are only being approximated in methods such as those for images that rely pretrained classifier model.
Mean squared minimum distance: The simplest of our metrics, the mean squared minimum dis- tance (MSMD) takes two samples S1 and S2 and computes
MSMD(S,, S2)
1 S
Σ S1ES;
min | 1
E3ES2
This naïve approach is not as robust to different distributions as the methods that follow, but in the case that S2 collection of points uniformly distributed some known space (e.g. a circle in space corresponding to the true fiber over a point), this number is more readily interpretable as an approximate measure of how "realistic" the points in S1 are, given the region containing S2.
Kullback-Leine divergence: The KL-divergence, or relative entropy was a test introduced by Kullback & Leibler (195T) apply an enformationntheoretii approach the problem of detecting divergence of two samples by attempting to quantify the amount of signal (or information) that one could use to distinguish between the two. For instance, given two distributions and Q on with densities p and this quantity is given by
[ p(x) log
p(x) g(x)
(PII)) -
dæ.
problem that needs be solved to apply the Kl.-divergence to our samples is to fix method of density estimation. A suggestion by Wang al. (2009) is to (for some integer hyperparameter k) approximate the density using the 4 nearest neighbor of a point in each population. This approx- imation is fast and easily made to be differentiable and for this reason function and as an evaluation metric.
it both in our loss
Note that the KL-divergence is not an honest metric and in particular is not symmetric. The for- ward divergence (KL-fwd) is D (PIIQ) where P is the true/target distribution and the backward divergence (KI.-bwd) is DRI. (QI)P) One way of thinking of this difference is that D (Pll))
13

---

Published as a conference paper at ICLR 2022
measures how well covers P and vice versa. We include both separately in results to give holistic view of the performance.
Wasserstein metric: The Wasserstein family of metrics discovered in many works including (Kan- torovich 1940) and [Vasersteinn 1969) are based on the idea of measuring the amount of work that would need to be done to transport the (probability) mass of one distribution another. Let (X, dx) be a metric space and jr, y probability measures on X. For any p > O, one defines the p- p-Wasserstein distance between these two distributions be
1/D
I (XN)EXX dx y) dy v)
W W,(,,
inf (TTT
where [[,,,, p) denotes the space of probability measures on X' x X' with margins and v. To implement this, we piggyback off the work of (Feydy et al., 2019), who provide wonderful library geomloss of differentiable Py PoTchh implementations of many two-sample distances.
Maximum mean discrepancy: First introduced (Gretton al. 2012). the maximum mean dis- crepancy (MMD) between two samples is computed by appealing to a dual problem: consider collection functions F from our probability space X' to R. Then the MMD can be thought as an expression of how "different" two distributions can look under different choices of f E F.. More concretely,
MMD(F:P.Q) sup (E-~r
fEF
An implementation of MMD can be found in geomloss (Feydy et al. 2019) using single Gaus- sian kernel.
A.3 HYPERPARAMETERS
Below document our choices of hyperparameters for our models. 'Neural net depth/width' de- scribes the structure the MLP that used each model (although in subtly different ways). For our model, these parameters describe the neural nets that predict the weights and biases for our RNVP blocks. For the GAN models, these parameters control the size/shape of the generator and discriminator. 'Num. neighborhoods' is only applicable in models that compute neighborhoods to condition upon (i.e. BundleNet and CGAN-nbhd).. "Num. invertible blocks' gives the number of RNVP blocks and only applies to architecture. 'Num. circular priors' again only applies to our model and implicitly defines the number of Gaussian priors we use as well. The formula is B + 2C + G D where C is the number of circles, B the dimension of the output data, G the number of Gaussian priors, and D the total/latent dimension.
Mathematically, the combination of mne-dimeniionll priors represents (tensor) product of distribu- tions, but in practice this has simple form. For each neighborhood, data from the bundle is pushed forward through the encoder to get sample statisties including the coordinate-wise means and stan- dard deviations as well as the mean and standard deviation of the magnitudes for each circular prior. These sample statistics are used to pick radius for each circular prior and to reparameterize stan- dard normal distribution for the Gaussian priors. Every circle is encoded in two parameters circle with the radius determined above and every Gaussian (again, formed using sample statistics) occupies single parameter. The values sampled independently from their respective distribu- tions and then concatenated into a vector in latent space representing sample from Dz.
A.3.1 EFFECT OF NUMBER OF NEIGHBORHOODE
GENERATION
In this section we explore how the number of neighborhoods used to partition Y affects the (global and fiberwise) generative performance of BundleNet. We note that for both synthetic datasets the topological character of the inverse images of neighborhoods in input space X changes dramatically as one moves from single neighborhood (corresponding to global trivialization) to two more open neighborhoods. When we use a single neighborhood torus Möbius band respectively. When we use two more neighborhoods, π -U) admits deformation retraction onto the circle g! in the case of the torus, and contractible in the case of the Möbius band. Under the assumption that space becomes easier to model as its topological complexity decreases, we hypothesized that we would see a significant improvement in performance when moving from 2 neighborhoods, but would not substantial improvements afterwards.
Y. then म -U) is of course a
14

---

Published as a conference paper at ICLR 2022
Table 5: Hyperparameter choices used in all models trained and evaluated during the course of this work (unless otherwise noted in the main text). If multiple values are given, the format is (torus, Möbius band, wine, airfoil).
MODEL HYPERPARAMETERS
BundleNet (ours)
WGAN
CGAN
CGAN-nbie
Initial learning rate Latent dimension
10-1
10-1
10-4
10-1
(8, 8, 8, 12)
10
10
10
Neural net depth.
5
Neural net width.
512
128-1024
128-1024
128-1024
Num. neighborhoods
(25, 25, 13, 25)
N/A
N/A
(25, 25, 13, 25)
Num. invertible blocks
N/A
N/A
N/A
Num. circular priors
N/A
N/A
N/A
Table 6: Generative performance (Wasserstein-l distance) of BundleNet the Torus dataset in terms of the number of neighborhoods Y used. Each model was trained once and evaluated multiple times to generate 95% Cls for a particular of model weights.
TORUS GENERATION PERFORMANCE NUMBER OF NEIGHIORIIOODS
Number of Neighborhoods
Global
Fiberwise
0.579 + 0.030
6.059 0.289
2
0.575 + 0.052
0.282 + 0.012
5
0.573 + 0.037
0.303 0.007
10
0.660 + 0.034
0.425 + 0.016
25
0.373 + 0.036
0.230 0.017
100
0.632 + 0.028
0.439 + 0.024
The results of our tests (found table 6 fit well with our observations about topological com- plexity. We find that global generative performance seems be strongly tied to the number of neighborhoods but that fiberwise performance increases dramatically as one moves from to neighborhoods. For any number of neighborhoods greater than we do see particular im- provement to either global or fiberwise performance. These results suggest that the use of local trivializations is mainly helpful for the task of modeling fibers themselves. but may not be as criti- cal to modeling the distribution as whole. Note that this is somewhat in contrast to the improved performance of CGAN-local over CGAN, which indicated that least within the GAN framework, use of local neighborhoods was helpful for both global and fiberwise metrics.
A.3.2 EFFECT OF TOPOLOGICAL PRIOR CHOICE ON MODEL PERFORMANCE
Table 7: Generative performance (Wasserstein-1 distance) of BundleNet on the Airfoil dataset in
terms of the number of circular priors used in latent space. The total dimension of the latent space was 18. 1-dimensional Gaussians account for additional dimensions where circle priors were not used.
AIRFOIL GENERATION PERFORMANCE BY NUMBER OF CIRCULAR PRIORS
Number of Circular Priors
Global
Fiberwise
1,283 + 0.013
2.943 = 0.056
2
1.037 = 0.008
2.899 = 0.095
1.172 = 0.014
2.925 = 0.093
1.124 + 0.007
3.072 + 0.072
15

---

Published as a conference paper at ICLR 2022
3.0
2.5
2.0
1.5
1.0
Global Fiberwise
Number circular oriors
Figure 5: Effect on generation error (Wasserstein-I distance) on the Airfoil Noise dataset by number of circular priors. Error bars represent 95% Cls.
In this section we examine how the choice of topology for the priors (circles vs. Gaussians) affects generative performance in the wild. Recall that we found a slight advantage to using circles when exploring our synthetic datasets in section 6 We trained BundleNet using 18-dimensional latent space with different numbers of circular priors (with the rest of the parameters one- mne-dimensional Gaussians). The results can be found in table[aand figure For the most part, these results show that Airfoil Noise generative capacity is not strongly affected by the number of circular priors chosen. We hypothesize that this is due to two factors: first, this dataset a difficult one (for instance, we that the margins between different models are very slim) and second, there are parameters in this dataset that are naturally modeled as circle. The fact that we saw better performance on our synthetic datasets suggests that there still value tuning this hyperparameter depending on the data under consideration. We believe that the relative invariance BundleNet shows to choice of priors speaks the robustness of our model a wide array of hyperparameter choices.
A.4 A COMPARISON BUNDLENET AND OTHER MODELS
BundleNet
WGAN
CGAN
CGAN-local
Torus
Mobius Band
Figure 6: Visualizations of synthetic dataset reconstruction for each model type. To produce each image, 100 evenly distributed base points were chosen and we generated 200 points in the fiber over each (where such capability available). The colors correspond to the angle parameterizing the circle in the base space. Note that do not expect the colors in WGAN's reconstruction to be well-organized since this model has no way of consistently modeling specific fibers like the other models do.
16

---

Published as a conference paper at ICLR 2022
We begin this section with visual and qualitative analysis of the performance of BundleNet, WGAN, CGAN, and CGAN-local models on our synthetic datasets. The images in this section restricted by being 2D projections of 3D point clouds, but they should serve to help illustrate why the tables in section 5.2 and section A.5 appear as they do.
Perhaps the most immediate observation that can be made is that CGAN suffers complete collapse on the torus dataset. This corresponds to high loss both the global and fiberwise regimes (it is the worst-performing model this dataset). CGAN does considerably better on the Möbius band dataset (it is the second-best performing according to the Wasserstein-1 metric, c.f. table []. Even in the case of good global generation metrics, however, CGAN fails capture fiber structure, as can be seen in the way that different colors which should be localized to specific sections of the band, are distributed throughout the reconstruction. We discuss this failure more carefully below in section A.4.1 We also discuss how we attempted to train CGAN in different ways so as to make it more amenable fiber discovery.
Of all the GAN architectures, the reconstructions generated by CGAN-local appear to be the most faithful to the true distribution. Nevertheless, one can detect ways that CGAN-local's reconstruc- tions deviate from the torus and Möbius band respectively. In both reconstructions, CGAN-local tends to cluster neighborhoods together giving the resulting visualization a "banded" quality. Note that the bands in the Bundlenet torus not due to similar failing, but actually visible indi- vidual fibers (100 evenly-distributed fibers around the base circle). This behavior could account for CGAN-local's (relatively) poor global generative performance in the main results. We also intriguing failure in CGAN-local's reconstruction of the Möbius band where some of the fibers are smeared outside of the band's bounds. We speculate that this smearing occurs because CGAN-local struggles to capture the twist in the band. This can also be seen in the sliced torus in figure 4 which we analogously hypothesized was related the singularity present where the fiber becomes point. An interesting avenue for further research is how topology affects the ability of generative model to approximate distribution.
WGAN is somewhat unique among tested architectures in that it does not incorporate conditional aspect to learn fibers over point. As such, the point clouds generated by WGAN map basepoints uniformly throughout the generated data. WGAN has difficullt time recreating the circular fibers for the torus dataset, but gives reasonable reconstruction of the Möbius band. Again, these visual- izations align with our quantitative results.
A.4.1 CGAN, FIBERS, AND TRAINING
Table 8: Generative performance metries for the torus dataset for CGAN variations
GLOBAL
BundleNet (ours)
CGAN
CGAN-local-mixed
CGAN-local
MSMD
0.073 = 0.001
36.41 0.365
58.63 0.355
0.930 0.013
MMD (x10-3)
0.406 0.018
35.04 0.214
173.4 1.173
2.336 + 0.054
KW-
0.812 0.003
10.62 - 0.021
11.41 = 0.023
3.615 0.028
Kwd
0.221 0.021
11.97 0.013
15.78 + 0.010
4.070 + 0.044
Wi
0.461 0.020
7
7.821 + 0.017
1.610 0.031
W2
0.793 + 0.108
25.76 0.097
31.60 - 0.123
2.715 0.118
FIBERWISE
BundleNet (ours)
CGAN
CGAN-local-mixed
CGAN-local
MSMD
0.017
1.911
0.370
0.148 0.011
MMD (x10=3)
10.02 4.557
83.46
153.8 1.394
28.74 1.941
KL-
5.047 = 0.130
16.91 + 0.110
18.01 = 0.064
7.440 0.180
Kwd
1.907 0.090
9.422 + 0.059
11.45 0.040
3.122 + 0.194
Wi
0.251 0.013
7.996 + 0.015
7.980 = 0.021
0.450 = 0.018
W2
0.101 + 0.010
32.60 0.114
32.73 + 0.167
0.183 0.010
17

---

Published as a conference paper at ICLR 2022
In this section we focus CGAN and its failure modes on synthetic data. We show how improve CGAN by leveraging some of the same methods developed for BundleNets. To this end, document attempt to explore two different training strategies for CGAN, which designate CGAN-local-mixer and CGAN-local. In both incarnations, we compute neighborhoods of and their representatives and then these representatives to condition upon. During training, of two natural choices can be made: constrain each batch to points from a single neighborhood or sample each batch from across all neighborhoods. We call the former CGAN-local and the latter CGAN-local-mixet We view the progression from CGAN to CGAN-local- mixed to CGAN-local as capturing the development of a conditional generative architecture that is better suited to the fiber task. Furthermore, we see the poor performance of CGANC CGGN--ocal-mixed as evidence that BundleNet's performance cannot be attributed simply reducing the conditioning set from a continuum to finite number of neighborhood representatives.
The results in tableB]occmmen the relative performance of these models on the torus dataset. We see that the naive CGAN is poorly suited to the task and that switching to neighborhood conditioning degrades performance across several metrics. On the other hand, restricting
with CGAN-local-miss each batch to single neighborhood (CGAN-local) results in model that performs much more reasonably (but still inferior to) BundleNet. We view CGAN-loca as our best effort to equip a GAN with the power perform the fiber generation task without fundamentally altering the architecture. Thus this serves as our primary point of comparison the main paper.
A.5 FULL RESULTS
Below we recreate the full table of results with all metries tested.
Table 9: Generative performance metries (global). Each metric is applied to trained model as detailed in section 5.1 with 95% confidence intervals determined by bootstrapping. Conditioning, when built into the model, is used to generate points over (uniformly sampled) base points to con- struct the point cloud.
TORUS
BundleNet (ours)
WGAN
CGAN
CGAN-local
MSMD
0.073 1 0.001
4.858 + 0.042
4..4160..363
0.930 0.013
MMD (x10=3)
0.406 = 0.018
3.832 = 0.039
35.04 0.214
2.336 = 0.054
Kwd
0.812 + 0.035
6.300 + 0.035
10.62 0.021
3.615 + 0.028
Kl.-Bwd
0.221 + 0.021
+
11.97
4.070 + 0.044
Wi
0.461 + 0.020
2.011
1.610 0.033
W2
0.793 0.108
3.442
25.76 1 0.097
2.1118
MÖBIUS BAND
BundleNet (ours)
WGAN
CGAN
CGAN-local
MSMD
0.029 0.001
3.187 + 0.026
1.167 0.016
0.404 + 0.007
MMD (x (x10-3)
0.509 + 0.026
5.875 + 0.038
6.009+0.00
4.236 + 0.045
KLdFwd
1.031 + 0.017
7.601 + 0.040
5.936 0.023
4.686 + 0.041
KL-Bwd
0.002 + 0.018
7.174
6.723 1 0.054
0.054
Wi
0.264 + 0.004
2.048 + 0.013
6.497
+ 0.101
W2
0.060 0.009
2.829 + 0.053
2.384 + 0.051
40.63 0.814
18

---

Published as a conference paper at ICLR 2022
WINE QUALITY
BundleNet (ours)
WGAN
CGAN
CGAN-local
MSMD
1.581 + 0.023
68.26
1.436
6.630
3.275
MMD (x 1x000
0.820 0.014
95.59 + 1.488
10.67
1.09
KL-Fwd
3.158 + 0.062
0.142
9.966
6.393 + 0.054
KL-Bwd
2.902 + 0.120
115.7 0.336
0.132
10.067 10067=0000 0.097
W1
1.733 + 0.011
99.22 0.392
= +.054
2.114 0.011
W/2
2.155 + 0.036
6271. + 40.40
8.950 + 0.066
2.567 + 0.028
AIRFOIL NOISE
BundleNet (ours)
WGAN
CGAN
CGAN-local
MSMD
0.512 1 0.022
24.54 + 0.356
3.474 0.057
1.300 + 0.030
MMD (x10 10)3
1.840 + 0.013
38.86 + 0.480
4.222 + 0.044
2.742 0.066
KL-Fwd
5.810 + 0.089
14.87 0.059
1.131=00006
7.247 = 0.095
KL-Bwd
2.430 + 0.018
15.96 #
6.716 0.036
1
Wi
1.448 + 0.022
4.741 0.034
2.508 + 0.025
1.735 = 0.013
W22
1,888 + 0.123
13.95 + 0.198
4.964 + 0.120
2.121 + 0.050
Table 10: Generative performance metrics (fibewwise). Results were obtained by randomly sampling 5 base points and generating points in the fiber over each. Each metric is applied to the generated points and points sampled from the true distribution and averaged. Intervals represent 95% CIs 10 repeated experiments. WGAN is excluded since it has no built-in way to condition on the base point.
TORUS
BundleNet (ours)
CGAN
CGAN-local
MSMD
0.017 + 0.002
29.81
0.148 0.011
MMD (x10 3)
10.02 0.557
83.46 1.230
28.74
KL.-Fwd
5.047 + 0.130
16.91
0.110
7.440 +.188000100 0.180
KL-Bwd
1.907 + 0.090
9.422
3.122 +
Wi
0.251 + 0.013
7.9015
0.450
+ 0.018
W2
0.101 + 0.010
0.110
0.183 0.010
MÖBIUS BAND
BundleNet (ours)
CGAN
CGAN-local
MSMD
0.009 = 0.001
0.067 = 0.011 68.85 144.0
0.125
MMD
(x10-3)
20.82 - 2.724
KL-Fwd
7.646 = 0.125
1
10.33
KL-Bwd
1.856 = 0.106
1566
5.507 0.158
Wi
0.279 + 0.011
9.860 0.038
8.228 + 0.904
W22
0.071 + 0.008
58.47 + 0.364
193.8 + 38.25
WINE QUALITY
BundleNet (ours)
CGAN
CGAN-local
MSMD
3.234 0.883
10.70
5.581
MMD
(x10-3)
21.84 13.08
48.37 40.40
71.77
KL-Fwd
-0.091 = 1.186
5.942 + 1.303
2.245
Kwd
0.856 0.474
10.56 + 1.468
7.164
Wi
1.917 = 0.172
3.666 - 0.233
2.926 0.169
W2
2.407 0.763
7.941 + 0.951
5.263 0.634
19

---

Published as a conference paper at ICLR 2022
AIRFOIL NOISE
BundleNet (ours)
CGAN
CGAN-local
MSMD
1.920 = 0.157
2.793 0.366
2.199
MMD (x 10 3)
35.872 + 1.253
0.9935
37.96 2.121
KL-Fwd
2..944 + 0.430
-1.259
0.5543
KL-Bwd
3.250 = 0.085
4.908 0.175
4.863
W1
3.124 = 0.089
3.563 0.158
3.076 0.066
W2
6.744 + 0.459
8.054 + 0.711
6.394 + 0.261
20


Reviews:
[{"review_id": "zaO9n0_0Ix", "reviewer": "Reviewer_6GTJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths of the paper\n\n- I like the idea to model many-to-one tasks in machine learning using fiber bundles. I find this natural and elegant. The authors do a good job of motivating introducing and motivating the model.\n- The proposed architecture is natural and more or less directly maps the fiber bundle technology to neural networks to learn fiber bundles from data.\n- The author propose an effective training strategy to optimize the parameters of the proposed model which includes multiple well-motivated loss functions.\n- Empirically the proposed architecture seems to outperform conditional generative models.\n- I very much appreciate the discussion of the topological aspects and the exploration of the influence of topology of the latent space on performance.\n- Overall the paper is well written and easy to follow. The illustrations help understand the technical points in the text.\n\n\n### Weaknesses of the paper\n\n- At a high level the fiber bundle idea (in a machine learning context) is very similar to that of conditional generative models. The authors argue that a specialized architecture is warranted and that networks like conditional flows cannot work well. These discussions are however at a rather heuristic level, with argument such as ... . To me it remains unclear why conditional networks underperform by such a significant margin.\n- One claim the authors make is that the improved performance of their model (relative to conditional generators) comes from space partitioning, but partitioning can be interpreted as a particular choice of conditional architecture. In conditional generative models the only conditioning object is $y$; same seems to hold in BundleNet except that $y$ enters the architecture in two different ways, once through the argmin calculation of the nearest mean, and once on the \"reverse\" input on the right hand side. The question that comes to mind is then: why not use a slightly more advanced conditional generator architecture? My gut feeling is that it should perform well.\n- I am uncertain about the significance of the \"CGAN-nbhd\" model in your comparisons. You state that this model is your best effort at modifying a standard architecture to sample from fibers, but its performance is _worse_ than that of its unmodified version. I am not sure about the message: that including fiber-bundle-specific details in a model makes its performance worse? Arguing from empirics, it means that the unmodified model is closer to the bundle ideal, no? I would suggest to remove this model from all comparisons and perhaps add a proper \"in-between\" model that actually does better than vanilla conditional generators. It is impossible to interpret why the performance of CGAN-nbhd is the way it is (it may have to do with things like hyperparameters, architecture details, training strategy, ...)\n- Overall, I would like to see a much more thorough and convincing comparison between the various models and a better effort at training the existing conditional architectures. (Again, there are no convincing hypotheses, tested numerically, for why conditional generative models perform worse.)\n- The current architecture seems to be suitable for very low-dimensional datasets; things like $k$-means are hard to motivate in higher dimension. Could you discuss examples and implications of applying your ideas to high-dimensional problems?\n- I believe that considerable care is required in interpreting the empirical statements about the influence of topology. I am not convinced that these empirical results are broadly representative (they are likely to strongly depend on the chosen models and hyperparameters). What is more, many real-world data distributions are believed or known to be low-dimensional yet invertible flows model them without difficulty. The example shown by the authors might be an artifact of low dimension for two reasons: (topological) low-dimensional topology and embeddings are more quirky than high-dimensional and (expressivity) the flow networks used have limited expressivity when approximating functions $\\mathbb{R}^2 \\to \\mathbb{R}^2$. They might work better in high dimension.\n- A minor point: I would appreciate having the dataset descriptions in the main text. The main results of the paper are empirical, on Airfoil Noise and Wine Quality datasets—it is hard to build intuition without knowing the relevant details of the datasets. One would need to go to the appendix, but even there you refer to some external resources.\n\n### Notes and questions\n- Out of curiosity: your explore the case where the fiber bundle structure is essentially known. Are there applications where such a structure would have to be discovered? Could you adapt your network to these scenarios?\n- Is there a connection between your model and the recently-proposed multichart flows? (I am thinking about the label space partitioning)\n- How are the conditioning vectors $r_i$ structured? (are they simply scalars or they have the structure of labels $y$ or inputs $x$?)\n- Since you're using an invertible network, it seems to me that $\\text{dim}(y) + \\text{dim}(z) = \\text{dim}(x)$, but $x$ corresponds to points in the ambient space. Many datasets are believed to be intrinsically low-dimensional. Would it then make sense to use injective flows like https://arxiv.org/abs/2102.10461 or https://arxiv.org/abs/2003.13913 instead of invertible flows?\n", "summary_of_the_paper": "This paper proposes a neural network to model data (or machine learning tasks) with fiber bundle structure. Machine learning tasks like regression or classification are often many to one. They assign a \"label\" (discrete or continuous) $Y \\ni y = \\pi(x)$ to each $x \\in X$. Many different $x$ may receive the same label. It is interesting to explore / parameterize the set of all $x \\in X$ that yield a given label $y$, that is to say, the set $\\pi^{-1}(y)$. This is nicely modeled by the topological construction of a fiber bundle (a space that is locally a product space). Operationally, it corresponds to a covering of the label space $Y = \\bigcup_{i} U_i$ by \"patches\" $U_i$ so that $\\pi^{-1}(U_i)$  are homeomorphic to product spaces $U_i \\times Z$, with $Z$ being the fiber space. \n\nThe proposed network is in essence a conditional generative model with the added twist that the conditioning is also performed on the \"patch\" that a label (or a regression target) belongs to (for training; inference is only performed in the \"reverse\" direction of fiber sampling.)\n\nThe authors show interesting results in modeling familiar low-dimensional bundles as well as real-world datasets. Comparisons with conditional generative models suggest that the new architecture may be a better fiber bundle model. Finally, the authors numerically explore the role of \"latent space\" topology, a welcome addition.\n", "summary_of_the_review": "This interesting paper suggests that fiber bundles are good models of many-to-one tasks in machine learning. The authors propose a deep neural network to model fiber bundles and argue that it outperforms conditional generative models. I find the premise of the paper natural and convincing, but the numerical results, evaluations of conditional generative models, and the empirical claims about the influence of topology lacking, hence my recommendation. Overall I think the paper has a lot of potential but the narrative and the comparisons should be much more solid, especially when it comes to existing conditional generative models and diagnosing the (topological) modes of failure. My initial recommendation is thus a rejection but I am more than willing to adjust it after the author response. (I would've chosen a 4 but it is not available anymore.)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lFHtphUC26t", "reviewer": "Reviewer_Hz2R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths] \n1) (A new problem) The formalization of a new machine learning problem, i.e., the problem of learning fibers of a ML task.  \n2) (A new solution) An effective scheme is proposed to solve the fiber learning problem with encouraging results.\n\n\n[Weakness]\n1) (The motivation). The proposed fiber learning problem is interesting and novel to me. However, the rationality of modeling the many-to-one relationship via the strict \"product structure\" seems not sufficiently explained. It is suggested to give more vivid examples in CV/ML to show the model's suitability.\n2) (Conditioning may weaken the expressive power). To avoid training a different model for each $U_i$ in $\\mathcal{U}$, conditioning is used in the proposed BundleNet. However, using conditioning instead of optimizing may limit the model's expressive power. \n3) (Clarity of introduction to BundleNet). The description of BundleNet seems not sufficiently clear. I find it a bit difficult to understand the whole process. For example, \"elements of $\\mathcal{R}$ are in bijection with neighborhoods contained in $\\mathcal{U}$ with the objective of having $\\Phi_i$ play the role of $\\psi_i^{-1}$\". It is suggested to rewrite this section for better readability. \n4) (Typos). There are typos in this paper. For example, \"Finally, each measurable subset $U\\in Y$\"  --> \"$U \\subset Y$\".\n \n\n\n ", "summary_of_the_paper": "This paper proposes the Bundle Networks as a framework to learn many-to-one maps. The empirical performance is promising on two synthetic datasets and two real datasets. \nThe main contribustions are three-fold: \n1) The problem of learning the fibers of a ML task is formalized.\n2) An approach is proposed to learn the fibers.\n3) A family of deep generative models called a Bundle Network is designed.\n ", "summary_of_the_review": "This paper proposes a new problem, i.e., the fiber learning problem of a ML task, as well as an effective solution to it. The proposal is novel and interesting to me. However, the motivation, the clearity, and the empirical support can be improved.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "k0CqIA5j9Et", "reviewer": "Reviewer_tPZF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pro*\n\n+ The paper is clearly written and easy to follow. The figures are of high quality.\n\n+ The method outperforms the baselines at the studied generative modeling tasks.\n\n+ The method is based on a standard construction in differential topology. As such the underlying concepts are mathematically well-founded. Bringing these ideas to machine learning is quite interesting on its own.\n\n*Contra*\n\n- The labelling function on real world data is typically not a fiber bundle $Z\\to \\operatorname{supp}(\\mathcal{D_X}) \\stackrel{\\pi}{\\to} Y$. In other words, the local trivialization the networks tries to learn do not exist.  \nAdmittedly, this limitation is discussed towards the end of the paper in section 6. However, from my point of view, this should be stated more prominently already in section 3.1, when introducing the general idea. Also, it's theoretical implication should have been explored to some extent.\n\n  The empirical examination of this issue in section 6 is insufficient. The topological space which is studied therein, is locally homeomorphic to $\\mathbb R \\times \\mathbb S^1$ except on a measure zero set (with respect to the uniform distribution). So it is unsurprising, that the algorithm is able to learn this space. Overall, the setting is too simple to transfer to real world data, whose topology usually is much more difficult.\n\n  I also want to remark, that Figure 4 reveals problems with predicting the geometry (opposed to the topology) close to the singularity. Do the baselines (WGAN, CGAN) suffer from similar artifacts? Could you provide similar plots as Figure 4 for them?\n\n- The method introduces new hyperparameters, in particular the fiber space $Z$, a probability distribution $\\mathcal {D_Z}$ on the fiber and the number $q$ of local homeomorphisms (centroids for k-means). Analysis of the robustness with respect to these choices on real world data is missing.  \n\n   Regarding $q$, according to the theory, there should not be an issue, if $q$ is sufficiently large. Is this indeed true? Does choosing $q$ too large lead to a noticable increase of computational cost.  \n\n  Regarding $Z$ and $\\mathcal{D_Z}$. Choosing these seems not obvious at all on non-synthetic data. Are there any heuristics? Is the prediction robust with respect to these choices? Are there any tradeoffs regading the fiber dimensionality?\n\n- (minor) Figure 6 in Appendix A.4 is not sufficiently described. It is unclear, which points come from training data and which ones come from the reconstruction. Is only the bottom right reconstructed?", "summary_of_the_paper": "The paper introduces a new architecture for generative modeling, called Bundle Networks. This is based on the mathematical concept of fiber bundles and allows to explore the networks fibers above the labels (their preimages). Training the network then corresponds to learning the local trivialization of the fiber bundle, i.e., of the labeling function. ", "summary_of_the_review": "The idea of the submission is quite interesting and I really like its presentation in the paper. However, the theoretical setting seems not to be applicable to real word data. Furthermore, a study on the robustness with respect to the additional hyperparameters is missing. I therefore give a score of 5.\n\n----\n\n**Update**\n\nI thank the authors for addressing my concerns in the update. It seems that the good performance is rather independent of particular hyperparameter choices and prior distributions, but mostly from the guiding idea of allowing the model to be a *local* (as opposed to *global*) trivialization, as indicated by the experiments with q=1. Yet, a convincing argument is missing. In general, I still think, that there should have been a theoretical study under which conditions and  assumptions on the data Bundle Networks can be successful. \n\nI raise my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "6-uTfGjeb5", "reviewer": "Reviewer_asbk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is presented clearly and succinctly and is generally well written, and the technical contributions are novel as far as I can tell. I believe this work has potential in influencing future papers and could become important in the future. \n\nWhen the range is continuous it seems to me that by choosing the clusters to be large or small, one has a trade-off between the \"true\" fibers for very small clusters, and a regularised version for large ones. It would be interesting to see this discussed in a bit more depth. I would also like to see a little more elaboration on how the number of neighbourhoods for k-means affects the performance, as this seems to affect the overall architecture quite a bit. \n\nIt would also add a lot if the task of learning the fiber structure was motivated more by e.g. an application where the fiber structure is helpful in analysing the problem.\n\nMy main point of contention is that the datasets considered are very low dimensional, and the manuscript mentions that the model trains very slowly in higher dimensions, but this isn't shown and I suspect that the huge improvement in performance over the baselines would drop off pretty fast as the dimensionality increases. It would be interesting if this could be expanded on (maybe in the appendix).\n\nOverall I think this is a well written and stimulating paper with theoretically motivated technical innovation. The experimental results look impressive besides the lack of any higher dimensional results, but as this is a more theoretical paper, this omission does not hurt the manuscript too much.\n", "summary_of_the_paper": "The manuscript introduces Bundle Networks which are neural networks designed with an explicit fiber bundle in the architecture. With this in hand one has an explicit description of the fibers of the fibers associated with a machine learning task. The model is demonstrated on both artificial data, showing that it is capable of learning the fiber bundles where these can be explicitly computed, as well as real world data sets where the fiber bundles are of interest.\n", "summary_of_the_review": "I think this is an important addition to the literature and has the possibility of being useful for future work about explainability and disentanglement.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of learning fiber distributions associated with a machine learning task, in which the goal is to predict Y, given X. One chooses a fiber space / distribution Z / D_Z, and learns a trivialization \varphi : (Y,Z) -> X. The proposed architecture first clusters the label space Y. Within each cluster i it fixes a fiber space and distribution, and then learns a mapping \Phi_i, parameterized by an invertible neural network, by minimizing the discrepancy between the generated distribution and the distribution of the training data. The paper performs experiments on the wine dataset and a dataset coming from an aerospace application, as well as synthetic data with fiber bundle structure. Since the task here is generative modeling, the paper compares to standard GAN architectures (WGAN and conditional GAN) and argues that they are not fiber-learners, in the sense of this paper. 

Initial reviews were split, with reviewers appreciating the novelty of the fiber learning task, while also raising questions about the paper’s relationship to conditional GANs, some points of clarity, and limitations of the experiments. After interaction in the response period, the reviewers converged to a decision to accept. The paper’s primary strength is its clear formulation — the paper provides useful language for describing conditional generative models (in particular, for discussing when a factorization of the distribution over Y and Z is appropriate), a valuable contribution to the discussion in this area.
Decision: Accept (Poster)