Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps

1 INTRODUCTION . In the last decade , generative methods have made remarkable strides toward the goal of faithfully modeling complex data distributions . Families of model architectures such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , and normalizing flows ( Rezende & Mohamed , 2015 ) have enabled a range of tasks that were previously thought to be unachievable for complex ( and often high-dimensional ) distributions such as those representing natural images . Applications include image to image translation ( Mao et al. , 2019 ; Lee et al. , 2018 ) , music generation ( Mogren , 2016 ) , image super-resolution ( Ledig et al. , 2017 ) , and density estimation ( Dinh et al. , 2016 ) . In this work we introduce a new task in machine learning which revolves around the problem of modeling the structure of many-to-one maps . The majority of classification and regression tasks in machine learning involve many-to-one maps . That is , maps that send a range of distinct inputs to the same output . For example , most image recognition tasks are many-to-one since they send an infinite set of natural images to a finite number of classes . Of course , there is no reason that a model ’ s target space needs to be discrete—consider , for instance , a sentiment analysis task where sentiment is naturally represented by a continuum . There are likely a range of distinct expressions that result in the same sentiment value . This paper was inspired by a problem in materials science wherein one wishes to explore all the processing conditions that will result in a material with a specific fixed property ( e.g. , strength ) . Understanding all such processing conditions allows the materials scientist to choose among a range of options that will all produce a material achieving their desired performance metrics . While the focus of most research on classification and regression tasks is training models that give correct predictions , it can also be useful to be able to fix an output value y and explore , quantify , or sample from all those x which map to y . Such an exercise allows one to understand different modes of variation that a model collapses in the process of making its predictions . If function π : X → Y ∗Work completed during an internship at Pacific Northwest National Laboratory . is the ground truth pairing between input space X and output space Y , this amounts to calculating the inverse image π−1 ( y ) , or fiber of π at y . Our goal in the present work is to ( 1 ) formalize this problem and ( 2 ) describe a deep learning framework which readily enables this kind of analysis . We take as our inspiration the notion of a fiber bundle from topology ( Seifert , 1933 ; Whitney , 1935 ) . Consider the projection map on X = Y × Z , π : Y × Z → Y , which sends π ( y , z ) = y . For any y ∈ Y , the inverse image π−1 ( y ) is easily calculated as { y } × Z ∼= Z . Interpreted in terms of the machine learning task , Y is the component of X that we want to predict and Z encodes the remaining variation occurring among all x that map to y . Unfortunately , in our nonlinear world data distributions can rarely be decomposed into a product space on a global level like this . Instead , it is more realistic to hope that for each sufficiently small neighborhood U of Y , we can find a data distribution preserving homeomorphism U × Z ∼−→ π−1 ( U ) . This informally summarizes the idea of a fiber bundle , where locally the space X is the product of a base space ( some subset of Y ) and a fiber Z . Following the mathematical literature , we call this task local trivialization of X . A local trivialization is useful since it allows us to understand and explore the fibers of π . In our experiments we show that popular generative models including Wasserstein GANs ( Wu et al. , 2018 ) and conditional GANs ( Mirza & Osindero , 2014 ) , are unable to accurately model the fiber structure of a machine learning task . To address this , we describe a new deep learning architecture called a Bundle Network which is designed to produce local trivializations of an input space X and thus enable one to model fibers of a task . Bundle Networks uses a clustering algorithm to divide up the output space Y into neighborhoods { Ui } and then learn a conditioned , invertible mapping that decomposes the inverse image of each of these neighborhoods π−1 ( Ui ) into a product Ui × Z that preserves the distributions induced by the data . We show that Bundle Networks can effectively model fibers for a range of datasets and tasks , from synthetic datasets that represent classic examples of fiber bundles , to the Wine Quality ( Cortez et al. , 2009 ) and Airfoil Noise ( Brooks et al. , 1989 ) datasets from ( Dua & Graff , 2017 ) . In summary , our contributions in this paper include the following . • We identify and formalize the problem of learning the fibers of a machine learning task . • We describe one approach to solving this problem , using a fiber bundle framework to learn how to locally decompose the input space of the task into a product of label space and a fiber space that parametrizes all additional variation in the data . • We introduce a new family of deep generative models called Bundle Networks and show via empirical experiments that , while other popular architectures struggle to model the fibers of a machine learning task , Bundle Networks yield strong results . 2 RELATED WORK . Conditional and manifold-based generative models : The local trivialization task that we describe in Section 3.1 can be interpreted as a special type of conditional generative task . That is , a task where data generation is conditioned on some additional variables . Common models within this framework are the various flavors of conditional variational autoencoders ( VAE ) ( Sohn et al. , 2015 ) and conditional generative adversarial networks ( GAN ) ( Mirza & Osindero , 2014 ) . While our approach utilizes conditional components to capture differences in model behavior at different locations in input space , additional aspects of the task , such as the local decomposition of input space into fiber and label space , mean that standard generative models do not give strong results . This paper also shares similarities to other work at the intersection of generative modeling and manifold theory . In Kalatzis et al . ( 2021 ) for example , the authors propose a new method , multichart flows ( MCFs ) , for learning distributions on topologically nontrivial manifolds . Similar to the present work , the authors use coordinate charts to learn non-Euclidean distributions using otherwise Euclidean machinery . Unlike our work , MCFs are designed to learn general distributions on manifolds and do not have the specialized architectural adaptations or novel training routine designed for modeling many-to-one problems . Invertible neural networks : Invertible neural networks have recently become a topic of interest within the deep learning community . Some of the first network layers that were both invertible and learnable were suggested in ( Dinh et al. , 2015 ; 2016 ) and are in essence affine transformations followed by orthogonal transformations . Ardizzone et al . ( 2018 ) and Sorrenson et al . ( 2020 ) are among the works that have used invertible networks to solve problems of distribution approximation with the latter introducing general incompressible-flow networks ( GINs ) which are volume-preserving versions of invertible neural networks . The most common instance of invertible neural networks are normalizing flows , which were popularized by ( Rezende & Mohamed , 2015 ) . Feature disentanglement : Distribution disentanglement is ( roughly ) the task of learning a latent space for a distribution such that orthogonal directions capture independent and uncorrelated modes of variation in the data . Many recent methods of solving this problem leverage popular generative models such as generative adversarial networks ( GANs ) and variational autoencoders ( VAEs ) ( Chen et al. , 2018 ; Kim & Mnih , 2018 ; Dupont , 2018 ; Chen et al. , 2016 ; Lin et al. , 2020 ) . Like disentanglement , the local trivialization task described in this paper involves factoring a data distribution . Unlike the problem of feature disentanglement , which generally seeks to decompose a distribution into all independent modes of variation , we only seek to distinguish between those modes of variation defined by the classes or targets of the supervised task and all other variation . The local nature of our factorization further sets us apart from work on feature disentanglement . 3 FIBER BUNDLES AND LOCAL TRIVIALIZATIONS . One way to understand complicated geometric spaces is to decompose them into simpler components . For example , topologically a torus T can be understood to be the product of two circles T ∼= S1 × S1 . Such decompositions into product spaces are particularly useful in applications since many algorithms on the entire space can then be reduced to independent , component-wise calculations . However , even in situations where local neighborhoods look like a product , such decompositions may not be possible globally because of “ twists ” that occur in a space ( for example in the Möbius band , see Figure 1 ) . First developed in the work of Seifert ( 1933 ) and Whitney ( 1935 ) , fiber bundles are a geometric structure designed to capture the locally product-like nature of such spaces . Formally , a fiber bundle is a triple ( E , B , Z ) of topological spaces along with a ( surjective ) continuous map π : E → B and a collection of local trivializations T = { ( Ui , φi ) } i∈I subject to the conditions that ( i ) { Ui } i∈I is an open cover of B and ( ii ) for each i ∈ I , φi gives an homeomorphism φi : Ui × Z ≃−→ π−1 ( Ui ) such that the obvious projection from Ui × Z to Ui agrees with the map π restricted from π−1 ( Ui ) to Ui . This last condition is equivalent to saying that for all i ∈ I , the following diagram commutes . π−1 ( Ui ) Ui × Z Ui φi φ−1i proj πi ( 1 ) The space E is generally called the total space , B is called the base space , and Z is called the fiber . One can think of this definition as rigorously capturing the idea that for small enough neighborhoods , E looks like the product of a subset of B and Z . Some standard fiber bundles are found in figure 1 . 3.1 THE LOCAL TRIVIALIZATION OF A MACHINE LEARNING TASK AND MODELING DATA ON . A FIBER Suppose that T is a supervised machine learning task that involves taking data from X and predicting a correspond element of Y . Let DX be the data distribution on X and let measurable function π : supp ( DX ) → Y , define a ground truth label for each x in the support of DX . Then for any measurable open set U ⊆ Y , DX induces a conditional distribution , Dπ−1 ( U ) on π−1 ( U ) . In a similar fashion , via the pushforward of DX using π , we get a label distribution DY on Y . Finally , each measurable subset U ⊆ Y inherits a conditional distribution DU from DY . We call the sets π−1 ( U ) with distribution Dπ−1 ( U ) the trivializable neighborhoods of task T . A local trivialization of X with respect to π , involves : 1 . A choice of open cover U = { Ui } of Y such that ∪iUi = Y . 2 . A choice of fiber space Z with associated fiber distribution DZ . 3 . Learning an invertible , trivializing map φi : Ui×Z → π−1 ( Ui ) for each Ui ∈ U , such that ( a ) φi maps distribution DUi ⊗DZ to Dπ−1 ( Ui ) and ( b ) φ −1 i makes Diagram 1 commute . Note that in the above formulation , the space X takes the role of the total space E from section 3 and Y takes the role of the base space B . Though good choices of fiber Z and measure ν can occasionally be guessed based on the nature of the data , they generally must be arrived at through experimentation . Further , the use of local trivializations should be seen as a convenient framework that enables the input space to be locally approximated as the product of two subspaces : one corresponding to the label and one capturing variation unrelated to the label . One should not expect that a task and its fibers are literally a fiber bundle “ on the nose. ” The trivializing maps { φi } i allow one to explore the fibers of π , which includes sampling from them . For example , if we choose some y ∈ Y then we can sample from the fiber π−1 ( y ) by ( 1 ) identifying the neighborhood Ui that y belongs to , ( 2 ) sampling a collection of points { z1 , . . . , zℓ } using DZ , and ( 3 ) applying the map φi to ( y , zi ) for each z ∈ { z1 , . . . , zℓ } .

Summary: This paper proposes a neural network to model data (or machine learning tasks) with fiber bundle structure. Machine learning tasks like regression or classification are often many to one. They assign a "label" (discrete or continuous) $Y \ni y = \pi(x)$ to each $x \in X$. Many different $x$ may receive the same label. It is interesting to explore / parameterize the set of all $x \in X$ that yield a given label $y$, that is to say, the set $\pi^{-1}(y)$. This is nicely modeled by the topological construction of a fiber bundle (a space that is locally a product space). Operationally, it corresponds to a covering of the label space $Y = \bigcup_{i} U_i$ by "patches" $U_i$ so that $\pi^{-1}(U_i)$  are homeomorphic to product spaces $U_i \times Z$, with $Z$ being the fiber space.   The proposed network is in essence a conditional generative model with the added twist that the conditioning is also performed on the "patch" that a label (or a regression target) belongs to (for training; inference is only performed in the "reverse" direction of fiber sampling.)  The authors show interesting results in modeling familiar low-dimensional bundles as well as real-world datasets. Comparisons with conditional generative models suggest that the new architecture may be a better fiber bundle model. Finally, the authors numerically explore the role of "latent space" topology, a welcome addition.