Bundle Networks: Fiber Bundles, Local Trivializations, and a Generative Approach to Exploring Many-to-one Maps

1 INTRODUCTION . In the last decade , generative methods have made remarkable strides toward the goal of faithfully modeling complex data distributions . Families of model architectures such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , and normalizing flows ( Rezende & Mohamed , 2015 ) have enabled a range of tasks that were previously thought to be unachievable for complex ( and often high-dimensional ) distributions such as those representing natural images . Applications include image to image translation ( Mao et al. , 2019 ; Lee et al. , 2018 ) , music generation ( Mogren , 2016 ) , image super-resolution ( Ledig et al. , 2017 ) , and density estimation ( Dinh et al. , 2016 ) . In this work we introduce a new task in machine learning which revolves around the problem of modeling the structure of many-to-one maps . The majority of classification and regression tasks in machine learning involve many-to-one maps . That is , maps that send a range of distinct inputs to the same output . For example , most image recognition tasks are many-to-one since they send an infinite set of natural images to a finite number of classes . Of course , there is no reason that a model ’ s target space needs to be discrete—consider , for instance , a sentiment analysis task where sentiment is naturally represented by a continuum . There are likely a range of distinct expressions that result in the same sentiment value . This paper was inspired by a problem in materials science wherein one wishes to explore all the processing conditions that will result in a material with a specific fixed property ( e.g. , strength ) . Understanding all such processing conditions allows the materials scientist to choose among a range of options that will all produce a material achieving their desired performance metrics . While the focus of most research on classification and regression tasks is training models that give correct predictions , it can also be useful to be able to fix an output value y and explore , quantify , or sample from all those x which map to y . Such an exercise allows one to understand different modes of variation that a model collapses in the process of making its predictions . If function π : X → Y ∗Work completed during an internship at Pacific Northwest National Laboratory . is the ground truth pairing between input space X and output space Y , this amounts to calculating the inverse image π−1 ( y ) , or fiber of π at y . Our goal in the present work is to ( 1 ) formalize this problem and ( 2 ) describe a deep learning framework which readily enables this kind of analysis . We take as our inspiration the notion of a fiber bundle from topology ( Seifert , 1933 ; Whitney , 1935 ) . Consider the projection map on X = Y × Z , π : Y × Z → Y , which sends π ( y , z ) = y . For any y ∈ Y , the inverse image π−1 ( y ) is easily calculated as { y } × Z ∼= Z . Interpreted in terms of the machine learning task , Y is the component of X that we want to predict and Z encodes the remaining variation occurring among all x that map to y . Unfortunately , in our nonlinear world data distributions can rarely be decomposed into a product space on a global level like this . Instead , it is more realistic to hope that for each sufficiently small neighborhood U of Y , we can find a data distribution preserving homeomorphism U × Z ∼−→ π−1 ( U ) . This informally summarizes the idea of a fiber bundle , where locally the space X is the product of a base space ( some subset of Y ) and a fiber Z . Following the mathematical literature , we call this task local trivialization of X . A local trivialization is useful since it allows us to understand and explore the fibers of π . In our experiments we show that popular generative models including Wasserstein GANs ( Wu et al. , 2018 ) and conditional GANs ( Mirza & Osindero , 2014 ) , are unable to accurately model the fiber structure of a machine learning task . To address this , we describe a new deep learning architecture called a Bundle Network which is designed to produce local trivializations of an input space X and thus enable one to model fibers of a task . Bundle Networks uses a clustering algorithm to divide up the output space Y into neighborhoods { Ui } and then learn a conditioned , invertible mapping that decomposes the inverse image of each of these neighborhoods π−1 ( Ui ) into a product Ui × Z that preserves the distributions induced by the data . We show that Bundle Networks can effectively model fibers for a range of datasets and tasks , from synthetic datasets that represent classic examples of fiber bundles , to the Wine Quality ( Cortez et al. , 2009 ) and Airfoil Noise ( Brooks et al. , 1989 ) datasets from ( Dua & Graff , 2017 ) . In summary , our contributions in this paper include the following . • We identify and formalize the problem of learning the fibers of a machine learning task . • We describe one approach to solving this problem , using a fiber bundle framework to learn how to locally decompose the input space of the task into a product of label space and a fiber space that parametrizes all additional variation in the data . • We introduce a new family of deep generative models called Bundle Networks and show via empirical experiments that , while other popular architectures struggle to model the fibers of a machine learning task , Bundle Networks yield strong results . 2 RELATED WORK . Conditional and manifold-based generative models : The local trivialization task that we describe in Section 3.1 can be interpreted as a special type of conditional generative task . That is , a task where data generation is conditioned on some additional variables . Common models within this framework are the various flavors of conditional variational autoencoders ( VAE ) ( Sohn et al. , 2015 ) and conditional generative adversarial networks ( GAN ) ( Mirza & Osindero , 2014 ) . While our approach utilizes conditional components to capture differences in model behavior at different locations in input space , additional aspects of the task , such as the local decomposition of input space into fiber and label space , mean that standard generative models do not give strong results . This paper also shares similarities to other work at the intersection of generative modeling and manifold theory . In Kalatzis et al . ( 2021 ) for example , the authors propose a new method , multichart flows ( MCFs ) , for learning distributions on topologically nontrivial manifolds . Similar to the present work , the authors use coordinate charts to learn non-Euclidean distributions using otherwise Euclidean machinery . Unlike our work , MCFs are designed to learn general distributions on manifolds and do not have the specialized architectural adaptations or novel training routine designed for modeling many-to-one problems . Invertible neural networks : Invertible neural networks have recently become a topic of interest within the deep learning community . Some of the first network layers that were both invertible and learnable were suggested in ( Dinh et al. , 2015 ; 2016 ) and are in essence affine transformations followed by orthogonal transformations . Ardizzone et al . ( 2018 ) and Sorrenson et al . ( 2020 ) are among the works that have used invertible networks to solve problems of distribution approximation with the latter introducing general incompressible-flow networks ( GINs ) which are volume-preserving versions of invertible neural networks . The most common instance of invertible neural networks are normalizing flows , which were popularized by ( Rezende & Mohamed , 2015 ) . Feature disentanglement : Distribution disentanglement is ( roughly ) the task of learning a latent space for a distribution such that orthogonal directions capture independent and uncorrelated modes of variation in the data . Many recent methods of solving this problem leverage popular generative models such as generative adversarial networks ( GANs ) and variational autoencoders ( VAEs ) ( Chen et al. , 2018 ; Kim & Mnih , 2018 ; Dupont , 2018 ; Chen et al. , 2016 ; Lin et al. , 2020 ) . Like disentanglement , the local trivialization task described in this paper involves factoring a data distribution . Unlike the problem of feature disentanglement , which generally seeks to decompose a distribution into all independent modes of variation , we only seek to distinguish between those modes of variation defined by the classes or targets of the supervised task and all other variation . The local nature of our factorization further sets us apart from work on feature disentanglement . 3 FIBER BUNDLES AND LOCAL TRIVIALIZATIONS . One way to understand complicated geometric spaces is to decompose them into simpler components . For example , topologically a torus T can be understood to be the product of two circles T ∼= S1 × S1 . Such decompositions into product spaces are particularly useful in applications since many algorithms on the entire space can then be reduced to independent , component-wise calculations . However , even in situations where local neighborhoods look like a product , such decompositions may not be possible globally because of “ twists ” that occur in a space ( for example in the Möbius band , see Figure 1 ) . First developed in the work of Seifert ( 1933 ) and Whitney ( 1935 ) , fiber bundles are a geometric structure designed to capture the locally product-like nature of such spaces . Formally , a fiber bundle is a triple ( E , B , Z ) of topological spaces along with a ( surjective ) continuous map π : E → B and a collection of local trivializations T = { ( Ui , φi ) } i∈I subject to the conditions that ( i ) { Ui } i∈I is an open cover of B and ( ii ) for each i ∈ I , φi gives an homeomorphism φi : Ui × Z ≃−→ π−1 ( Ui ) such that the obvious projection from Ui × Z to Ui agrees with the map π restricted from π−1 ( Ui ) to Ui . This last condition is equivalent to saying that for all i ∈ I , the following diagram commutes . π−1 ( Ui ) Ui × Z Ui φi φ−1i proj πi ( 1 ) The space E is generally called the total space , B is called the base space , and Z is called the fiber . One can think of this definition as rigorously capturing the idea that for small enough neighborhoods , E looks like the product of a subset of B and Z . Some standard fiber bundles are found in figure 1 . 3.1 THE LOCAL TRIVIALIZATION OF A MACHINE LEARNING TASK AND MODELING DATA ON . A FIBER Suppose that T is a supervised machine learning task that involves taking data from X and predicting a correspond element of Y . Let DX be the data distribution on X and let measurable function π : supp ( DX ) → Y , define a ground truth label for each x in the support of DX . Then for any measurable open set U ⊆ Y , DX induces a conditional distribution , Dπ−1 ( U ) on π−1 ( U ) . In a similar fashion , via the pushforward of DX using π , we get a label distribution DY on Y . Finally , each measurable subset U ⊆ Y inherits a conditional distribution DU from DY . We call the sets π−1 ( U ) with distribution Dπ−1 ( U ) the trivializable neighborhoods of task T . A local trivialization of X with respect to π , involves : 1 . A choice of open cover U = { Ui } of Y such that ∪iUi = Y . 2 . A choice of fiber space Z with associated fiber distribution DZ . 3 . Learning an invertible , trivializing map φi : Ui×Z → π−1 ( Ui ) for each Ui ∈ U , such that ( a ) φi maps distribution DUi ⊗DZ to Dπ−1 ( Ui ) and ( b ) φ −1 i makes Diagram 1 commute . Note that in the above formulation , the space X takes the role of the total space E from section 3 and Y takes the role of the base space B . Though good choices of fiber Z and measure ν can occasionally be guessed based on the nature of the data , they generally must be arrived at through experimentation . Further , the use of local trivializations should be seen as a convenient framework that enables the input space to be locally approximated as the product of two subspaces : one corresponding to the label and one capturing variation unrelated to the label . One should not expect that a task and its fibers are literally a fiber bundle “ on the nose. ” The trivializing maps { φi } i allow one to explore the fibers of π , which includes sampling from them . For example , if we choose some y ∈ Y then we can sample from the fiber π−1 ( y ) by ( 1 ) identifying the neighborhood Ui that y belongs to , ( 2 ) sampling a collection of points { z1 , . . . , zℓ } using DZ , and ( 3 ) applying the map φi to ( y , zi ) for each z ∈ { z1 , . . . , zℓ } .

Summary: The paper introduces a new architecture for generative modeling, called Bundle Networks. This is based on the mathematical concept of fiber bundles and allows to explore the networks fibers above the labels (their preimages). Training the network then corresponds to learning the local trivialization of the fiber bundle, i.e., of the labeling function.