COMPOSITION AND DECOMPOSITION OF GANS

In this work , we propose a composition/decomposition framework for adversarially training generative models on composed data - data where each sample can be thought of as being constructed from a fixed number of components . In our framework , samples are generated by sampling components from component generators and feeding these components to a composition function which combines them into a “ composed sample ” . This compositional training approach improves the modularity , extensibility and interpretability of Generative Adversarial Networks ( GANs ) - providing a principled way to incrementally construct complex models out of simpler component models , and allowing for explicit “ division of responsibility ” between these components . Using this framework , we define a family of learning tasks and evaluate their feasibility on two datasets in two different data modalities ( image and text ) . Lastly , we derive sufficient conditions such that these compositional generative models are identifiable . Our work provides a principled approach to building on pre-trained generative models or for exploiting the compositional nature of data distributions to train extensible and interpretable models . 1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have proven to be a powerful framework for training generative models that are able to produce realistic samples across a variety of domains , most notably when applied to natural images . However , existing approaches largely attempt to model a data distribution directly and fail to exploit the compositional nature inherent in many data distributions of interest . In this work , we propose a method for training compositional generative models using adversarial training , identify several key benefits of compositional training and derive sufficient conditions under which compositional training is identifiable . This work is motivated by the observation that many data distributions , such as natural images , are compositional in nature - that is , they consist of different components that are combined through some composition process . For example , natural scenes often consist of different objects , composed via some combination of scaling , rotation , occlusion etc . Exploiting this compositional nature of complex data distributions , we demonstrate that one can both incrementally construct models for composed data from component models and learn component models from composed data directly . In our framework , we are interested in modeling composed data distributions - distributions where each sample is constructed from a fixed number of simpler sets of objects . We will refer to these sets of objects as components . For example , consider a simplified class of natural images consisting of a foreground object superimposed on a background , the two components in this case would be a set of foreground objects and a set of backgrounds . We explicitly define two functions : composition and decomposition , as well as a set of component generators . Each component generator is responsible for modeling the marginal distribution of a component while the composition function takes a set of component samples and produce a composed sample ( see figure 1 ) . We additionally assume that the decomposition function is the inverse operation of the composition function . We are motivated by the following desiderata of modeling compositional data : • Modularity : Compositional training should provide a principled way to reuse off-the-shelf or pre-trained component models across different tasks , allowing us to build increasingly complex generative models from simpler ones . • Interpretability : Models should allow us to explicitly incorporate prior knowledge about the compositional structure of data , allowing for clear “ division of responsibility ” between different components . • Extensibility : Once we have learned to decompose data , we should be able to learn component models for previously unseen components directly from composed data . • Identifiability : We should be able to specify sufficient conditions for composition under which composition/decomposition and component models can be learned from composed data . Within this framework , we first consider four learning tasks ( of increasing difficulty ) which range from learning only composition or decomposition ( assuming the component models are pre-trained ) to learning composition , decomposition and all component models jointly . To illustrate these tasks , we show empirical results on two simple datasets : MNIST digits superimposed on a uniform background and the Yelp Open Dataset ( a dataset of Yelp reviews ) . We show examples of when some of these tasks are ill-posed and derive sufficient conditions under which tasks 1 and 3 are identifiable . Lastly , we demonstrate the concept of modularity and extensibility by showing that component generators can be used to inductively learn other new components in a chain-learning example in section 3.4 . The main contributions of this work are : 1 . We define a framework for training compositional generative models adversarially . 2 . Using this framework , we define different tasks corresponding to varying levels of prior knowledge and pre-training . We show results for these tasks on two different datasets from two different data modalities , demonstrating the lack of identifiability for some tasks and feasibility for others . 3 . We derive sufficient conditions under which our compositional models are identifiable . 1.1 RELATED WORK . Our work is related to the task of disentangling representations of data and the discovery of independent factors of variations ( Bengio et al . ( 2013 ) ) . Examples of such work include : 1 ) methods for evaluation of the level of disentaglement ( Eastwood & Williams ( 2018 ) ) , 2 ) new losses that promote disentaglement ( Ridgeway & Mozer ) , 3 ) extensions of architectures that ensure disentanglement ( Kim & Mnih ( 2018 ) ) . Such approaches are complementary to our work but differ in that we explicitly decompose the structure of the generative network into independent building blocks that can be split off and reused through composition . We do not consider decomposition to be a good way to obtain disentangled representations , due to the complete decoupling of the generators . Rather we believe that decomposition of complex generative model into component generators , provides a source of building blocks for model construction . Component generators obtained by our method trained to have disentangled representations could yield interpretable and reusable components , however , we have not explored this avenue of research in this work . Extracting GANs from corrupted measurements has been explored by Bora et al . ( 2018 ) . We note that the noise models described in that paper can be seen as generated by a component generator under our framework . Consequently , our identifiability results generalize recovery results in that paper . Recent work by Azadi et al . ( 2018 ) is focused on image composition and fits neatly in the framework presented here . Along similar lines , work such as Johnson et al . ( 2018 ) , utilizes a monolithic architecture which translates text into objects composed into a scene . In contrast , our work is aimed at deconstructing the monolithic architectures into component generators . 2 METHODS . 2.1 DEFINITION OF FRAMEWORK . Our framework consists of three main moving pieces : Component generators gi ( zi ) A component generator is a standard generative model . In this paper , we adopt the convention that the component generators are functions that maps some noise vector z sampled from standard normal distribution to a component sample . We assume there are m component generators , from g1 to gm . Let oi : = gi ( zi ) be the output for component generator i . Composition function ( c : ( Rn ) m → Rn ) Function which composes m inputs of dimension n to a single output ( composed sample ) . Decomposition function ( d : Rn → ( Rn ) m ) Function which decomposes one input of dimension n to m outputs ( components ) . We denote the i-th output of the decomposition function by d ( · ) i . Without loss of generality we will assume that the composed sample has the same dimensions as each of its components . Together , these pieces define a “ composite generator ” which generates a composed sample by two steps : • Generating component samples o1 , o2 , ... , om . • Composing these component samples using c to form a composed sample . The composition and/or decomposition function are parameterized as neural networks . Below we describe two applications of this framework to the domain of images and text respectively . 2.2 EXAMPLE 1 : IMAGE WITH FOREGROUND OBJECT ( S ) ON A BACKGROUND . In this setting , we assume that each image consists of one or more foreground object over a background . In this case , m ≥ 2 , one component generator is responsible for generating the background , and other component generators generate individual foreground objects . An example is shown in figure 1 . In this case the foreground object is a single MNIST digit and the composition function takes a uniform background and overlays the digit over the background . The decomposition function takes a composed image and returns both the foreground digit and the background with the digit removed . 2.3 EXAMPLE 2 : COHERENT SENTENCE PAIRS . In this setting , we consider the set of adjacent sentence pairs extracted from a larger text . In this case , each component generator generates a sentence and the composition function combines two sentences and edits them to form a coherent pair . The decomposition function splits a pair into individual sentences ( see figure 2 ) . 2.4 LOSS FUNCTION . In this section , we describe details of our training procedure . For convenience of training , we implement a composition of Wasserstein GANs introduced in Arjovsky et al . ( 2017 ) ) but all theoretical results also hold for standard adversarial training losses . Notation We define the data terms used in the loss function . Let x1 be a component sample . There are m such samples . Let y be a composite sample be obtained by composition of components x1 , x2 , ... , xm . For compactness , we use oi as an abbreviation for gi ( zi ) . We denote vector L1 norm by ‖a‖1 ( ‖a‖1 = ∑ i |ai| ) . Finally , we use capital D to denote discriminators involved in different losses . Component Generator Adversarial Loss ( lgi ) Given the component data , we can train component generator ( gi ) to match the component data distribution using loss lgi ≡ Exi∼pdata ( xi ) [ Di ( xi ) ] − Ezi∼pz [ Di ( gi ( zi ) ) ] . Composition Adversarial Loss ( lc ) Given the component generators and composite data , we can train a composition network such that generated composite samples match the composite data distribution using loss lc ≡ Ey∼pdata ( y ) [ Dc ( y ) ] − Ez1∼pz1 , ... , zm∼pzm [ Dc ( c ( o1 , ... , om ) ) ] Decomposition Adversarial Loss ( ld ) Given the component and composite distributions , we can train a decomposition function d such that distribution of decomposed of composite samples matches the component distributions using loss ld ≡ Ez1∼pz1 , ... , zm∼pzm [ Df ( o1 , ... , om ) ] − Ey∼pdata ( y ) [ Df ( d ( y ) ) ] . Composition/Decomposition Cycle Losses ( lc−cyc , ld−cyc ) Additionally , we include a cyclic consistency loss ( Zhu et al . ( 2017 ) ) to encourage composition and decomposition functions to be inverses of each other . lc−cyc ≡ Ez1∼pz1 , ... , zm∼pzm [ ∑ i ‖d ( c ( o1 , ... , om ) ) i − oi‖1 ] ld−cyc ≡ Ey∼pdata ( y ) [ ‖c ( d ( y ) ) − y‖1 ] Table 2 summarizes all the losses . Training of discriminators ( Di , Dc , Df ) is achieved by maximization of their respective losses .

Summary: In this paper, the authors studied the problem of composition and decomposition of GANs. Motivated by the observations that images are naturally composed of multiple layouts, the authors proposed a new framework to study the compositional image generation and its decomposition by defining several tasks. On those various tasks, the authors demonstrate the possibility of the proposed model to composing image components and decompose the images afterwards. These results are interesting and insightful to some extent.