Title: COMPOSITION AND DECOMPOSITION OF GANS

Abstract: In this work, we propose a composition/decomposition framework for adversarially training generative models on composed data -data where each sample can be thought of as being constructed from a fixed number of components. In our framework, samples are generated by sampling components from component generators and feeding these components to a composition function which combines them into a "composed sample". This compositional training approach improves the modularity, extensibility and interpretability of Generative Adversarial Networks (GANs) -providing a principled way to incrementally construct complex models out of simpler component models, and allowing for explicit "division of responsibility" between these components. Using this framework, we define a family of learning tasks and evaluate their feasibility on two datasets in two different data modalities (image and text). Lastly, we derive sufficient conditions such that these compositional generative models are identifiable. Our work provides a principled approach to building on pre-trained generative models or for exploiting the compositional nature of data distributions to train extensible and interpretable models.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) have proven to be a powerful framework for training generative models that are able to produce realistic samples across a variety of domains, most notably when applied to natural images. However, existing approaches largely attempt to model a data distribution directly and fail to exploit the compositional nature inherent in many data distributions of interest. In this work, we propose a method for training compositional generative models using adversarial training, identify several key benefits of compositional training and derive sufficient conditions under which compositional training is identifiable. This work is motivated by the observation that many data distributions, such as natural images, are compositional in nature -that is, they consist of different components that are combined through some composition process. For example, natural scenes often consist of different objects, composed via some combination of scaling, rotation, occlusion etc. Exploiting this compositional nature of complex data distributions, we demonstrate that one can both incrementally construct models for composed data from component models and learn component models from composed data directly.

In our framework, we are interested in modeling composed data distributions -distributions where each sample is constructed from a fixed number of simpler sets of objects. We will refer to these sets of objects as components. For example, consider a simplified class of natural images consisting of a foreground object superimposed on a background, the two components in this case would be a set of foreground objects and a set of backgrounds. We explicitly define two functions: composition and decomposition, as well as a set of component generators. Each component generator is responsible for modeling the marginal distribution of a component while the composition function takes a set of component samples and produce a composed sample (see figure 1 ). We additionally assume that the decomposition function is the inverse operation of the composition function.

We are motivated by the following desiderata of modeling compositional data:

• Modularity: Compositional training should provide a principled way to reuse off-the-shelf or pre-trained component models across different tasks, allowing us to build increasingly complex generative models from simpler ones.

• Interpretability: Models should allow us to explicitly incorporate prior knowledge about the compositional structure of data, allowing for clear "division of responsibility" between different components.

• Extensibility: Once we have learned to decompose data, we should be able to learn component models for previously unseen components directly from composed data.

• Identifiability: We should be able to specify sufficient conditions for composition under which composition/decomposition and component models can be learned from composed data.

Within this framework, we first consider four learning tasks (of increasing difficulty) which range from learning only composition or decomposition (assuming the component models are pre-trained) to learning composition, decomposition and all component models jointly.

To illustrate these tasks, we show empirical results on two simple datasets: MNIST digits superimposed on a uniform background and the Yelp Open Dataset (a dataset of Yelp reviews). We show examples of when some of these tasks are ill-posed and derive sufficient conditions under which tasks 1 and 3 are identifiable. Lastly, we demonstrate the concept of modularity and extensibility by showing that component generators can be used to inductively learn other new components in a chain-learning example in section 3.4.

The main contributions of this work are:

1. We define a framework for training compositional generative models adversarially.

2. Using this framework, we define different tasks corresponding to varying levels of prior knowledge and pre-training. We show results for these tasks on two different datasets from two different data modalities, demonstrating the lack of identifiability for some tasks and feasibility for others.

3. We derive sufficient conditions under which our compositional models are identifiable.

1.1. RELATED WORK:

Our work is related to the task of disentangling representations of data and the discovery of independent factors of variations (Bengio et al. (2013) ). Examples of such work include: 1) methods for evaluation of the level of disentaglement (Eastwood & Williams (2018) ), 2) new losses that promote disentaglement (Ridgeway & Mozer) , 3) extensions of architectures that ensure disentanglement (Kim & Mnih (2018) ). Such approaches are complementary to our work but differ in that we explicitly decompose the structure of the generative network into independent building blocks that can be split off and reused through composition. We do not consider decomposition to be a good way to obtain disentangled representations, due to the complete decoupling of the generators. Rather we believe that decomposition of complex generative model into component generators, provides a source of building blocks for model construction. Component generators obtained by our method trained to have disentangled representations could yield interpretable and reusable components, however, we have not explored this avenue of research in this work.

Extracting GANs from corrupted measurements has been explored by Bora et al. (2018) . We note that the noise models described in that paper can be seen as generated by a component generator under our framework. Consequently, our identifiability results generalize recovery results in that paper. Recent work by Azadi et al. (2018) is focused on image composition and fits neatly in the framework presented here. Along similar lines, work such as Johnson et al. (2018) , utilizes a monolithic architecture which translates text into objects composed into a scene. In contrast, our work is aimed at deconstructing the monolithic architectures into component generators.

2. METHODS:



2.1. DEFINITION OF FRAMEWORK:

Our framework consists of three main moving pieces:

Method Learn components:



Learn composition:

Learn decomposition Generative model LR-GAN (Yang et al., 2017) Background True False True C-GAN (Azadi et al., 2018) False True True False ST-GAN (Zhang et al., 2017) False True False False InfoGAN (Chen et al., 2016) False False False True

Table 1 : Various GAN methods can learn some, but not all, parts of our framework. These parts may exist implicitly in each of the models, but their extraction is non-trivial.

Component generators g i (z i ) A component generator is a standard generative model. In this paper, we adopt the convention that the component generators are functions that maps some noise vector z sampled from standard normal distribution to a component sample. We assume there are m component generators, from g 1 to g m . Let o i := g i (z i ) be the output for component generator i.

Composition function (c : (R n ) m → R n )

Function which composes m inputs of dimension n to a single output (composed sample).

Decomposition function (d : R n → (R n ) m ) Function which decomposes one input of dimension n to m outputs (components). We denote the i-th output of the decomposition function by d(•) i .

Without loss of generality we will assume that the composed sample has the same dimensions as each of its components.

Together, these pieces define a "composite generator" which generates a composed sample by two steps:

• Generating component samples o 1 , o 2 , ..., o m .

• Composing these component samples using c to form a composed sample.

The composition and/or decomposition function are parameterized as neural networks.

Below we describe two applications of this framework to the domain of images and text respectively.

2.2. EXAMPLE 1: IMAGE WITH FOREGROUND OBJECT(S) ON A BACKGROUND:

In this setting, we assume that each image consists of one or more foreground object over a background. In this case, m ≥ 2, one component generator is responsible for generating the background, and other component generators generate individual foreground objects.

An example is shown in figure 1 . In this case the foreground object is a single MNIST digit and the composition function takes a uniform background and overlays the digit over the background.

The decomposition function takes a composed image and returns both the foreground digit and the background with the digit removed.

Components Composed Example:

Composition Decomposition Figure 1 : An example of composition and decomposition for example 1.

2.3. EXAMPLE 2: COHERENT SENTENCE PAIRS:

In this setting, we consider the set of adjacent sentence pairs extracted from a larger text. In this case, each component generator generates a sentence and the composition function combines two sentences and edits them to form a coherent pair. The decomposition function splits a pair into individual sentences (see figure 2 ).

Nice restaurant.

Dry corn bread.

Terrible restaurant.

Dry corn bread.

Terrible restaurant.

Dry corn bread. 

Composition Decomposition:



Composed Example Components:



2.4. LOSS FUNCTION:

In this section, we describe details of our training procedure. For convenience of training, we implement a composition of Wasserstein GANs introduced in Arjovsky et al. ( 2017) ) but all theoretical results also hold for standard adversarial training losses.

Notation We define the data terms used in the loss function. Let x 1 be a component sample. There are m such samples. Let y be a composite sample be obtained by composition of components x 1 , x 2 , ..., x m . For compactness, we use o i as an abbreviation for g i (z i ). We denote vector L 1 norm by a 1 ( a 1 = i |a i | ). Finally, we use capital D to denote discriminators involved in different losses.

Component Generator Adversarial Loss (l g i ) Given the component data, we can train component generator (g i ) to match the component data distribution using loss

l g i ≡ E xi∼p data (xi) [D i (x i )] -E zi∼pz [D i (g i (z i ))].

Composition Adversarial Loss (l c ) Given the component generators and composite data, we can train a composition network such that generated composite samples match the composite data distribution using loss

l c ≡ E y∼p data (y) [D c (y)] -E z1∼pz 1 ,...,zm∼pz m [D c (c(o 1 , ..., o m ))]

Decomposition Adversarial Loss (l d ) Given the component and composite distributions, we can train a decomposition function d such that distribution of decomposed of composite samples matches the component distributions using loss

l d ≡ E z1∼pz 1 ,...,zm∼pz m [D f (o 1 , ..., o m )] -E y∼p data (y) [D f (d(y))].

Composition/Decomposition Cycle Losses (l c-cyc , l d-cyc ) Additionally, we include a cyclic consistency loss (Zhu et al. (2017) ) to encourage composition and decomposition functions to be inverses of each other.

l c-cyc ≡ E z1∼pz 1 ,...,zm∼pz m i d(c(o 1 , ..., o m )) i -o i 1 l d-cyc ≡ E y∼p data (y) [ c(d(y)) -y 1 ]

Table 2 summarizes all the losses. Training of discriminators (D i , D c , D f ) is achieved by maximization of their respective losses.

2.5. PROTOTYPICAL TASKS AND CORRESPONDING LOSSES:

Under the composition/decomposition framework, we focus on a set of prototypical tasks which involve composite data. 

l g i E xi∼p data (xi) [D i (x i )] -E zi∼pz [D i (g i (z i ))] l c E y∼p data (y) [D c (y)] -E z1∼pz 1 ,...,zm∼pz m [D c (c(o 1 , ..., o m ))] l d E z1∼pz 1 ,...,zm∼pz m [D f (o 1 , ..., o m )] -E y∼p data (y) [D f (d(y))] l c-cyc E z1∼pz 1 ,...,zm∼pz m [ i d(c(o 1 , ..., o m )) i -o i 1 ] l d-cyc E y∼p data (y) [ c(d(y)) -y 1 ]

Task 1: Given component generators g i , i ∈ {1, . . . , m} and c, train d.

Task 2: Given component generators g i , i ∈ {1, . . . , m}, train d and c.

Task 3: Given component generators g i , i ∈ {1, . . . , m -1} and c, train g m and d.

Task 4: Given c, train all g i , i ∈ {1, . . . , m} and d

To train generator(s) in these tasks, we minimize relevant losses:

l c + l d + α(l c-cyc + l d-cyc ),

where α ≥ 0 controls the importance of consistency. While the loss function is the same for the tasks, the parameters to be optimized are different. In each task, only the parameters of the trained networks are optimized.

To train discriminator(s), a regularization is applied. For brevity, we do not show the regularization term (see Petzka et al. ( 2017)) used in our experiments.

The tasks listed above increase in difficulty. We will show the capacity of our framework as we progress through the tasks.

Theoretical results in Section 4 provide sufficient conditions under which Task 1. and Task 3. are tractable.

3. EXPERIMENTS:



3.1. DATASETS:

We conduct experiments on three datasets:

1. MNIST-MB MNIST digits LeCun & Cortes (2010) are superimposed on a monochromic one-color-channel background (value ranged from 0-200) (figure 3 ). The image size is 28 x 28.

2. MNIST-BB MNIST digit are rotated and scaled to fit a box of size 32 x 32 placed on a monochrome background of size 64 x 64. The box is positioned in one of the four possible locations (top-right, top-left, bottom-right, bottom-left), with rotation between (-π/6, π/6) (figure 4 ).

3.. Yelp-reveiws:

We derive data from Yelp Open Dataset Yelp Inc. (2004) (2005) (2006) (2007) (2008) (2009) (2010) (2011) (2012) (2013) (2014) (2015) (2016) (2017) (2018) . From each review, we take the first two sentences of the review. We filtered out reviews containing sentences shorter than 5 and longer than 10 words. By design, the sentence pairs have the same topic and sentiment. We refer to this quality as coherence. Incoherent sentences have either different topic or different sentiment.

3.2. NETWORK ARCHITECTURES:

MNIST-MB, MNIST-BB The component generators are DCGAN (Radford et al. (2015) ) models.

Decomposition is implemented as a U-net (Ronneberger et al. (2015) ) model. The inputs to the composition network are concatenated channel-wise. Similarly, when doing decomposition, the (2015) . Input to composition network is a concatenation of two sentences separated by a delimiter token. Following the setting of Seq- GAN Yu et al. (2017) , the discriminator (D c ) network is a convolutional network for sequence data.

3.3. EXPERIMENTS ON MNIST-MB:

Throughout this section we assume that composition operation is known and given by

c(o 1 , o 2 ) i = o 1,i if o 2,i = 0 o 2,i otherwise.

In tasks where one or more generators are given, the generators have been independently trained using corresponding adversarial loss l g i .

Task 1: Given g i , i ∈ {1, 2} and c, train d. This is the simplest task in the framework. The decomposition network learns to decompose the digits and backgrounds correctly (figure 5 ) given c and pre-trained generative models for both digits and background components.

Task 2: Given g 1 and g 2 train d and c. Here we learn composition and decomposition jointly 6.

We find that the model learns to decompose digits accurately; interestingly however, we note that backgrounds from decomposition network are inverted in intensity Task 3: Given g 1 and c, train g 2 and d. Given digit generator and composition network, we train decomposition network and background generator (figure 7 ). We see that decomposition network learns to generate nearly uniform backgrounds, and the decomposition network learns to inpaint. FID evaluation In Table 3 we illustrate performance of learned generators trained using the setting of Task 3, compared to baseline monolithic models which are not amenable to decomposition. As a complement to digits we also show results on Fashion-MNIST overlaid on uniform backgrounds (Xiao et al., 2017) 

(see appendix for examples of task 3 and chain learning on Fashion-MNIST):

Task 4: Given c, train g 1 , g 2 and d. Given just composition, learn components and decomposition. We show that for a simple composition function, there are many ways to assign responsibilities to different components. Some are trivial, for example the whole composite image is generated by a single component (see figure 11 in Appendix).

3.4. CHAIN LEARNING -EXPERIMENTS ON MNIST-BB:

In task 3 above, we demonstrated on the MNIST-MB dataset that we can learn to model the background component and the decomposition function from composed data assuming we are given a model for the foreground component and a composition network. This suggests the natural followup question: if we have a new dataset consisting of a previously unseen class of foreground objects on the same distribution of backgrounds, can we then use this background model we've learned to learn a new foreground model?

We call this concept "chain learning", since training proceeds sequentially and relies on the model trained in the previous stage. To make this concrete, consider this proof-of-concept chain (using the MNIST-BB dataset): As shown in figure 8 we are able to learn both the background generator (in step 1) and the foreground generator for "2" (in step 2) correctly. More generally, the ability to learn a component model from composed data (given models for all other components) allows one to incrementally learn new component models directly from composed data. (Heusel et al., 2017) for generators trained using different datasets and methods. The "Foreground" column and "Foreground+background" reflect performance of trained generators on generating corresponding images. WGAN-GP is trained on foreground and composed images. Generators evaluaged in the "By decomposition" row are obtained as described in Task 3 -on composed images, given background generator and composition operator. The information processing inequality guarantees that the resulting generator cannot beat the WGAN-GP on clean foreground data. However, the composed images are better modeled using the learned foreground generator and known composition and background generator.

Example of coherent sentence composition Inputs the spa was amazing ! the owner cut my hair Baseline the spa was amazing ! the owner cut my hair . l c the spa was clean and professional and professional . our server was friendly and helpful . l d-cyc + l c the spa itself is very beautiful . the owner is very knowledgeable and patient .

Failure modes for l d-cyc + l c Inputs green curry level 10 was perfect . the owner responded right away to our yelp inquiry . l d-cyc + l c the food is amazing ! the owner is very friendly and helpful . Inputs best tacos in las vegas ! everyone enjoyed their meals . l d-cyc + l c the best buffet in las vegas . everyone really enjoyed the food and service are amazing .

Figure 9 : Composition network can be trained to edit for coherence. Only the model trained using loss l d-cyc + l c achieves coherence in this example. For this model, we show illustrative failure modes. In the first example, the network removes specificity of the sentences to make them coherent. In the second case, the topic is changed to a common case and the second sentence is embellished.

3.5. EXPERIMENTS ON YELP DATA:

For this dataset, we focus on a variant of task 1: given d and g 1 , g 2 , train c. In this task, the decomposition function is simple -it splits concatenated sentences without modification. Since we are not learning decomposition, l c-cyc is not applicable in this task. In contrast to MNIST task, where composition is simple and decomposition non-trivial, in this setting, the situation is reversed. Other parts of the optimization function are the same as section 2.4.

We follow the state-of-the-art approaches in training generative models for sequence data. We briefly outline relevant aspects of the training regime.

As in Seq-GAN, we also pre-train the composition networks. The data for pre-training consist of two pairs of sentences. The output pair is a coherent pair from a single Yelp review. Each of the input sentences is independently sampled from a set of nearest neighbors of the corresponding output sentences. Following Guu et al. (2017) we use Jaccard distance to find nearest neighbor sentences. As we sample a pair independently, the input sentences are not generally coherent but the coherence can be achieved with a small number of changes.

Discrimination in Seq-GAN is performed on an embedding of a sentence. For the purposes of training an embedding, we initialize with GloVe word embedding Pennington et al. (2014) . During adversarial training, we follow regime of Xu et al. (2017) by freezing parameters of the encoder of the composition networks, the word projection layer (from hidden state to word distribution), and the word embedding matrix, and only update the decoder parameters.

To enable the gradient to back-propagate from the discriminator to the generator, we applied the Gumbel-softmax straight-through estimator from Jang et al. (2016) . We exponentially decay the temperature with each iteration. Figure 9 shows an example of coherent composition and two failure modes for the trained composition network.

4. IDENTIFIABILITY RESULTS:

In the experimental section, we highlighted tasks which suffer from identifiability problems. Here we state sufficient conditions for identifiability of different parts of our framework. Due to space constraints, we refer the reader to the appendix for the relevant proofs. For simplicity, we consider the output of a generator network as a random variable and do away with explicit reference to generators. Specifically, we use random variables X and Y to refer to component random variables, and Z to a composite random variable. Let range(•) denote range of a random variable. We define indicator function, 1[a] is 1 if a is true and 0 otherwise. In the following results, we use assumptions: Assumption 1. X, Y are finite discrete random variables. Assumption 2. For variables X and Y , and composition function c, let random variable Z be distributed according to

EQUATION

Theorem 1. Let Assumptions 1 and 2 hold. Further, assume that resolving matrix of X and c has full column-rank. If optimum of inf

EQUATION

is achieved for some random variable Y , then Y and Y have the same distribution. Theorem 2. Let Assumptions 1 and 2 hold. Further, assume that c is bijective. If optimum of

inf d E X,Y [ d (c (X, Y )) x -X 1 ] + E Z d (c (X, Y )) y -Y 1 + E Z [ c(d(Z)) -Z 1 ] (3)

is 0 and it is achieved for some d then d is equal to inverse of c.

5. CONCLUSION:

We introduce a framework of generative adversarial network composition and decomposition. In this framework, GANs can be taken apart to extract component GANs and composed together to construct new composite data models. This paradigm allowed us to separate concerns about training different component generators and even incrementally learn new object classes -a concept we deemed chain learning. However, composition and decomposition are not always uniquely defined and hence may not be identifiable from the data. In our experiments we discover settings in which component generators may not be identifiable. We provide theoretical results on sufficient conditions for identifiability of GANs. We hope that this work spurs interest in both practical and theoretical work in GAN decomposability.

or equivalently, ∀z ∈ range(Z): sup

EQUATION

is achieved for some random variable Y , then Y and Y have the same distribution.

Proof. Let Z be distributed according to

EQUATION

The objective in equation 6 can be rewritten as

EQUATION

where dependence of Z on Y is implicit.

Following Arjovsky et al. (2017) , we note that W (Z, Z ) → 0 implies that p(Z) D -→ p(Z ), hence the infimum in equation 8 is achieved for Z distributed as Z . Finally, we observe that Z and Z are identically distributed if Y and Y are. Hence, distribution of Y if optimal for equation 6.

Next we show that there is a unique of distribution of Y for which Z and Z are identically distributed, by generalizing a proof by Bora et al. (2018) For a random variable X we adopt notation p x denote a vector of probabilities p x,i = p(X = i). In this notation, equation 1 can be rewritten as 

EQUATION

) Since R is of rank |range(Y )| then R T R is of size |range(Y )| × |range(Y )| and non-singular. Con- sequently, (R T R) -1 R T p z is

d E X,Y [ d (c (X, Y )) x -X 1 ] + E Z d (c (X, Y )) y -Y 1 + E Z [ c(d(Z)) -Z 1 ] (10)

is 0 and it is achieved for some d then d is equal to inverse of c.

Proof. We note that for a given distribution, expectation of a non-negative function -such as norm -can only be zero if the function is zero on the whole support of the distribution.

Assume that optimum of 0 is achieved but d is not equal to inverse of c, denoted as d * . Hence, there exists a z such ( We use U-Net (Ronneberger et al., 2015) architecture for MNIST-BB for decomposition and composition networks. The input into U-Net is of size 28x28 (28x28x2) the outputs are of size 28x28x2 (28x28) for decomposition (composition). In these networks filters are of size 5x5 in the deconvolution and convolution layers. The convolution layers are of size 32, 64, 128, 256 and deconvolution layers are of size 256, 128, 64, 32. We use leaky rectifier linear units with alpha of 0.2. We use sigmoidal units in the final output layer.

For MNIST-MB and Fashion-MNIST composition networks, we used 2 layer convolutional neural net with 3x3 filter. For decomposition network on these datasets, we used fully-convolutional network (Long et al., 2015) . In this network filters are of size 5x5 in the deconvolution and convolution layers. The convolution layers are of size 32, 64, 128 and deconvolution layers are 128, 64, 32. We use leaky rectifier linear units with alpha of 0.2. We use sigmoidal units in the output layer.

The standard generator and discriminator architecture of DCGAN framework was used for images of 28x28 on MNIST-MB and Fashion-MNIST, and 64x64 on MNIST-MB dataset.

7.2. ARCHITECTURE FOR YELP-REVIEWS:

We first tokenize the text using the nltk Python package. We keep the 30,000 most frequently occuring tokens and represent the remainder as "unknown". We encode each token into a 300 dimensional word vector using the standard GloVe (Pennington et al., 2014) embedding model.

We use a standard sequence-to-sequence model for composition. The composition network takes as input a pair of concatenated sentences and outputs a modified pair of sentences. We used a encoderdecoder network where the encoder/decoder is a 1-layer gated recurrent unit (GRU) network with a hidden state of size 512. In addition, we implemented an attention mechanism as proposed in Luong et al. (2015) in the decoder network.

We adopt the discriminator structure as described in SeqGAN (Yu et al., 2017) . We briefly describe the structure at a high level here, please refer to the SeqGAN paper for additional details. SeqGAN takes as input the pre-processed sequence of word embeddings. The discriminator takes the embedded sequence and feeds it through a set of convolution layers of size (200, 400, 400, 400, 400, 200, 200, 200, 200, 200, 320, 320) and of filter size (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20) . These filters further go through a max pooling layer with an additional "highway network structure" on top of the feature maps to improve performance. Finally the features are fed into a fully-connected layer and produce a real value.

7.3. TRAINING:

We kept the training procedure consistent across all experiments. During training, we initialized weights as described in He et al. (2015) , weights were updated using ADAM (Kingma & Ba, 2014) (with beta1=0.5, and beta2=0.9) with a fixed learning rate of 1e-4 and a mini-batch size of 100. We applied different learning rate for generators/discriminators according to TTUR (Heusel et al., 2017) . The learning rate for discriminators is 3 * 10 -4 while for generator is 10 -4 . We perform 1 discriminator update per generator update. Results are reported after training for 100000 iterations.

:



Figures and Tables:

Figure 2: An example of composition and decomposition for example 2.

Figure 3: Examples of MNIST-MB dataset. 5x5 grid on the left shows examples of MNIST digits (first component), middle grid shows examples of monochromatic backgrounds (second component), grid on the right shows examples of composite images.

Figure 4: Examples of MNIST-BB dataset. 5x5 grid on the left shows examples of MNIST digits (first component), middle grid shows examples of monochromatic backgrounds with shifted, rotated, and scaled boxes (second component), grid on the right shows examples of composite images with digits transformed to fit into appropriate box.

(t(b) = 255 -b) and that the model learns to undo this inversion in the composition function (t(t(b)) = b) so that cyclic consistency (d(c(o 1 , o 2 )) ≈ [o 1 , o 2 )] and c(d(y)) ≈ y is satisfied. We note that this is an interesting case where symmetries in component distributions results in the model learning component distributions only up to a phase flip.

Figure 5: Given component generators and composite data, decomposition can be learned.

Figure 7: Given one component, decomposition function and the other component can be learned.

Figure8: Some results of chain learning on MNIST-BB. First we learn a background generator given foreground generator for "1" and composition network, and later we learn the foreground generator for digit "2" given background generator and composition network.

A resolving matrix, R, for a composition function c and random variable X, is a matrix of size |range(Z)| × |range(Y )| with entries R z,y = x∈range(X) p(X = x)1[z = c(x, y)] (see figure10). Definition 2. A composition function c is bijective if it is surjective and there exists a decomposition function d such that1. d(c(x, y)) = x, y; ∀x ∈ range(X), y ∈ range(Y ) 2. c(d(z)) = z; ∀z ∈ range(Z)equivalently, c is bijective when c(x, y) = c(x , y ) iff x = x and y = y . We refer to decomposition function d as inverse of c.

x)1[x = d(z) x ]1[y = d(z) y ] = 0 α d(z)y P (X = d(z) x ) = 0 (5) using the the definition of R in the first equality, making the substitution 1[z = c(x, y)] = 1[x = d(z) x ]1[y = d(z) y ] implied by the bijectivitiy of c in the second equality and rearranging / simplifying terms for the third. Since P (X = x) > 0 for all x ∈ range(X), α y = 0 for all y ∈ {y | y = d(z) y }. By the surjectivity of c, α y = 0 for all y ∈ range(Y ), and R has full column rank. Theorem 3. Let Assumptions 1 and 2 hold. Further, assume that resolving matrix of X and c has full column-rank. If optimum of inf p(Y )

x , y ) = d (z ) = d * (z ) = (x * , y * ). By optimality of d , c(d (z )) = z or the objective would be positive. Hence, c(x , y ) = z . By Definition 2, c(d * (z )) = z , hence c(x * , y * ) = z . However, d (c(x * , y * )) = (x * , y * ) and expectation in equation 10 over X or Y would be positive. Consequently, the objective would be positive, violating assumption of optimum of 0. Hence, inverse of c is the only function which achieves optimum 0 in equation 10. 7 IMPLEMENTATION DETAILS 7.1 ARCHITECTURE FOR MNIST / FASHION-MNIST

Table for all losses Loss name Detail

Table content: <table/>

We show Frechet inception distance

Table content: <table><tr><td>Methods</td><td>Digits</td><td colspan="2">Foreground Fashion</td><td>Foreground+background Digits Fashion</td></tr><tr><td>WGAN-GP</td><td colspan="2">6.622 ± 0.116</td><td colspan="2">20.425 ± 0.130 25.871 ± 0.182 21.914 ± 0.261</td></tr><tr><td>By decomposition</td><td colspan="2">9.741 ± 0.144</td><td colspan="2">21.865 ± 0.228 13.536 ± 0.130 21.527 ± 0.071</td></tr></table>

a unique solution of equation 9. Hence, optimum of equation 6 is achieved only Y which are identically distributed as Y .

Table content: <table><tr><td>Corollary 1. Let Assumptions 1 and 2 hold. Further, assume that c is a bijective. If, an optimum</td></tr><tr><td>of equation 6 is achieved is achieved for some random variable Y , then Y and Y have the same</td></tr><tr><td>distribution.</td></tr></table>

Peer review (iclr2019):
 The paper proposes a framework for training generative models that work on composed data. The models are trained in an adversarial fashion. The authors apply it to decompose foreground/background parts on MNIST images, and to perform sentence composition/decomposition. High level comments: * Clarity: In terms of language and writing style, the paper is written very clearly and easy to follow. In terms of presentation, there are some details that are omitted which would have made understanding easier and the work more reproducible. * Quality: The idea that is introduced seems intuitive and reasonable, but the experiments does not have enough details to prove that this method works (i.e. no quantitative results presented). Moreover, the presentation of the method is not very well done (missing details), especially since the authors used the upper limit of 10 pages. * Originality: I am not familiar with the literature of generative models to judge this precisely, but according to the related work section it sounds like an original idea that is worth sharing. * Significance: I believe the idea of modeling data composition explicitly sounds intuitive and interesting, and it is worth sharing. However, the experimental section does not have enough evidence that it is actually possible to learn this, so it is not clear whether the contribution is significant. Pros: - interesting new problem formulation - simple and clear language - the theoretical analysis in the last section could be interesting more generally in the context of GANs - the framework is applied on 2 different modalities: images and text. Cons: - hard to tell whether this approach works since the metrics for evaluation are not specified and there are no quantitative results in the experimental section (only 1 qualitative example per task) - the work is not reproducible due to the lack of details (see more explanations below) - the theoretical analysis is a standalone piece of the paper, without any discussion about the implications, or making connections to the previous sections. Detailed comments: 1. I believe the weakest part of this paper is the evaluation section. The authors run their framework on 4 tasks of increasing difficulty. While the MNIST examples make for a nice and intuitive qualitative analysis, the are no quantitative results at all. The only result that is reported for each task is one qualitative picture. The authors make statements such as “The decomposition network learns to decompose the digits and backgrounds correctly” , “Given one component, decomposition function and the other component can be learned.” but there is not mention for how these conclusion are made (no metrics, no numbers). Indeed, it is difficult in general to quantify the results of generative models, but most other GAN papers introduce some sort metric that can be used to aggregate the evaluation on an entire dataset. If the authors manually inspected the results, they should at least report how many images they inspected and how many looked correct. 2. Aside from evaluation, there are some other details missing from the presentation. The individual details may not be major, but because all of these are missing together, it really affects the overall quality of the paper. For example:  the authors state: “To train discriminator(s), a regularization is applied. For brevity, we do not show the regularization term (see Petzka et al. (2017)) used in our experiments.”. For reproducibility purposes, I believe it is important to at least mention the type of regularization, at least in the appendix.  There is a parameter alpha used to balance the losses. What values was used in the experiments?  Choices of models are often not explained. Why did you choose that form for c(o1, o2) in section 3.3? Why DCGAN for component generators, and U-net for decomposition?  It is not explained in detail how the Yelp-reviews dataset is altered to achieve coherence. The authors mention that “As we sample a pair independently, the input sentences are not generally coherent but the coherence can be achieved with a small number of changes.”. However, the specific algorithm by which these changes are made is not specified, and thus it can’t be reproduced. 3. The theoretical section is an interesting contribution, but the paper just states a list of theorems without making any connections to the applications used before, or a broader discussion about how these fit in the context of GANs more generally. 4. My understanding is that both datasets used are created by the authors by making alterations to MNIST and Yelp-reviews dataset, thus making them to some extent synthetic datasets suited to fit this problem formulation. I would have like to see how this composition/decomposition works on existing datasets with no alterations. Does it still work? 5. In section 2.3, in the coherent sentence experimental setting, I don’t fully understand the design of the task. Figure 2 shows an example where composition and decomposition are not symmetric (i.e. composing then decomposing does not go back to the input sentences), although one of your losses is supposed to ensure exactly this cyclic consistency. Why not choose another problem that doesn’t directly violate your assumptions? Minor issues: 6. From the related work section, it is not clear how your approach is different from Azadi et al. (2018). Please include more details. 7. In section 2.4, you mention using Wasserstein GANs, with no further details about this model (not even a one line description). Without reading their paper, the readers of your paper could not easily follow through this section. The losses further introduced are also not explained intuitively (e.g. what do the two expectation terms in l_g_i represent?). 8. I believe there are some errors in which tasks reference which figures in section 3.3. Should Task 2 refers to Figure 6, and Task 3 to Figure 7? 9. What exactly is range(.) in section 4? If this refers to the interval of values that a variable can take, the saying “is a matrix of size |range(Z)| × |range(Y )|” doesn’t exactly make sense. Please define formally. Final remarks and advice: Overall, I believe the paper introduces some interesting ideas. There is definitely value in the problem definition and theoretical analysis. However, I believe the paper needs more work on presentation and evaluation, especially since the authors opted for 10 pages and according to ICLR guidelines “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.”. Hopefully the above comments will help the authors improve this work!