Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators
Authors: Daniel Stoller
Sebastian Ewert
Simon Dixon
Venue:  (2020)
Keywords: Adversarial Learning
Semi-supervised Learning
Image generation
Image segmentation
Missing Data

Abstract: Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting.
However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels. Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance. To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies. This allows splitting the discriminator in a GAN into multiple "sub-discriminators" that can be independently trained from incomplete observations. Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework. We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available. For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 14.9% over CycleGAN while using only 25 additional paired examples.

Published as a conference paper at ICLR 2020
TRAINING GENERATIVE ADVERSARIAL NETWORKS FROM INCOMPLETE OBSERVATIONS USING FACTORISED DISCRIMINATORS
Daniel Stoller
Sebastian Ewert Spotify
Simon Dixon
Queen Mary University
Queen Mary University
London, UK
Berlin, Germany
London, UK
d. s.stolle tetooler@@mull. acc.ul
sewe rt eseerttspfff.....m
di xon@gmul ac. uk
ABSTRACT
Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting. However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels. Therefore, methods such the CycleGAN use way to leverage additional labelled data for improved performance. To address this shortcoming, we show how to factorise the joint data distribution into set of lower-dimensional distributions along with their dependencies. This allows splitting the discriminator in a GAN into multiple "sub-discriminators" that can be independently trained from incomplete observations. Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework. We apply our method to image gencration, image segmentation and audio source separation, and obtain improved performance over standard GAN when additional incomplete training examples are available. For the Cityscapes segmentation task in particular, method also improves accuracy by absolute 14.9% over CycleGAN while using only 25 additional paired examples.
easily available unlabelled data, but do not offer
INTRODUCTION
In generative adversarial networks (GANs) (Goodfellow et al. (014) generator network is trained to produce samples from given target distribution. To achieve this, a discriminator network is employed to distinguish between "real" samples from the dataset and "fake" samples from the generator network. The discriminator's feedback used by the generator to improve its output. While GANs have become highly effective at synthesising realistic examples even for complex data such as natural images (Radford et al.) 2015; Karras et al. 2018). they typically rely on large training datasets. These are not available in many cases, especially for prediction tasks such audio source separation (Stoller et al. 2018) or image-to-image translation (Zhu et al.) 2017). Instead, one often encounters many incomplete observations, such as unpaired images image-to-image translation, or isolated source recordings in source separation. However, standard GANs cannot be trained with these observations. Recent approaches that work with unpaired data can not make use of additional paired data (Zhu et al. 2017) or lead to computational overhead due to additional generators and discriminators that model the inverse of the mapping of interest (Almahairi al., 2018 Gan et 2017). For training the generator, multiple losses are combined whose interactions are not clear and that do not guarantee that the generator converges the desired distribution.
In this paper, we adapt the standard GAN framework to enable training predictive models with both paired and unpaired data as well as generative models with incomplete observations. To achieve this, we split the discriminator into multiple "marginal" discriminators, each modelling separate set of dimensions of the input. As this modification on its own would ignore any dependencies between these parts, incorporate two additional "dependency discriminators". each focusing only on inter-pan relationships. We show how the outputs from these marginal and dependency discriminators be recombined and used estimate the same density ratios in the original

---

Published as a conference paper at ICLR 2020
GAN framework - which enables training any generator network in unmodified form. In contrast to previous GANs, our approach only requires full observations to train the smaller dependency discriminator and can leverage much bigger, simpler datasets to train the marginal discriminators, which enables the generator to model the marginal distributions more accurately. Additionally, prior knowledge about the marginals and dependencies can be incorporated into the architecture of each discriminator. Deriving from first principles, we obtain a consistent adversarial learning framework without the need for extra losses that rely on more assumptions or conflict with the GAN objective.
In our experiments, we apply our approach ("FactorGAN") | to two image generation tasks (Sec- tions 4.1 and 4.2), image segmentation (Section 4.3) and audio source separation (Section 4.4), and observe improved performance in missing data scenarios compared to GAN. For image segmen- tation, we also compare to the CycleGAN (Zhu et al. 2017), which does not require images to be paired with their segmentation maps. By leveraging both paired and unpaired examples with a unified adversarial objective, we achieve substantially higher segmentation accuracy even with only 25 paired samples than GAN and CycleGAN models.
2
METHOD
After a brief summary of GANs in Section 2.1 we introduce our method from a missing data perspective in Section 2.2 before extending it to conditional generation (Section 2.3 and the case of independent outputs (Section 2.4
2.1 GENERATIVE ADVERSARIAL NETWORKS
To model a probability distribution Ps over x E Rdd we follow the standard GAN framework and introduce generator model G In - IR that maps n-dimensional input Pa to d-dimensional sample (z))), resulting in the generator distribution dz- To train GG such that Oz approximates the real data density Pa, a discriminator Dg Idd + (0, 1) is trained to estimate whether given sample is real generated:
arg max Ex-p- log Dx(x) Ex log(1 Da(x)).
In the non- parametric limit Goodfellow et al. 2014), D&(x) approaches D(x) := Pa (.) (xc at
Pr,
every point x. The generator is updated based on the discriminator's estimate of D(x). this paper, we use the alternative loss function for Go as proposed by Goodfellow et al. (014):
arg max Ez-p, log (z)
#
2.2 ADAPTATION TO MISSING DATA
In the following we consider the case that incomplete observations are available in addition to our regular dataset (i.e. simpler yet larger datasets). particular, we partition the set of input dimensions of x into K (2 < K < d) non-overlapping subsets ...... DR- For each i E {1..... K), an incomplete ("marginal") observation x' can drawn from ptg which is obtained from ps: after marginalising out all dimensions not in D. Analogously, d' denotes the i-th marginal distribution of the generator G Next, we extend the existing GAN framework such we can employ the additional incomplete observations. In this context, main hurdle is that a standard GAN discriminator is trained with samples from the full joint Pz- To eliminate this restriction, we note that D(x) can be mapped to
a "joint density ratio" Ps (x( by applying the bijective function h [0. 1) > R hh(a) = - For our approach, we exploit that this joint density ratio can be factorised into product of density ratios:
(x(x)
(x)
(x(x)
k(D(x))
with
Oz (x)
co
(x)
(x+)
(3)
(x)
qu (x)
cp(x)
and cx(
11 pt (x*)
1
(x!)
'Code available wttphhttt//t/th/ee http://...oooo//////tteeeeee
2

---

Published as a conference paper at ICLR 2020
Each "marginal density ratio" g. (x')
[xxx) captures the generator's output quality for one marginal vari- able x", while the op and eo terms describe the dependency structure between marginal variables in the real and generated distribution, respectively. Note that our theoretical considerations assume that the densities Ps and qu are non-zero everywhere. While this might not be fulfilled practice, our implementation does not directly compute density ratios and instead relies on the same assump- tions as Goodfellow et al. (2014). We can estimate each density ratio independently by training a "sub-disrriminator" network, and combine their outputs to estimate Dxx), as shown below.
Estimating the marginal density ratios: To estimate e'(x') for each {I,..., K) we train (x(x
"marginal discriminator network" De, RIP: - (0, 1) with parameters 0; to determine whether
marginal sample x' is real or generated following the GAN discriminator loss in Equation I This
allows making use of the additional incomplete observations. In the con-parmetrric limit, Do. (x*)
will approach (x = p2(x (x so that we can use h (x²)) as an estimate of P_(x))
24 (x')
(x*))
Estimation of (x(x) and cQ(xx: Note that ep and co are also density ratios, this time containing distribution over x in both the numerator and denominator the main difference being that in the latter the individual parts x' are independent from each other. To approximate the ratio cp, we can apply the same principles as above and train a "p-dependency discriminator" D2p Rdd - (0, 1) to distinguish samples from the two distributions, i.e. to discriminate real joint samples from samples where the individual parts are real but were drawn independently of each other (i.e. the individual parts might not originate from the same real joint sample). Again, in the non- con-parmmetri limit, its response approaches D D*(()
pa TT (i.()) and thus op can be approximated via h D&p-
(x(
Analogously, the co term is estimated with "q-dependency discriminator" D here, we compare joint generator samples with samples where the individual parts were shuffled across several generated samples (to implement the independence assumption).
Joint discriminator sample complexity: In contrast to CQ. where the generator provides an infinite number of samples, estimating ep without overfitting the limited number of joint training samples can be challenging. While standard GANs suffer from the same difficulty, our factorisation into specialised sub-units allows for additional opportunities to improve the sample complexity. In particular, we can design the architecture of the p-dependency discriminator to incorporate prior knowledge about the dependency structure
Combining the discriminators: As the marginal and the p- and q-dependency sub-discriminators provide estimates of their respective density ratios, we can multiply them and apply h- to obtain the desired ratio D(x), following Equation 3 This can be implemented in simple and stable fashion using a linear combination of pre-activation sub-discriminator outputs followed by a sigmoid (see Section .A.4)on/AAAle for details and proof). The time for generator update step grows linearly with the number of marginals K, assuming the time to update each of the K marginal discriminators remains constant.
2.3 ADAPTATION TO CONDITIONAL GENERATION
Conditional generation, such as image segmentation or inpainting, can be performed with GANs by using generator Go that maps conditional input x' and noise to output x², resulting output probability 4o [x(x) [x)).
ieesutinn in an
When viewing x and x" as parts of joint variable x = (x1, x²) with distribution Pa. we can also frame the above task matching Pz to the joint generator distribution (x) := p (x2 (xx))x(( (x" xx)). In standard conditional GAN, the discriminator is asked to distinguish between joint samples from ps and q. which requires paired samples from P, and is inefficient as the inputs x! are the same in
'Samples are drawn from pi, and d: instead of Ps and gr. respectively. IIf only certain features of marginal variable influence the dependencies, can limit the input to the p-dependency discriminator these features instead of the full marginal sample prevent overfitting.
3

---

Published as a conference paper at ICLR 2020
both Pa and dr- In contrast, applying factorisation principle from Equation 3 to x and x² (for the special case K = 2) yields
(x)
Pu (x) Qu (x)
p x
P
(x2)
(x(x) (x²2 CQ (x) (x2)
(x)
(x2)
4 (x²)
suggesting the use of p- and q-dependency discriminator to model the input-output relationship. and marginal discriminator over x" that matches aggregate generator predictions from d to real output examples from v Note that do not need marginal discriminator for x², which increases computational efficiency. This adaptation can also involve additionally partitioning x² into multiple partial observations shown in Equation
2.4 ADAPTION TO INDEPENDENT MARGINALS
In case the marginals can be assumed to be completely independent, one can remove the p-dependency discriminator from our framework, since (x(x) = for all inputs x. This approach can be useful in the conditional setting, when each output is related to the input but their marginals independent from each other. In this setting, our method is related to adversarial ICA (Brakel & Bengio, 2017). Note that the q-dependency discriminator still needs to be trained the full generator outputs if the generator should not introduce unwanted dependencies between the marginals.
2.5 FURTHER EXTENSIONS
There are many more ways of partitioning the joint distribution into marginals. We discuss two additional variants (Hierarchical and auto-regressive FactorGANs) of our approach in Section
RELATED WORK
For conditional generation, "CycleGAN" (Zhu et al. 2017) exploits unpaired samples by assuming one-to-one mapping between the domains and using bidirectional generators (along with Gan ct al. (2017). while FactorGAN makes no such assumptions and instead uses paired examples to learn the dependency structure. Almahairi et al. (2018) and Tripathy al. 2018) learn from paired examples with an additional reconstruction based loss, but use a sum of many different loss terms which have be balanced by additional syper-parameters. Additionally, it can not be applied to generation tasks with missing data or prediction tasks with multiple outputs. Brakel & Bengio (2017) perform independent component analysis in an adversarial fashion using discriminator identify correlations. Similarly to our q-dependency discriminator, the separator outputs are enforced to be independent, but our method fully adversarial and can model arbitrary dependencies with the p-dependency discriminator. GANs were also used for source separation, but dependencies were either ignored (Zhang et al., 2017) or modelled with an additional 1.2 loss (Stoller al., 2018) that supports only deterministic separators.
Pu et al. 2018) use GANs for joint distribution modelling by training a generator for each possible factorisation of the joint distribution, but this requires K! generators for K marginals, whereas we assume either all parts or exactly one part the variable of interest is observed to avoid functional redundancies between the different networks. Karaletsos (2016) propose adversarial inference local factors high-dimensional joint distribution and factorise both generator and discriminator based independence assumptions given by Bayesian network, whereas keep a joint sample generator and model all dependencies. Finally, Yoon et al.) (2018) randomly mask the inputs to a GAN generator it learns to impute missing values, whereas our generator aims learn a transformation where inputs are fully observed.
4 EXPERIMENTS
To validate method, compare FactorGAN with the regular GAN approach, both for unsupervised generation as well as supervised prediction tasks. For the latter, we also compare to the CycleGAN (hhu al, 2017) unsupervised baseline. To investigate whether FactorGAN makes
4

---

Published as a conference paper at ICLR 2020
efficient use of all observations, vary the proportion of the training samples available for joint sampling (paired), while using the rest to sample from the marginals (unpaired). We train all models using a single NVIDIA GTX 1080 GPU.
Training procedure For stable training, we employ spectral normalisation (Miyato et al. 2018 on each discriminator network to ensure they satisfy Lipschitz condition. Since the overall output used for training the generator is simply linear combination of the individual discriminators (see Section A.4)). the generator gradients are also constrained in magnitude accordingly. Unless otherwise noted, we use an Adam optimiser with learning rate 10-1 and a batch size of 25 for training all models. We perform two discriminator updates after each generator update.
4.1 PAIRED MNIST
Our first experiment will involve "Paired MNIST", a synthetic dataset of low complexity whose dependencies between marginals can be casily controlled. More precisely, we generate paired version of the original MNIST datasefbby creating samples that contain a pair of vertically stacked digit images. With a probability of X, the lower digit chosen during random generation is the same as the upper one, and different otherwise. For FactorGAN, we model the distributions of upper and lower digits as individual marginal distributions (K* = 2).
Experimental setup We compare the normal GAN with our FactorGAN, also including a variant without p-dependency discriminator that assumes marginals to be independent ("FactorGAA-nno- We conduct the experiment with À = 0.1 and = 0.9 and also vary the amount of training samples available in paired form, while keeping the others as marginal samples only usable by FactorGAN. For both generators and discriminators, we used simple multi-layer perceptrons (Tables I
and 2
To evaluate the quality of generated digits, we adopt the "Frecht Inception Distance" (FID) as metric (Heusel et al. 2017]. It is based on estimating the distance between the distributions of hidden layer activations of pre-trained Imagenet object detection model for real and fake examples. To adapt the metric to MNIST data, we pre-train a classifier to predict MNIST digits (see Table on the training set for 20 epochs, obtaining a test accuracy of 98%. We input the top and bottom digits in each sample separately to the classifier and collect the activations from the last hidden layer (FC1) to compute FIDs for the top and bottom digits, respectively. We use the average of both FIDs to measure the overall output quality of the marginals (lower value is better).
Since the only dependencies in the data are digit correlations controlled by X, we can evaluate how well FactorGAN models these dependencies. We compute PD Ds) as the probability for a real sample to have digit D; E {0... ........9 9] at the top and digit D2 E {O...... 9} at the bottom, along with marginal probabilities ph and pD (Da) (and analogously qD Ds) for generated data). Since we do not have ground truth digit labels for the generated samples, we instead use the class predicted ratio between a joint and the product of
by the pre-trained classifier. We encode the dependency its marginals, where the ratios for real and generated data are ideally the same. Therefore, we take their absolute difference for all digit combinations as evaluation metric (lower is better):
9
PD(De: Do)
(D Do) of (D.)g))))
daep
(5)
100
(D(r)))
D,=0 D,=0
Note that the metric computes how well dependencies in the real data are modelled by a generator, but not whether it introduces any additional unwanted dependencies such top and bottom digits sharing stroke thickness, and thus presents only a necessary condition for a good generator.
The results of our experiment are shown in Figure I Since FactorGANN co-cp trains on all samples independently of the number of paired observations, both FID and daep are constant. As expected, CactorGAN-no-cee delivers good digit quality, and performs well for X = 0.1 (as it assumes independence) and badly for X = 0.9 with regards to dependency modelling.
Results
FactorGAN outperforms GAN with small numbers of paired samples in terms of FID by exploiting the additional unpaired samples, although this gap closes as both models eventually have access to
http:://yam lecun. wwwpeetb//cccppotpt/ctttpppoo m st./
5

---

Published as a conference paper at ICLR 2020
102
GAN, X-0.1
Dependency day
CAN, A-0.1
GAN, k=0.9
FattorGAM 1-0.1
FachoGAN A-0.1
FactorGAN A=0.9
FactorGAN, X-09
Nonoopp A
FactnCAA/--no-p.. À 000.1
101
FactorGAN op.
FactorGAN
10
10000 20000
60C00
1000
5000
10000
20000
60000
Paired samples
samples
(a) FID value, averaged over both digits
(b) Dependency metric
Figure I: Performance with different numbers of paired training samples and settings for À compared between GAN and FactorGAN with and without dependency modelling.
the same amount of data. FactorGAN also consistently improves in modelling the digit dependencies with increasing number of paired observations. For À 0.1, this also applies to the normal GAN, although its performance is much worse for small sample sizes as it introduces unwanted digit dependencies. Additionally, its performance appears unstable for À 0.9, where it achieves the best results for a small number of paired examples. Further improvements in this setting could be gained by incorporating prior knowledge about the nature of these dependencies into the p-dependency discriminator to increase its sample efficiency, but this is left for future work.
4.2 IMAGE PAIR GENERATION
In this section, we use GAN and FactorGAN for generating pairs of images in an unsupervised way to evaluate how well FactorGAN models more complex data distributions.
Datasets We use the "Cityscapes" dataset (Cordts et al. 2016) and the "Edges2Shoes" dataset (Isola al. 20166). To keep the outputs in continuous domain, we treat the segmentation maps in the Cityscapes dataset as RGB images, instead of set of discrete categorical labels. Each input and output image is downsampled complexity and to ensure stable GAN training.
64 x 64 pixels as preprocessing step reduce computational
Experimental setup We define the distributions of input well as output images marginal distributions. Therefore, FactorGAN uses two marginal discriminators and a p- and q-dependency discriminator. All discriminators employ convolutional architecture shown in Table 5 with W - 6 and H = 6. To control for the impact of discriminator size, we also train a GAN with twice the number of filters in each discriminator layer to match its size with the combined size of the FactorGAN discriminators. The same convolutional generator shown in Table I used for GAN and FactorGAN. Each image pair is concatenated along the channel dimension to form sample, so that C 6 for the Cityscapes and C = 4 for the Edges2Shoes dataset (since edge maps are greyscale). We make either 100, 1000, or all training samples available in paired form, investigate whether FactorGAN can improve upon GAN by exploiting the remaining unpaired samples or match its quality if there are none.
For evaluation, we randomly assign 80% of validation data to a "test-train" and the rest to a "test-test" partition. We train an LSGAN discriminator (Mao et al. 2017) with the architecture shown in Table (but half the filters in each layer) on the test-train partition for 40 epochs to distinguish real from generated samples, before measuring its loss on the test set. We continuously sample from the generator during training and testing instead of using fixed set of samples to better approximate the true generator distribution. As evaluation metric, we use the average test loss over 10 training runs, which was shown to correlate with subjective ratings of visual quality (Im et al. 2018) and also with our own quality judgements throughout this study. A larger value indicates better performance, as we flipped sign compared to Im et al. (2018). While the quantitative results appear indicative of output quality, accurate GAN evaluation is still an open problem and so we encourage the reader to judge generated examples given in Section A.5

---

Published as a conference paper at ICLR 2020
Edges2Stioss
Chyscaps
Model
GAN FactorGAN GAM (bug)
Paired
Faired samales
Figure 2: GAN and FactorGAN output quality estimated by the LS metric for different datasets and numbers of paired samples. Error bars show 95% confidence intervals.
(a) GAN
(b) FactorGAN
Figure 3: Examples generated for the Edges2Shoes dataset using 100 paired samples
Results Our FactorGAN achieves better or similar output quality compared to the GAN baseline in Figure 2 For the Edges2Shoes dataset, the performance gains most pro-
in all cases, nounced for small numbers of paired samples. On the more complex Cityscapes dataset, FactorGAN outperforms GAN by large margin independent of training set size, even when the discriminators are closely matched in size. This suggests that FactorGAN converges with fewer training iterations for Ga. although the exact cause is unclear and should be investigated in future work.
We show some generated examples in Figure 3 Due to the small number of available paired samples, we observe a strong mode collapse the GAN in Figure Bal while FactorGAN provides high-fidelity, diverse outputs, as shown in Figure 33b Similar observations can be made for the Cityscapes dataset when using 100 paired samples (see Section A.5.2).
4.3 IMAGE SEGMENTATION
Our approach extends the case of conditional generation (see Section 2.3 and important image segmentation task on the Cityscapes dataset, where we ask the generator to predict segmentation map for a city scene (instead of generating both from scratch in Section 4.2).
tackle complex
Experimental setup We downsample the scenes and segmentation maps to 128 x 128 pixels and use U-Net architecture (Ronneberger et al. 2015) (shown in Table 6 with W = 7 and = 3) as segmentation model. For FactorGAN, we use marginal discriminator to match the distribution of real and fake segmentation maps to ensure realistic predictions, which enables training with isolated city scenes and segmentation maps. To ensure the correct predictions for each city scene, p- and g-dependency discriminator learns the input-outpi relationship using joint samples, both employing convolutional architecture shown in Table Note that in Section 4.2 output segmentation maps in the RGB space instead of performing classification. In addition to the MSE in the RGB space, we compute the widely used pixel-wise classification accuracy (Cordts al.) 2016) by assigning each output pixel to the class whose colour has the lowest Euclidean distance in RGB space.
Using the same experimental setup (including network architectures), also implement the Cycle- GAN (Zhu et al. 2017) as an unsupervised baseline. For the CycleGAN objective, the same GAN losses as shown and are used)
Results The results in Figure demonstrate that approach exploit additional unpaired samples to deliver better MSE and accuracy than a GAN and less noisy outputs as seen in Figure
CCode to perform training iteration and default loss weights taken from the official codebase at https: http:::///tte thubtthttbtttto/// www.:/www..momnnmaaaaeee http://ggotoo.....mmmm///////oo/ Cyel eGAN- and-pix2pix
7

---

Published as a conference paper at ICLR 2020
6.04
Moan squarel ever
Accumns (%)
Medel
GAN
FartoGAN
CydeGAMi
0.03
100
900
3000
2000
100
1000
3000
Pained samples
Painel samples
Figure 4: MSE (left) and accuracy (right) obtained on the Cityscapes dataset with different numbers of paired training samples for the GAN and FactorGAN
(a) GAN
(b) FactorGAN
Figure 5: Segmentation predictions made the Cityscapes dataset for the of test inputs, compared between models, using 100 paired samples for training
When using only 25 paired samples, FactorGAN reaches 71.6% accuracy, outperforming both GAN and CycleGAN by an absolute 17.7% and 14.9%, respectively. CycleGAN performs better than GAN only in this setting, and increasingly falls behind both GAN and FactorGAN with growing number of paired samples, likely since GAN and FactorGAN are able to improve their input-output mapping gradually while CycleGAN remains reliant on cycle consistency assumption. These findings suggest that FactorGAN can efficiently learn the dependency structure from few paired samples with more accuracy than a CycleGAN that is limited by its simplistic cycle consistency assumption.
4.4 AUDIO SOURCE SEPARATION
We apply method audio source separation another conditional generation task to investigate whether transfers across domains. Specifically, we separate music signals into singing voice and accompaniment, detailed in Section A.2) As in Section 4.3) we find that FactorGAN provides better separtion than GAN, suggesting that our factorisation is useful across problem domains.
5 DISCUSSION
We find that FactorGAN outperforms GAN across all experiments when additional incomplete samples are available, especially when they are abundant in comparison the number of joint samples. When using only joint observations, FactorGAN should be expected to match the GAN in quality, and it does so quite closely in most of our experiments. Surprisingly, outperforms GAN in some scenarios such as image segmentation even with matched discriminator sizes phenomenon we do fully understand yet and should be investigated the future. For image segmentation, FactorGAN substantially improves segmentation accuracy compared to the fully unsupervised CycleGAN model even when only using 25 paired examples, indicating that it can efficiently exploit the pairing information.
Since the p-dependency discriminator does not rely on generator samples that change during training. it could be pre- traitrd to reduce computation time, but this led to sudden training instabilities in experiments. We suspect that this is due to mismatch between training and testing conditions for the p-dependency discriminator since it trained real but evaluated fake data, and neural networks
8

---

Published as a conference paper at ICLR 2020
yield overly confident predictions outside the support of the training set (Gal & Ghahramani 2016 Therefore, we expect classifiers with better uncertainty calibration to alleviate this issue.
CONCLUSION
In this paper, we demonstrated how a joint distribution can factorised into a set of marginals and dependencies, giving rise to the FactorGAN GAN in which the discriminator is split into parts that can be independently trained with incomplete observations. For both generation and conditional prediction tasks in multiple domains, additional incomplete observations are available. For Cityscapes scene segmentation in particular, FactorGAN achieves much higher accuracy than the supervised GAN well the unsupervised CycleGAN, while requiring only 25 of all examples to be annotated.
find that FactorGAN outperforms the standard GAN when
Factorising discriminators enables incorporating more prior knowledge into the design of neural architectures in GANs, which could improve empirical results in applied domains. The presented factorisation is generally applicable independent of model choice, many existing GAN- -ased approaches. Since the joint density can be factorised in different ways. multiple extensions are conceivable depending on the particular application (as shown Section A..).. This paper derives FactorGAN from the original GAN proposed by Goodfellow et al. (2014) by exploiting the probabilistic view of the optimal discriminator. Adapting the FactorGAN to alternative GAN objectives (such as the Wasserstein GAN (Arjovsky et al., (017)) might be possible as well. Instead of relying on additional techniques such as spectral normalisation to ensure training stability, which our theory does not explicitly incorporate, this would enable the use of an inherently more stable GAN variant with the same theoretical guarantees.
be readily integrated into
ACKNOWLEDGMENTS
We thank Emmanouil Benetos for his helpful feedback. Daniel Stoller is funded by EPSRC grant EP/L01632X//.
REFERENCES
Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data. CoRR, hts/1802.1015.. 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN, CoRR, abs// 70117701.7777 701.07875 2017.
Brakel and Y. Bengio. Learning Independent Features with Adversarial Nets for Non-linear ICA. ArXiv e-prints, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty Deep Learning. In Proc. of the International Conference on Machine Learning (ICML), volume 48, pp. 1050-1059, 2016.
Zhe Gan, Liqun Chen, Weiyao Wang, Yunchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin. Triangle Generative Adversarial Networks. CoRR, abs/1709.065488 September 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.

---

Published as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances In Neural Information Processing Systems, pp. 6629-6640, 2017.
Daniel Jiwoong Im, Alllan He Ma, Graham W. Taylor, and Kristin Branson. Quantitatively Evaluating GANs With Divergences Proposed for Training, In Proc. of the International Conference on Learning Representations (ICLR), 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros, Image-to-Image Translation with Conditional Adversarial Networks. CoRR, abs/ 1tss/6111.0004 1.07004, November 2016.
Theofanis Karaletsos. Adversarial Message Passing For Graphical Models. CoRR, abs/1612.050044 December 2016.
Tero Karras, Samuli Laine, and Timo Aila. Style-Based Generator Architecture for Generative Adversarial Networks. CoRR, abs/1812.04948, December 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K.
Lau, Zhen Wang, and Stephen Paul Smolley. Least Squares Generative Adversarial Networks. In Proc. of the IEEE International Conference on Computer Vision (ICCV), October 2017.
Takeru Miyato. Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. arXiv: /802.05957 [cs, stat), February 2018.
Olof Mogren, C-RNN-GAN: A continuous recurrent neural network with adversarial training. In Constructive Machine Learning Workshop (CML) at NIPS 2016, 2016.
Yunchen Pu, Shuyang Dai, Zhe Gan, Weiyao Wang, Guoyin Wang, Yizhe Zhang, Ricardo Henao, and Lawrence Carin Duke. JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets. In Jennifer Dy and Andreas Krause (eds.), Proc. of the International Conference Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 4151-4160, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. CoRR, abs/1511 1.064444,, 2015.
Zafar Rafii Antoine Liutkus, Fabian-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The Musdb 18 Corpus For Music Separation, December 2017.
O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Proc. of the International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 234-24442222 Springer, 2015.
Casper Kaae Sønderby. Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised map inference for image super-resolution. In Proc. of the International Conference on Learning Representations (ICLR), 2017.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction. In Proc. of the IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2391 2295,, Calgary, Canada, 2018. IEEE.
Soumya Tripathy, Juho Kannala, and Esa Rahtu. Learning image-to-image translation using paired and unpaired training samples. CoRR, abs/805.031 189, May 2018.
E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing. 14(4):1462 1469, 2006. ISSN 1558-7916. doi: 10. 10.1109/TSA.2000 858005.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets. CORR, abs/ 806.02920, June 2018.
Ning Zhang, Junchi Yan, and Yu Chen Zhou Unsupervised Audio Source Separation via Spectrum Energy Preserved Wasserstein Learning. CoRR, abs/1711.0411 2017.
Jun Yan Zhu, Taesung Park. Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. of the IEEE International Conference on Computer Vision (ICCV, pp. 2223-2232, 2017.
10

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 TABLES
Table 1: The architecture of generator the MNIST dataset. All layers have biases.
Layer
Input shape
Outputs
Output shape
Activation
128
128
ReLUU
FC
128
128
128
ReLU
FC
128
1568
56 x 28
X
Sigmoid
Table The architecture of our discriminators on the paired MNIST dataset. W 28 for marginal, W 56 for dependency discriminators.
Layer
Input shape
Outputs
Output shape
Activation
FC
W 28
128
128
LeakyReLU
128
LeakyReLU
FC
128
Table 3: The architecture of our MNIST classifier. Dropout with probability 0.5 is applied to FC1 outputs.
Layer
Input shape
Filter size
Stride
Outputs
Output shape
Activation
Conv
28 x 28 x 1
5 x 5
1
10
28
28
x 10
AvgPool
28 28 x 10
2
2 2
10
12
12
10
LeakyReLU
Conv
12 12 x
5
1x
12
AvgPool
12 x 12 x 20
2x2
2
20
4
x
4 x 20
LeakyReLU
FCI
320
50
50
LeakyReLU
FC2
50
10
10
Table 4: The architecture our convolutional generator. "ConvT" represent transposed convolutions.
All layers have biases. The number of output channels depends the task.
Layer
Input shape
Filter size
Stride
Outputs
Output shape
Activation
ConvT
50
4
1 x
1
1024
x x 1024
ReLU
ConvT
4 x 1024
4 4x4
2 2
512
8 8 8 x 512
ReLU
ConvT
8 8 x 512
x
2 2
256
16 x 16 x 256
ReLU
ConvT
16 x 16 256
4 x
2 2
128
32 32 128
ReL.U
ConvT
32 32 128
4
2 2
64 64 64
ReL.U
Conv
64 64 x
4
1
64 C
Sigmoid
11

---

Published as a conference paper at ICLR 2020
Table 5: The architecture of our convolutional discriminator. All layers except FC have biases. W,
and C' are set for each task so that the dimensions the imput data are matched.
Layer
Input shape
Filter size
Stride
Outputs
Output shape
Activation
Conv
2" 2d x C
4
2
32
2 x x 32
LeakyReLU
Conv
2 x 2
32
4 4 4
2 2
64
2W 22 x 2 x 64
LeakyReLU
Conv
2 x x2#=2 64
4x4
2 2
128
2 2W x 128
LeakyReLU
Conv
2w x 2 128
4x4
2x2
256
2 x 2 x 256
LeakyReLUU
Conv
2 22 256
4x4
2x2
512
2 2 512
LeakyReLU
FC
2 w5-5, 2 512
1
LeakyReLU
Table The architecture of our U-Net. The height H and number of input channels depends on the experiment. MP is maxpooling with stride 2. FC has noise as input. UpConv performs transposed convolution with stride 2. Concat concatenates the current feature map with one from the downstream path. The final output is computed depending the task (see text for more details)
Layer
Input (shape)
Outputs
Output shape
DoubleConv
2w x 128 x C
32
2W x 128 x 32
MP1
2" x 128 32
2
32
DoubleComv2
2W x 64 x 32
64
2W x 64 x 64
MP2
2
64
64
2
64
DoubleComv:
2W-2 x 32 x 64
64
2W-2 x 32 x 128
MP3
2 32 128
128
2 16 128
DoubleConv4
2 2W 16 x 128
256
2 2W 16 x 256
MP4
2 2W 16 256
256
2W 8 256
DoubleConv5
2W x 8 x 256
256
2 x 256
FC
50 DoubleConv5
2W 16
2W x 2
Concat
2 -
UpConv
2 x 8 x 258
256
2W x 16 x 258
Concat
DoubleConv4 2 x 16 x 514
514
2 16 x 514
Conv
128
2W 16 x 128
UpConv
2 W-3 x 16 x 128
128
2W-2 x 32 x 128
Concat
DoubleConv3
256
2W 32 256
Conv
2 W-2 32 x 256
64
2W x 32 x 64
UpConv
2 2W-2 x 32 x 61
64
2
64 x 64
Concat
DoubleConv2 2 2 64 x 128
2 641 x 128
Conv
32
2 x 64 x 32
UpConv
2 2 64 32
32
2W x 128 x 32
Concat
DoubleConvl 2 x 128 x 64
64
2W x 128 x 64
Conv
32
2W x 128 x 32
Conv
2W 128 32
2
128
Table 7: The DoubleConv neural network block used in the U-Net. Conv uses a x 3 filter size.
Layer
Input shape
Outputs
Output shape
Conv
W x
x x
BatchNorm & ReLU
W X H
W X H
Conv
W X X
X H
X
BatchNorm & ReLU
W x H
W X H
12

---

Published as a conference paper at ICLR 2020
Ascompanimens
500
Muee SDR
Modk
GAN
FacGAV
Songs
Songs
Figure 6: GAN and FactorGAN separation performance for different numbers of paired samples
A.2 AUDIO SOURCE SEPARATION EXPERIMENT
For our audio source separation experiment, our generator G. takes music spectrogram along with noise z and maps it to an estimate of the accompaniment and vocal spectra a and v, implicitly defining output probability (a, v v/m). We define the joint real and generated distributions that should be matched as p( m, a, v) and q(m, a, v) = 4s (a, v ()mpp(m) Since the source signals in our dataset are simply added the time-domain spectrogram as well, so we assume that p(m|a, a, v) 6(m a v). We can constrain our generator G to make predictions that always satisfy this condition, thereby taking care of the input-output relationship manually, similarly to Sonderby et al. (2017). Instead of predicting the sources directly, mask b with values in the range [0, 1 computed, and the accompaniment and vocals are estimated as b @ and (b- 1) @ respectively. As result, qmmaa, v) = p(m a, v), so we can simplify the joint density ratio to
produce the mixture, this approximately applies the
(m,, a, v) p(a, v)(mma, v) p(a, v) ca v) p(a) p(v) (m, a, q(a, ())
(6)
v(a.v) (a v) ((a) q(v)*
meaning that the discriminator(s) the GAN and the FactorGAN only require (a. v) pairs, but not the mixture additional input, the correct input-output relationship is already incorporated into the generator. Furthermore, the last equality suggests FactorGAN application with one marginal discriminator for each source along with dependency discriminators to model source dependencies.
Dataset We use MUSDB (Rafi et al. (017) as multi-track dataset for our experiment, featuring 100 songs for training and 50 songs for testing. Each song is downsampled 22.05 KHz before spectrogram magnitudes are computed, using an STFT with a 512-sample window and a 256-sample ho Snippets with 128 timeframes each are created by cropping each song's full spectrogram at regular intervals of 64 timeframes. Thus, the generator only separates snippets m E _ 256xx228 and outputs predictions of the same shape, however this does not change the derivation presented in Equation (6), and longer inputs at test time can processed by partitioning them into snippets and concatenating the model predictions.
For our generator, we use the U-Net architecture detailed in Table with W 8 and C 1. We use the convolutional discriminator described in Table with W 8, H and C 1. The source dependency discriminators take two sources as input via concatenation along the channel dimension, they use 2.
Experimental setup
In each experiment, we vary the number of training songs whose snippets are available for paired training between 10, 20 and 50 and compare between GAN and FactorGAN. The spectrograms predicted on the test set are converted to audio with the inverse STFT by reusing the phase from the mixture, and then evaluated using the iignal-to-distortio ratio (SDR), a sell-establissed evaluation metric for source separation (Vincent al. 2006).
Results Figure 6 shows our separation results. Compared to a GAN, the separation performance significantly higher using FactorGAN. As expected, FactorGAN improves slightly with paired examples, which is not the case for the GAN here we find that the vocal output becomes too quiet
Thii results in 257 frequency bins but discard the bin with the highest frequency to obtain power of and thus avoid padding issues in network architectures.
13

---

Published as a conference paper at ICLR 2020
when increasing the number of songs for training, possibly sign of mode collapse. Similarly the results seen the image pair generation experiments, we suspect that the FactorGAN discriminator might approximate the joint density (xx) more closely than the GAN discriminator due to its use of multiple discriminators, although the reasons for this not yet understood.
A.3 POSSIBLE EXTENSIONS
We can decompose the joint density ratio Px(x)
in other ways than shown in Equation in the paper.
(x(x)
In the following, we discuss two additional possibilities.
A.3.1 HIERARCHICAL FACTORGAN
The decomposition of the joint density ratio could be applied recursively, splitting the obtained marginals further into "sub-marginals" and their dependencies, which could be repeated multiple times. In addition to training with incomplete observations where only a single part given, this also allows making use of samples where only sub-parts of these parts are given and is thus more flexible than single factorisation as used in the standard FactorGAN.
As demonstration, we split each marginal x" further into a group of J; marginals, J; S [D;], and their dependencies, without further recursion for simplicity:
(x) (x(x)
(x(x)
(x)))
II
II
4- (x) cQ(x)
The
(x*)
cQ
(x)
(x
dp and o are dependency terms analogously to ep and eq. but only defined on marginal variable x whose J "sub-marginals" are denoted by x x
Such hierarchical decomposition might also be beneficial if the data is known be generated from a hierarchical process. We leave the empirical exploration of this concept to future work.
A.3.2 AUTOREGRESSIVE FACTORGAN
For a multi-dimensional variable [x' x2 ...., x" composed of T elements arranged in sequence, such as time series data, the joint density ratio can also be decomposed in a causal, auto-regressive fashion:
(x)
(xx')
Cp ,..,,")))) (x()
4n (x) (x!
cQ (x] ,..., x)) O: (x*)
(xc)
Pa (x; X
(xx))
Oz [x, X1 ;
Note that cp is defined here as an auto-regressive version of FactorGAN in which the generator output quality at each time-step
p(x)
)x(]) (cQ analogously using qz). Equation g suggests
p(xt,......
i is evaluated using a marginal discriminator that estimates px(x') combined with dependency discriminators that model the dependency between the current and all past time-steps.
(x(x))
The final product formulation Equation 9 reveals a close similarity to auto rugo-regressive models and suggests modification of the normal GAN with an auto-regressive discriminator that rates an input at each time-step given the previous ones. Using a derivation analogous to the one shown in Section A.4 this implies taking the unnormalised discriminator outputs at each time-step, summing them, and applying sigmoid non- non-linearity to obtain the overall estimate of the probability Dxx). A similar implementation was used before in Mogren| (2016), attempting to stabilise GAN training with recurrent neural networks as discriminators, but for the first time, we provide rigorous theoretical justification for this practice here.
A.4 DISCRIMINATOR COMBINATION
Definition A.1. Sigmoid discriminator output. Let Do, (x') (do, (x!)) do, :RP.. + R for all i (1,..., K}. analogously define Dj, (x) and Do (x).
14

---

Published as a conference paper at ICLR 2020
Definition A.2. Combined discriminator. Let D (xx == o(dip (x) - dog (x)
L== do, (x')) be
K
the output of the combined discriminator that is used for training Go using Equation
Theorem 1. Combined discriminator approximates D(x). Under definitions
A.1 and A.2 and
assuming optimally trained sub-discriminators, D D(x)
(x
Po (x)
Proof. Proof of Theorem
[siin Definitions
A. and A.2
D (x)
K
dop (x) - dg0 (x) +
do, (x))
=o
K IIe
-1
1
dp
(x)
e
do;
(x+t)
i=1
1+ 1 - D (x)
Pa
(x)
K
Do. (x²)
D& (x)
D (x)
Do. (x))
(10)
+ II KK Pe (x(x))
(x(
K
(x*))
Ps (x)
Il Ih (xx)
Per (x')
2=1
yz (x)
-1
(x)
Pn (x)
Pr (x) + (x)
15

---

Published as a conference paper at ICLR 2020
A.5
GENERATED EXAMPLES
A.5.1
PAIRED MNIST
3
8
3
7
3
7
2
3
3
7
G
3 B
8 /''
8
3 3
7
7
5 5
C
7
7
GAN
8
7
S
O
6 /
5
7 7
8
^ Z
7
6
S
6
7
6
4
>
O
6
8
0
9
3
5
44
5
7
5
7
5
3
7
9
3 3
9
3
2
O
2
7
7
3
depGAN
2
!
5
3
2
2
7
G+
7
//-
4 7
9
O
8 8
8 1
/ 1
3
!
3
3
5
G
7
4
L/
2
S
3
O
4
7
6
9
3
9
0
40
7
9
5
100
500
20000
Figure 7: Paired MNIST examples generated by GAN and FactorGAN for different number of paired training samples, using À 0.9.
16

---

Published as a conference paper at ICLR 2020
A.5.2
IMAGE PAIRS
Figure 8: GAN generating image pairs for the Cityscapes dataset using 100 paired samples.
Figure 9: GAN (big) generating image pairs for the Cityscapes dataset using 100 paired samples.
17

---

Published as a conference paper at ICLR 2020
Figure 10: FactorGAN generating image pairs for the Cityscapes dataset using 100 paired samples.
Figure 11: GAN generating image pairs for the Cityscapes dataset using 1000 paired samples.
18

---

Published as a conference paper at ICLR 2020
Figure 12: GAN (big) generating image pairs for the Cityscapes dataset using 1000 paired samples.
Figure 13: FactorGAN generating image pairs for the Cityscapes dataset using 1000 paired samples.
19

---

Published as a conference paper at ICLR 2020
Figure 14: GAN generating image pairs using the full Cityscapes dataset.
Figure 15: GAN (big) generating image pairs using the full Cityscapes dataset.
20

---

Published as a conference paper at ICLR 2020
Figure 16: FactorGAN generating image pairs using the full Cityscapes dataset.
(a) GAN
(b) FactorGAN
Figure 17: Image pairs generated for the Edges2Shoes dataset using 100 paired samples.
GAN
FactorGAN
Figure 18: Image pairs generated for the Edges2Shoes dataset using 1000 paired samples.
21

---

Published as a conference paper at ICLR 2020
(a) GAN
(b) FactorGAN
Figure 19: Image pairs generated for the Edges2Shoes dataset using all samples paired.
22

---

Published as a conference paper at ICLR 2020
A.6 IMAGE SEGMENTATION
(a) GAN
(b) FactorGAN
Figure 20: Segmentation predictions made on the Cityscapes dataset for the same set of test inputs, compared between models, using 100 paired samples for training
(a) GAN
(b) FactorGAN
Figure 21: Segmentation predictions made on the Cityscapes dataset for the same set of test inputs, compared between models, using 1000 paired samples for training
23

---

Published as a conference paper at ICLR 2020
(a) GAN
(b) FactorGAN
Figure 22: Segmentation predictions made on the Cityscapes dataset for the same set of test inputs, compared between models, using all paired samples for training
Figure 23: CycleGAN generating image pairs for the Cityscapes dataset without any paired samples.
24


Reviews:
[{"review_id": "rkxZuQfJcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present FactorGANs, which handle missing data scenarios by constructing conditional marginal estimates from ratios of joint and marginal distributions, estimated with GANs. FactorGANs are applied to the problem of semi-supervised (paired+unpaired) translation and demonstrate good performance.\n\nStrengths:\n-Nice formulation, which I believe is novel. Well written, good initial results.\n\nLimitations:\n-The most serious limitation of the paper is that the technique is not compared with any other semi-supervised methods, such as the Augmented CycleGAN. Because of this it is not clear how the technique compares with SOTA, and so the significance of the paper is not clear.\n-The approach scales linearly with the number of marginals, which may limit its applicability to more general imputation tasks.\n-The title is the same as an the arxiv paper title, and so the double-blind requirement is trivially violated.\n\nOverall:\n\nA nice formulation, but weak experimental investigations (no comparisons to SOTA semi-supervised translation) make the significance of the paper unclear. This makes it a borderline paper. I strongly encourage the authors to update their experiments accordingly.\n\nPost Response:\n\nThank you to the authors for the detailed response and additional experimentation. I have updated my rating. It is a nice formulation, and the experimental validation of the technique has been strengthened. The additional experiments (i.e. comparing to the augmented cyclegan) that the authors are following through on will further improve the paper, making it a clear accept.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SygA0eq1qr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found this paper very easy and clear to follow - the authors present, what I believe to be an elegant, approach to training a GAN in the presence of missing data or where many marginal samples might be available but very few complete (e.g. paired) samples. The approach proceeds by identifying that the joint distributions (true and approximate) can be factored so as to yield a number of different density ratios which can then be estimated by specific discriminators; in particular, these include the appropriate marginal density ratios and then corresponding overall correction factors. As a caveat to the review I should point out that while I am familiar with GANs, they are not my main area of expertise so this should be taken into consideration - apologies if there is literature I have missed.\n\n\nExperiments: The authors provide a number of illustrative experiments that demonstrate the efficacy of the approach across a number of tasks. There are many differing GAN models but due to the nature of the problem I don't have a big issue with the majority of the comparisons being against a standard GAN since the tasks are suitably designed. For the paired MNIST experiment I found it hard to assess the qualitative results visually and am always concerned about the ad-hoc nature of Inception Distances - I find it difficult to attribute weight to them quantitatively since they are usually being used to assess things where they might suffer from a common error (e.g. they are both based on NNs). Also, I'm not fully on board with the dependency metric in (5) but then the authors also point out the same concerns. The other experiments I found more convincing.\n\nI appreciated having error bars on some of the plots to help assess significance - would it not be possible to put error bars on all plots?\n\nI found the additional extensions presented in the appendix to be interesting ideas as well and would be interested to see how the approach works with other GAN objectives as mentioned for future work.\n\nI am mostly very positive about this work - my main concern is really common to most GANs - all the analysis relies on the premise that the discriminators can be setup as good estimators for the density ratios. We know that this is not always the case since everything comes from samples and if the capacities of each of the discriminators are not set appropriately then I would expect problems to occur - has this been explored by the authors? It would be no detriment to the work to include failure examples where the authors purposefully make use of inappropriate architectures for some of the discriminators to check for this? For example, there will be large imbalances in the number of training samples used for the different discriminators - how does this affect stability?\n\n\nOther notes:\n\n- Whilst I understand the point about independent marginals in 2.4 I'm not sure I see the motivation as clearly since it seems that the model is much more useful when there is dependent information but maybe there's a use-case I'm not thinking of?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Ske4xC31ir", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is tackling the problem of training generative adversarial networks with incomplete data points. The problem appears to be important for semi-supervised training of image to image translation models, where we may have a lot of observations in both domains, but a little annotated correspondences between the domains.\n\nThe solution proposed by the authors involves an observation that discriminator in GANs is estimating the density ratio between real and fake distributions. This ratio can then be decomposed into a product of marginal density ratios, with two additional multipliers, corresponding to density ratios between a joint real/fake distribution and a product of its marginals. The authors then use discriminators to approximate all the ratios, which allows them to facilitate semi-supervised training.\n\nMy decision is \"weak accept\".\n\nIt is not clear to me to what extent does the proposed model outperform the regular CycleGAN on a large amount of paired training samples due to architectural changes (including spectral normalization).\n\nAlso, it would be nice if the comparison was carried out with a newer, possibly SotA models for unpaired image-to-image translation (MUNIT, FUNIT, BicycleGAN).\n\nMoreover, there are some simple modifications that can be made to a standard CycleGAN/Pix2pix training pipeline that would facilitate the small number of annotations (for example, see \"Learning image-to-image translation using paired and unpaired training samples\").\n\nIt is hard to evaluate the comparative performance of the method without the comparisons mentioned above.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers appreciate the new method (FactorGAN) for training generative networks from incomplete observations. At the same time, the quality of the experimental results can still be improved. On balance, the paper will make a good poster.
Decision: Accept (Poster)