Conditional Invertible Neural Networks for Guided Image Generation
Authors: Lynton Ardizzone
Carsten Lüth
Jakob Kruse
Carsten Rother
Ullrich Köthe
Venue:  (2020)
Keywords: Invertible neural networks
generative models
conditional generation

Abstract: In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). It combines the purely generative INN model with an unconstrained feed-forward network, which efficiently pre-processes the conditioning input into useful features.  All parameters of a cINN are jointly optimized with a stable, maximum likelihood-based training procedure. Even though INNs and other normalizing flow models have received very little attention in the literature in contrast to GANs, we find that cINNs can achieve comparable quality, with some remarkable properties absent in cGANs, e.g.  apparent immunity to mode collapse. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

Under review as a conference paper at ICLR 2020
CONDITIONAL INVERTIBLE NEURAL NETWORKS FOR GUIDED IMAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we address the task of natural image generation guided by condi- tioning input. We introduce a new architecture called conditional invertible neural network (cINN). It combines the purely generative INN model with an uncon- strained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of cINN are jointly optimized with a stable, maximum likelihood- other normalizing flow models have received very little attention in the literature in contrast to GANs, we find that cINNS can achieve comparable quality, with some remarkable properties absent in cGANs, e.g. apparent immunity mode collapse. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style an intuitive way.
training procedure. Even though INNs and
INTRODUCTION
Generative adversarial networks (GANs) produce ever larger and more realistic samples (Kar- et al., 2017; Brock et al., 2019). Hence they have become the primary choice for majority of image generation tasks. As such, their conditional variants (cGANs) would ap- pear to be the natural tool for conditional image generation as well, and they have suc- cessfully been applied in many scenarios (Ledig et al., 2017; Miyato & Koyama, 2018). However, lack diversity especially common when the condition itself is an image, and special precautions have to be taken to avoid
mode collapse and training stability continues to pose challenge.
Conditional variational autoencoders (cVAEs) do not suffer from the same problems. Training is generally stable, and since every data point is assigned region in latent space, sampling yields the full variety of data seen during train- ing. However cVAES come with drawbacks of their own: The assumption of Gaussian pos- terior on the decoder side implies L2 recon- struction loss, which is known to cause blur- riness. In addition, the partition of the latent space into diagonal Gaussians leads to either mode-mixing issues or regions of poor sample quality (Kingma et al., 2016). There has also been some success in combining aspects both approaches for certain tasks, such as (Isola et al., 2017: Zhu et al., 201 2017b; Park et al., 2019).
Figure 1: Diverse colorizations, which our network created for the same grayscale image. One of them shows ground truth colors, but which? Solution at the bottom next page.
We propose third approach, by extending In- vertible Neural Networks (INNs, Dinh et al.
Cooee available, but held back for the review process due anonymity.

---

Under review as a conference paper at ICLR 2020
(2016); Kingma & Dhariwal (2018); Ardizzone al. (2019)) for the task of conditional image generation, by adding conditioning inputs to their core building blocks. INNs are neural networks which are by construction bijective, efficiently invertible, and have tractable Jacobian determinant. They represent transport maps between the input distribution p(x) and a prescribed, easy-to-sample- from latent distribution p(z). During training. the likelihood of training samples from p(x) is maximized in latent space, while at inference time, z-samples can trivially be transformed back to the data domain. Previously, INNs have been used successfully for unconditional image generation, e.g. by Dinh et al. (2016) and Kingma & Dhariwal (2018).
Unconditional INN training is related that of VAEs, but it compensates for some key disadvantages: Firstly, since reconstructions are perfect by design, no reconstruction loss is needed, and generated images do not become blurry. Secondly, each x maps to exactly one z in latent space, and there no need for posteriors p(z x). This avoids the VAE problem of disjoint or overlapping regions in latent space. In terms of training stability and sample diversity, INNs show the same strengths as autoencoder architectures, but with superior image quality. We find that these positive aspects apply to conditional INNs (cINNs) as well.
One limitation of INNs is that their design restricts the use of some standard components of neural networks, such as pooling and batch normalization layers. Our conditional architecture alleviates this problem, the conditional inputs can be preprocessed by a conditioning network with standard feed-forward architecture, which can be learned jointly with the cINN to greatly improve its generative capabilities. We demonstrate the quality of cINNS for conditional image generation and uncover emergent properties of the latent space, for the tasks of conditional MNIST generation and diverse colorization of ImageNet.
Given this, we believe that the cINN architecture brings the research field of INNs and other normalizing flow models a substantial step forward. Despite the fact that INNs have received very little attention in the literature in contrast to GANs, we find that cINNs can achieve comparable quality cGANs, with some remarkable properties absent in eGANs. This includes diverse outputs by default, compared to sparse support or mode collapse in cGANs, easier explainability and interpretability of the learned latent representation, as well as simple and intuitive manipulation and interpolation of generated or existing images.
Our work makes the following contributions:
We propose new architecture called conditional invertible neural network (cINN), which combines an INN with an unconstrained forwwfd network for conditioning. It gener- ates diverse images with high realism on par with existing approaces, while adding some noteworthy and useful properties.
We demonstrate a stable, maximum likelihood- based training procedure for jointly optimiz-
ing the parameters of the INN and the conditioning network.
We take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space. We illustrate this for MNIST digit generation and image colorization.
RELATED WORK
Conditional Generative Modeling. Modern generative models learn to transform noise (usually sampled from multivariate Gaussians) into desired target distributions. Methods differ by the model- family these transformations are picked from and by the losses determining optimal solutions.
Conditional generative adversarial networks (cGANs, Mirza & Osindero (2014)) train a pair of neural networks: a generator transforms a pair of conditioning and noise vectors to images, and a discriminator penalizes unrealistic looking images. The conditioning information is either concate- nated to the noise (Mirza & Osindero, 2014), or fed into the network via conditional batch-norm layers (Dumoulin et al., 2017; Huang & Belongie, 2017; Park et al., 2019). Ensuring diversity of the generated images (for fixed conditioning) appears to be challenging in this approach. Recent BigGANs (Brock et al., 2019) successfully address this problem by using very large networks and batch sizes, but require parallel training on up to 512 TPUs. PacGANs (Lin et al., 2018) employ augmented discriminators, which evaluate entire batches of real generated images together rather than one image at a time. CausalGANs (Kocaoglu et al., 2017) train two additional discriminator networks, called "labeler" and "anti-labeler", with the latter explicitly penalizing the lack of diversity.
Quiz oonuionr Chnnne tow
ccnte
2

---

Under review as a conference paper at ICLR 2020
Pix2pix (Isola et al., 2017) addresses the important special case when the target is conditioned on an image in a different modality, c.g. to generate satellite images from maps. In addition to the discriminator loss, it minimizes the L1 distance between generated and ground-truth targets using a paired training set, which contains corresponding images from both modalities. This leads to impressive image quality, but lack of diversity seems to be especially hard problem in this case. In contrast, our method does not need explicit precautions to promote diversity.
Bidirectional architectures augment generator networks with complementary encoder networks that learn the generator's ' inverse and enable reconstruction losses, which exploit cycle consistency require- ments. Conditional variational autoencoders VAV,,, Sohn et al. (2015)) replace all distributions in a standard VAE (Kingma & Welling, 2013) by the appropriate conditional distributions, and are trained to minimize the evidence lower bound (ELBO loss). Since variational distributions are typically Gaussian, the reconstruction penalty is equivalent to squared loss, resulting in rather blurry generated images. This is avoided by AGE networks (Ulyanov et al., 2018) and CycleGANs (Zhu et al., 7a), which combine standard cGAN discriminators with L1 reconstruction loss in the data domain, and bidirectional conditional GANs (Jaiswal et al., 2017), which extend the GAN discriminator to act on the distributions in data and latent space jointly. SPADE (Park al., 2019), building upon pix2pix and pix2pixHD (Wang et al., 2018), augments cGANs with additional VAE encoders to shape the latent space such that diversity is ensured.
Instead of enforcing bijectivity through cycle losses, invertible neural networks are bidirectional by design, since encoder and generator realized by forward and backward processing within single bijective model. We focus on architectures whose forward and backward pass require the same computational effort. The coupling layer designs pioneered by NICE (Dinh al., 2014) and RealNVP (Dinh et al., 2016) emerged as very powerful and flexible model families under this restriction. Using additive coupling layers, i-RevNets (Jacobsen et al., 2018b) demonstrated that the lack of information reduction from data space to latent space does not cause overfitting. The Glow architecture (Kingma & Dhariwal, 2018) combines affine coupling layers with invertible 1x convolutions and achieves impressive attribute manipulations (e.g. age, hair color) in generated faces images. This approach was recently generalized to video by Kumar et al. (2019).
Thanks to tractable Jacobian determinants, the coupling layer architecture enables maximum likeli- hood training (Dinh et al., 2014; 2016), but experimental comparisons with other training methods inconclusive so far. For instance. Danihelka et al. (2017) found minimization of an adversarial loss to be superior to maximum likelihood training in RealNVPs, Schirmeesstr et al. (2018) trained i-RevNets in the same manner adversarial auto-encoders, i.e. with a discriminator acting in latent rather than data space, and Flow-GANs (Grover et al., 2018) performed best using bidirectional training, a combination of maximum likelihood and adversarial loss. On the other hand, maximum likelihood training worked well within Glow (Kingma & Dhariwal, 2018), and i-ResNets (Behrmann et al., 2018) could even be trained with approximated Jacobian determinants. In this work we reinforce the view that high-quality generative models can be trained by maximum likelihood loss alone. To the best of our knowledge, we are the first to apply the coupling layer design for conditional generative models, with the exception of Ardizzone et al. (2019), who use it to compute posteriors for (relatively small) inverse problems, but do not consider image generation. While Liu et al. (2019) recently proposed an effective post-hoc conditioning scheme for flow-based generative models, their method requires an additional cGAN to identify latent codes fulfilling the specified condition.
Colorization. State-of-the-art regression models for colorization produce visually pear-perfect images (lizuka et al., 2016), but do not account for the ambiguity inherent in this inverse problem. To address this, models would ideally define conditional distribution of plausible color images for given grayscale input, instead of just returning a single "best" solution.
Popular existing approaches for diverse colorization predict per-pixel color histograms from a U-Net (Zhang et al., 2016) or from hypercolumns of an adapted VGG network (I (Larsson et al., 2016). However, sampling from these local histograms independently can not lead spatially consistent colorization, requiring additional heuristic post-processing steps to avoid artefacts.
In terms of generative models, both VAES (Deshpande al., 2017) and cGANs (Isola al., 2017; Cao et al., 2017) have been proposed for the task. However, their solutions do not reach the quality of the regression-bee modifications and extensions to generative approaches have been developed, such as auto-regressive
models, and cGANs in particular often lack diversity. To compensate.
3

---

Under review as a conference paper at ICLR 2020
models (Guadarrama et al., 2017) and CRFs (Royer et al., 2017). However, these methods computationally very expensive and often unable to scale to realistic image sizes.
Conceptually closest to our proposed method is the work of (Ulyanov al., 2018), where encoder network maps color information to latent space and a generator network learns the inverse transform, both conditioned on the grayscale image. Their experiments however limited data set with only cars, and just three latent dimensions, leading to global, but no local diversity.
In contrast to the above, forward pass. It models the distribution of all pixels jointly, and allows for meaningful latent space manipulations.
flow-based cINN generates diverse colorizations in one standard feed-
METHOD
Our method an extension of the affine coupling block architecture established by Dinh et al. (2016). There, each network block splits its input into two parts [nj u2] and applies affine transformations between them that have strictly upper or lower triangular Jacobians:
V1= = 41 O exp (#1 (u2)) + +42)
V2 = U2 O exp (v2() + 12((1)
The outputs [Vi, V2] are concatenated again and passed to the next coupling block. The internal functions 8, and ij can be represented by arbitrary neural networks, and are only ever evaluated in the forward direction, even when the coupling block is inverted:
U2 (v2 tv(v1)) Ø exp (s2(v1))
(2)
U1 (v1 t (u2)) 2 exp (s1 (u2))
As shown by Dinh et (2016), the logarithm of the Jacobian determinant for such a coupling block is simply the sum of and 82 over image dimensions.
3.1 CONDITIONAL INVERTIBLE TRANSFORMATIONS
We adapt the design of Eqs. (1) and (2) to produce a conditional version of the coupling block. Because the subnetworks sy and l are never inverted, we can concatenate conditioning data c their inputs without losing the invertibility, replacing $1 (uz) with (u2 c) etc. Our conditional coupling block design illustrated in Fig. 2.
In general, we will refer to a cINN with network param- eters 9 as f(x; c, f), and the inverse as g(z; c, 0). For any fixed condition c, the invertibility is given
CC
(-;c,; = gc 0).
(3)
Figure 2: One conditional affine cou- pling block (CC).
3.2 MAXIMUM LIKELIHOOD TRAINING OF CINNS
By prescribing a probability distribution pz on latent space Z, the model f assigns any input a probability, dependent on both the network parameters 0 and the conditioning c, through the change-of-variibh formula:
Px(x;c,0) = pz (f(x; ()) det (af(xc))
(4)
0x
Here, we use the Jacobian matrix w.r.t. 0f(x; c) We will denote the determinant of the Jacobian, evaluated at some training sample x, as J; = det(Of/ūxxl.. the posterior over model parameters as p(0: x, c) xxx(x: 0) po (0). Our goal is to find network parameters that maximize its logarithm, i.c. we minimize the loss
Bayes' theorem gives us
L E, | log (px c, 0)) log (0), which is the same in classical Bayesian model fitting.
Inserting Eq. (4) with a standard normal distribution for pz (z), as well as a Gaussian prior on the weights 0 with 1/20 т, we obtain
[x c. 0)
C= E,
log [.1 + -110122
(6)
2
4

---

Under review as a conference paper at ICLR 2020
The latter term represents L2 weight regularization, while the former is the maximum likelihood loss. Training a network with this loss yields an estimate of the maximum likelihood network parameters MLL From there, we can perform conditional generation for a fixed e by sampling z and using the inverted network g: Xgen = (zz 0ML), with z Pz(z).
Training with the maximum likelihood method makes it virtually impossible for mode collapse to occur: If any mode in the training set has low probability under the current guess px (x; c, d). the corresponding latent vectors will lie far outside the normal distribution pz and receive big loss from the first 1.2-term in Eq. (6). In contrast, the discriminator of GAN only supplies weak signal, proportional to the mode's ' relative frequency in the training data, so that the generator is not penalized much for ignoring mode completely.
3.3 CONDITIONING NETWORK
In complex settings, expect that higher-level features of c need to be extracted for the conditioning to be effective, e.g. global semantic information from an image as in Section 4.2. In such cases, feeding the condition directly into the cINN would place an unreasonable burden on the and t networks, as higher-level features would have to be re-learned in each coupling block.
To address this issue, we introduce an additional ferdar conditioning network h, which transforms the condition e. to some intermediate representation è = h(c)., and replace c; in Eq. (6) with 6 h(c.). The network h can be pretrained, e.g. by using features from VGG architecture trained for image classification. Alternatively or additionally, / can be trained jointly with the cINN by propagating gradients from the maximum likelihood loss through the conditioning In this case, the conditioning network will learn to extract features which are particularly useful for embedding the cINN inputs x into latent variables z.
3.4 IMPORTANT DETAILS
For cINNs to match the performance of well-established architectures for conditional generation, we introduce a number of minor modifications and adjustments to the architecture and training procedure. With these adaptions, our training setup is very stable and converges every time.
Noise as data augmentation. We add a small amount of noise to the inputs as part of the standard data augmentation. This helps to smooth out quantization artifacts in the input, and prevents sparse gradients when large parts of the image are completely flat (as e.g. in MNIST).
Soft clamping of scale coefficients. We apply an additional nonlinear function to the scale co- efficients of the form Selamp (2a/ n Jarctanss/o). which yields Sclamp for |s| < and Sclamp R =o for |s| >> a. This prevents any instabilities stemming from exploding magnitude of the exponential exp(Sclamp) We find a 1.5 work well. A similar scheme is used in the realNVP architecture (Dinh et al., 2016), whereby a is a learned vector, instead of a fixed hyperparameter, We find that fixed a does not limit performance, while completely ruling out problems from exploding or vanishing scaling factors.
Initialization. Heuristically, we find that Xavier initialization (Glorot & Bengio, 2010) leads to stable training from the start. We experienced training instability when initial parameter values were too high. Similar to Kingma & Dhariwal (2018), we also initialize the last convolution in all and subnetworks to zero, so training starts from an identity transform.
Soft channel permutations. We use random orthogonal matrices to mix the information between the channels. This allows for more interaction between the two information streams u1 in the coupling blocks. similar technique was used by Kingma & Dhariwal (2018), but our matrices stay fixed throughout training and are guaranteed to be cheaply invertible.
Haar wavelet downsampling, All prior INN architectures use checkerboard patterns for reshaping to lower spatial resolutions. However, this possibly results in unnatural inputs to the following layers, where previously neighboring pixels are suddenly treated independently as distinct channels. We argue, supported by our ablation experiments, that deliberately chosen spatial decomposition can be more natural, and more closely resemble the effect of pooling operations in feed-forward networks. For these reasons, we perform downsampling with Haar wavelets (Haar, 1910), which essentially
5

---

Under review as a conference paper at ICLR 2020
as one-hot vodto
2
0 0
4
zwargee
deginal
Lub
Figure 3: Haar wavelet downsampling reduces spatial dimensions & separates lower frequency content (a) from higher frequencies (h,v,d).
Figure 4: cINN model for conditional MNIST generation.
0
0
0
1
0
000000
0
0
0
1
/
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3 3
3
3
3
3
3
3
3
3 4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
Latent
wishh
Latent line thecnness
Latert axts start
Figure 5: Axes in our MNIST model's latent space, which linearly encode the style attributes width, thickness and slant.
decompose images into 2 x 2 average pooling channel as well vertical, horizontal and diagonal derivatives, see Fig. 3. This also motivates which channels should be split off to directly become latent variables, namely the derivative channels with high-resolution information, and which should be transformed further in later stages of the cINN, i.c. the pooling channels (see Sec. 4.2), The wavelet transform also contributes to mixing the variables between layers, complementing the soft permutations. Similarly. Jacobsen et al. (2018a) 8a) use global discrete cosine transform as a final transformation their INN, to replace global average pooling.
Discussion. The soft clamping, along with the Haar wavelet downsampling are critical for training stability at sufficiently high learning rates. With them, the network is exceptionally stable, and converges to good optimum every time. The noise augmentation and initialization additionally increase the convergence speed. The permutations are necessary achieve sufficient expressive power and high quality results, but do not affect the training stability speed. Ablation curves for these improvements are given in the appendix.
4 EXPERIMENTS
We present results and explore the latent space of our models for two conditional image generation tasks: MNIST digit generation and image colorization.
4.1 CLASS-CONDITIONAL GENERATION FOR MNIST
As a first experiment, we perform simple conditondii generation of MNIST digits. We construct a cINN of 24 coupling blocks using fully connected subnetworks and f, which receive the conditioning directly as a one-hot vector (Fig. 4). No conditioning network h is used. For data augmentation we 0.02), as described Section 3.4.
only add a small amount of noise the images
Samples generated by the model are shown in Fig. 6. We find that the cINN learns latent representa- tions that are shared across conditions c. Keeping the latent vector z fixed while varying e produces different digits in the same style. This property, in conjunction with our network's 's invertibility, can directly be used for style transfer, as demonstrated Fig. This outcome not obvious the trained cINN could also decompose into 10 essentially separate subnetworks, one for each condition. In this case, the latent space of each class would be structured differently, and inter-class transfer of latent vectors would be meaningless. The structure of the latent space is further illustrated in Fig. 5, where we identify three latent axes with interpretable meanings. Note that while the latent space is learned without supervision, we found these axes in semi-automatic fashion: We perform PCA on the latent vectors of the test set, without the noise augmentation. Since disentanglement is only unique up to rotation in latent space, the directions corresponding to human interprrreeee properties do not necessarily align with the latent axes. We therefore manually find a set of orthogonal vectors in the subspace of the first four PCA components which represent such properties.

---

Figure 8: cINN model for diverse colorization. The conditioning network consists of a truncated VGG (Simonyan & Zisserman, 2014) pretrained to colorize ImageNet, with separate convolutional heads h, h2, h3.... tailoring the extracted features to each individual conditional coupling block (CC). After each group of coupling blocks, we apply Haar wavelet downsampling (Fig. 3) to reduce Under review as a conference paper at ICLR 2020
Tidy
0
1
2
3
4
5
6
7
8
9
9
0
2
3
4
5
7
8
Slanted, narrow
0
/
2
3
4
5
6
7
8
9
9
0
3
3
4
5
6
7
Slanted left, wide
0
1 2,
3
4
5
6
7
8
9
4
0
2
3
4
5
6
7
8
Messy
0
/ 2
3
4
5
6
?
0 8
9
9
0
3
4
6
7
Faint
{ 1
3
4
5
6
7
0
9
0
/
2
3
9
5 6 7 8
Bold
0
2
3
4
3
6
7
8
9
Outputs
Figure 6: MNIST samples from our cINN conditioned
Figure To perform style transfer, we determine the latent code z [xx c, 0) of a test image (left), then use the inverse network 9 f with different condi- tions to generate the other digits in the same style, & = (2;;,))
on digit labels. All ten digits within one row (O.....9) 99
were generated using the same latent code z, but chang- ing condition We see that each encodes single style consistently across digits, while varying z between rows leads to strong differences in writing style.
h
CC
CC
CC
2
the spatial dimensions and, where indicated by arrows, split off parts of the latent code z early.
4.2 DIVERSE IMAGE COLORIZATION
For more challenging task, turn colorization of natural images. The common approach for this task is to represent images in Lab color space and generate color channels a, b by model conditioned on the luminance channel L.
We train on the ImageNet dataset (Russakovsky et al., 2015), again adding low noise to the a, channels (a 0.05). As the color channels do require as much resolution the luminance channel, we condition on 256 x 256 pixel grayscale images. but generate 64 x 64 pixel color information. This is in accordance with the majority of existing colorization methods.
As with most generative INN architectures, we do not keep the resolution and channels fixed throughout the network, for the sake of computational cost. Instead, we use resolution stages, as illustrated in Fig. 8. At each stage, the data is reshaped to lower resolution and more channels, after which a fraction of the channels are split off as one part of the latent code. As the high resolution stages have smaller receptive field and less expressive power, the corresponding parts of the latent vector encode local structures and noise. More global information is passed on to the lower resolution sections of the cINN.
For the conditioning network h, we start with the same VGG-like architecture from Zhang et al. (2016) and pretrain the colorization task. By cutting off the network before the second-to-last convolution, we extract 256 feature maps of size 64 x 64 from the grayscale image L. We then add independent heads on top of this for each conditional coupling block in the cINN, indicated by small hexagons in Fig. 8. Thus each coupling block receives its own specialized conditioning _(k) c, h4 (h(c:)). Each head consists of up five strided convolutions, depending on its required output resolution, and a batch normalization layer. The ablation study in Fig. 14 confirms that the conditioning network is necessary capture semantic information.
We initially train the cINN and the hg. keeping the parameters of the conditioning network h fixed, for 30 000 iterations. After this, we train both jointly until convergence, for days on Nvidia
7

---

Under review as a conference paper at ICLR 2020
cINN cours)
VAE-MDN
cGAN
PRC
CNN
BW
Ground Inuch
MSF of 8)
1.31
4T86
+0.04
10.75
3.9
F0.09
6:77
LPIPS (best
0.167
+4.001
0.171
+0.001
0.245
+00001
0.147"
+0.003
0.163
+0,001
+0.2
4:
Varianee (LPIPS)
0,1754
10,001
0.0822
10.001
10.0
0.1755
1.00003
FID
25.13
10.30
25.9
10.28
21.11
10.27
Degano kondy
24.95
10.27
30.91
10.27
11.69
10.18
VGG top 5 acc.
85.00
+0.48
85.00
+0.48
84.02
+0.53
$7.44
+1.2
$6.86
#0.41
86.02
#0.43
91.66
+0.43
Table 1: Comparison of conditional generative models for diverse colorization (VAE-MDN: Desh- pande et al. (2017); cGAN: Isola et al. (2017)). We additionally compare to a state-of-the-aa regression model ('CNN', diversity, lizuka et al. (2016)), and the grayscale images alone ('BW)). For each of 5k ImageNet validation images, we compare the best pixel-wise MSE of 8 generated colorization samples, the pixel-wise variance between the 8 samples as an approximation of the diversity, the Fréchet Inception Distance (Heusel et al., 2017) as a measure of realism, and the top 5 accuracy of ImageNet classification performed on the colorized images, to check if semantic content is preserved by the colorization. Note: It was not possible to process all test images with PIC, due to the long inference times. Only 768 images are used. Full results will be included in the camera ready version.
GTXI080 GPUs The Adam optimizer is essential for fast convergence, and we lower the learning rate when the maximum likelihood loss levels off.
At inference time, we use joint bilateral upsampling (Kopf al., 2007) match the resolution of the generated color channels å, b that of the luminance channel L. This produces visually slightly more pleasing edges than bicubic upsampling, but has little to no impact on the results. It was not used the quantitative results table, ensure an unbiased comparison.
Latent space interpolations and color transfer are shown in Figs. 9 and 10, with more experiments in the appendix. In Table a quantitative comparison existing methods is given. While on par with VAE-MDN in other metrics, the cINN clearly has the best sample diversity, as summarized by the variance and best-of-8 accuracy. The cGAN completely ignores the latent code and relies only on the condition - there is no diversity, in line with results from Isola et al. (2017). In terms of FID score, the cGAN performs best, although its results do not appear more realistic to the human eye, cf. Fig. 13. This may be due the fact that FID is sensitive to outliers, which are unavoidable for a truly diverse method (see Fig. 12), or because the discriminator loss implicitly optimizes for the similarity of deep CNN activations. The VGG classification accuracy of colorized images is decreased for all generative methods equally, because occasional outliers may lead to misclassification.
Because the diverse COLORGAN by Cao al. (2017) exclusively trained the LSUN bedrooms dataset (Yu et al., 2015) at small resolution, and would not converge on ImageNet, we compare this separately and provide results in the appendix. In this simple case, the performance of the cINN and COLORGAN is almost equal in all apsects, with much shorter training time for the cINN.
5 CONCLUSION AND OUTLOOK
We have proposed conditional invertible neural network architecture which enables guided genera- tion of diverse images with high realism. For image colorization, we believe that even better results can be achieved when employing the latest tricks from large-scale GAN frameworks. Especially the non-invertible nature of the conditioning network makes cINNs suitable method for other computer vision tasks such as diverse semantic segmentation.
8

---

Under review as a conference paper at ICLR 2020
Grayscale input z = 0.0 =z"
z = 0.7 z
2 = 0.9 xz
2 1.0 zz
Z
1.25 =z
Figure 9: Effects of linearly scaling the latent code z while keeping the condition fixed. Vector is "typical" in the sense that ||2* || E ||z| and results in natural colors. As move closer to the center of the latent space (||z < || [], regions with ambiguous colors become desaturated, while less ambiguous regions (e.g. sky, vegetation) revert to their prototypical colors. In the opposite direction (|2) >
||z"
([), colors are enhanced to the point of oversaturation.
Inputs
New condition
Dutputs
Figure 10: For color transfer, we first compute the latent vectors z for different color images (L, a, b) (top row). We then send the same vectors through the inverse network with new grayscale condition L' (far left) to produce transferred colorizations a", b" (bottom row). Differences between reference and output color (e.g. pink rose) can arise from mismatches between the reference colors a, b and the intensity prescribed by the new condition L..

---

Under review as a conference paper at ICLR 2020
Figure 12: Failure cases of our method. Top: Sam- pling outliers. Bottom: cINN did not recognize an object's 's semantic class or connectivity.
cGAN
Figure 11: Diverse colorizations produced by cINN.
Alternative methods have lower diver- sity or quality, and suffer from inconsistencies within objects, or color blurriness and bleeding (compare Fig. 11, bottom).
Figure 13:
In ablation study, train cINN using the grayscale image directly conditional input, without a conditioning network h. The resulting colorizations largely ignore semantic content which leads exaggerated diversity. More ablations are found in the appendix.
Figure 14:
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In Inl. Conf. on Learning Representations, 2019. 2,
Jens Behrmann, David Duvenaud, and Jömn-Heniik Jacobsen. Invertible residual networks. arXiv: 1a1::11110000000 1.0095, 2018. 3
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Intl. Conf. on Learning Representations, 2019. 1,2
Yun Cao. Zhiming Zhou, Weinan Zhang, and Yong Yu. Unsupervised diverse colorization via generative adversarial networks. In Joint Eurup. Conf. on Machine Learning and Knowledge Discovery in Databases. pp. 151-166. Springer, 2017. 3, 8,
Ivo Danihelka, Balaji Lakshminarayanan, Benigno Uria, Daan Wierstra, and Peter Dayan. Comparison of maximum likelihood and GAN-hased training of RealNVPs. 17Xiv:7705.05266) 2017.
Aditya Deshpande, Jiajun Lu, Mao-Chuang Ych, Min Jin Chong, and David Forsyth. Learning diverse image colorization. Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 6837-6845,, 2017. 3, 8
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv: 2014. 3
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. 005.0006, 2016. 1, 2, 3, 5.5
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In Intl. Conf. on Learning Representations, 2017. 2
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. 13. Intl. Conf. Artificial Intelligence and Statistics, pp. 249-256, 2010. 5
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN; combining maximum likelihood and adversarial learning in generative models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 3
Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, and Kevin Murphy. Pixcolor: Pixel recursive colorization. 17Xiv:1705.07208, 2017.
Alfred Haar. 7ur Theorie der orthogonalen Funktionensssteeme. Mathematische Annalen, 69(3)3331-371, 1910. doi: 10. D0.100//FF0455326.. 5
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Advances Neural Information Processing Systems, pp. 6626 6637, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV'17 17, pp. 1501-1510, 2017.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultancous classification. ACM Transactions on Graphics (TOG), 35(4):1 10, 2016. 3,8
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR' 17, pp. 1125-1 134, 2017, 1,3,8 8
Jömn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. arXiv preprint arXiv:/811.0044 2018a. 6
Jöm-Henrik Jacobsen, Amold W.M. Smeulders, and Edouard Oyallon. i-RevNet: deep invertible networks. In International Conference on Learning Representations, 2018b.
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan. Bidircctional conditional generative adversarial networks. arXiv:1711.0746) 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. a710v /01.://990..06..... 2017.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. arXiv: 1800880073099 2018. 2, 3, 13
11

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, arXiv:1312.6114, 2013,
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 9743-4751, 2016.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. CausalGAN: Learning causal implicit generative models with adversarial training. arXiv: 02002, 2017. 2
Johannes Kopf, Michael F Cohen, Dani Lischinski, and Mat Uyttendacle. Joint bilateral upsampling. In ACM Transactions on Graphics (ToG), volume 26, pp. 96. ACM, 2007.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: flow-based generative model for video. urXiv:1903.01434 2019.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Europ. Conf. on Computer Vision, pp. 577-593. Springer, 2016.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Intl. Conf. on Computer Vision and Puttern Recognition, pp. 4681-46990 2017. 1
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1498-1501 2018. 2
Rui Liu, Yu Liu, Xinyu Gong, Xiaogang Wang, and Hongsheng Li. Conditional adversarial generative flow for controllable image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7992-8001, 2019. 3
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv: 141141141..1.. 2014. 2
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Representations, 2018. 1
Taesung Park, Ming-Yo Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. arXiv: 0001999307999 2019. 2,
Amelie Royer, Alexander Kolesnikov, and Christoph H. Lampert. Probabilistic image colorization. In British
Machine Vision Conference (BMVC), 2017. 4
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bersstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3) 2015. doi: 10.1007/s1 10007/s11233-1111 263-015-08866..
Robin Tibor Schirmeister, Patryk Chrabaszez, Frank Hutter, and Tonio Ball. Training generative reversible networks. 1rXi::1806.016/10 2018. 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, ahs/1449.1556. 2014.7
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. C. Cortes, N. D. Lawrence, D. D. Lec, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 3483-3491. 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. takes (only) two: Adversarial generator-encoder networks. In Thirry-Second AAAI Conference on Artificial Intelligence, 2018. 3,4
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798-8807, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsum: Construction of scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.:050663330 03365, 2015. 8, 13
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Europ, Eunf on Computer Vision, pp. 649-666, 2016. 3,
12

---

Under review as a conference paper at ICLR 2020
Jun-Yar Yan Zhu, Taesung Park. Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV'17. pp. 2223 2232, 2017a. 3
Jun-Ya Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Damell, Alexei A Efros, Oliver Wang. and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pp. 465-476, 2017b.
A ADDITIONAL EXPERIMENTS
A.1 ABLATION OF TRAINING IMPROVEMENTS
To demonstrate the improved stability and training speed through the improvements from Sec. 3.4, we perform ablations, see Fig. 15. The ablations for colorization were performed for the LSUN bedrooms task, due to training speed.
We find that for stable training at Adam learning rates of 10-3, the clamping and Haar wavelet downsampling are strictly necessary. Without these, the network has to be trained with much lower learning rates and more careful and specialized initialization, as used e.g. in Kingma & Dhariwal (2018). Beyond this, the noise augmentation and permutations lead to the largest improvement in final result. The effect of the noise more pronounced for MNIST, as large parts of the image are completely black otherwise. For natural images, dequantization of the data is likely to be the main advantage of the added noise. Note however, that the training curves of the models with and without noise augmentation is not directly comparable, as the loss differs an additional summand e ang.)) The effect on the training speed and stability is clearly visible regardless. The initialization only improves the final result by small margin, but also converges noticeably faster.
1.0
-2.3
MNIST
Colorization
0.5
Ablation
Final loss
Ablarion
Pinal loos
0.0
model
-3,344
-2.4-
Pull model
-2752
No noise aug.
-2,244
nolse sag
-2.701
-0.5-
No Xavier init.
-3.341
No Xavier init.
-2.730
-1.0
No perm
No clamp
-3.198
-0,808 (div.)
NLL loss
-2.-
No clamp
No
-2.200
0,107 (div.)
-15
No Haar
-2.570 (dix.)
-2.0
-2.6-
-25-
-3.0-
-2.-
-35
2000
4000
6000
8000
5000
LOC00
15000
20000
Gradient updates
Gradient updates
Figure 15: Training curves for each task, ablating the different improvements. "div." " denotes that the training diverges, and the lowest loss so far is given.
A.2 LSUN BEDROOMS
To provide simpler model for more in-depth experiments and ablations, additionally train cINN for colorization on the LSUN bedrooms dataset Yu et al. (2015). We use a smaller model than for ImageNet, and train the conditioning network jointly from scratch, without pretraining. Both the conditioning input, as well as the generated color channels have a resolution of 64 x 64 pixels. The entire model trains in under hours on a single GTX 1080Ti GPU.
To our knowledge, the only diversity-enforcii the colorGAN Cao al. (2017), which is also trained exclusively on the bedrooms dataset. Training the colorGAN for comparison, we find it requires over 24 hours to converge stably, after multiple restarts. The results are generally worse than those of the cINN, as shown in Fig. 16. While the resulting pixel-wise color variance is slightly higher for the colorGAN, it is not clear whether this captures the true variance, or whether it is due to unrealistically colorful outputs, such as in the second row in Fig. 16.
cGAN architecture previously used for colorization is
13

---

Under review as a conference paper at ICLR 2020
Metric
cINN
colorGAN
MSE best-of-8
6.14
6,43
Variance
33.69
39.46
FID
26.48
28.31
Table 2: Quantitative comparison between smaller cINN and colorGAN on LSUN bedrooms. The metries used are explained in Table 1.
cINN
COLORGAN
Figure 16: Qualitative comparison between smaller cINN and colorGAN on LSUN bedrooms.
14

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL FIGURES
B.1 COLORIZATION TRANSFER
The following figure contains an additional example of the color transfer, shown in Fig. 9 in the paper.
B.2 COLORIZATION INTERPOLATIONS
In the following, show 2-dimensional interpolations in latent space. Two random latent vectors (1), (2) are linearly combined:
= a + a2z (2)
with varying a2 E [-0.9... 0.9) across each axis of a grid. The center image has zt = 0. Note that the images in the corners have a larger magnitude than trained for, ||a*||2 [.3[[| leading some oversaturation artifacts, in Fig. 12 of the main paper.
15

---

Under review as a conference paper at ICLR 2020
60
16

---

Under review as a conference paper at ICLR 2020
17

---

Under review, as a denieerenee paper at ICLR 2020 0

---

Under review as a conference paper at ICLR 2020
19

---

Under review as a conference paper at ICLR 2020
20

---

Under review as a conference paper at ICLR 2020
B.3 COLORIZATION ADDITIONAL EXAMPLES
On the following pages, we provide some additional colorized images, as well as comparisons to alternative methods. All images are taken from the ImageNet 2012 validation set, and all methods were trained on ImageNet 2012. As we do not observe any significant diversity for the cGAN, only provide single sample.
B.3.1
GENERAL EXAMPLES
Ground tuth
CNN
Grayscale
oGAN
CINN (OuSS)
CVAE
Ground wuth
CNN
Gcayscale
cGAN
GINN
Ground
CNN
Gayy
6GAN
Giayscale
couns)
21

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Glayscalo
<RTAN
cINN (ours)
eVAF
Ground truth
CNN
scale
vGAN
GINN
eVAE
CNN
tuuh
CNN
Gnyscale
cGAN
cINN couns)
<VAF
Ground wuth
CNN
Grays
:GAN
cINN couns)
CVAL
22

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Grasscalo
oTTAN
cINN (ours)
eVAF
Ground truth
CNN
Scale
vGAN
cINN
eVAE
CNN
CNN
Gnyscale
uGAN
cINN (ouns)
<VAF
Ground tuth
CNN
Grayscale
:GAN
cINN couns)
CVAL
23

---

Under review as a conference paper at ICLR 2020
Ground muuhh
CNN
Grasscaoo
oTTAN
cINN (ours)
eVAF
Ground truth
CNN
Scaye
cGAN
cINN couns)
eVAE
CNN
CNN
Gnyscale
UGAN
cINN (ouns)
AVAE
Ground tuth
CNN
Grayscale
:GAN
cINN couns)
cVAL
24

---

Under review as a conference paper at ICLR 2020
Groan mruhh
CNN
Gnassclo
<TTAN
cINN (ours)
eVAF
Ground tuth
CNN
Scale
vGAN
cINN couns)
eVAE
CNN
eGAN
CNN
Cnyscale
oGAN
cINN couns)
CVAE
Ground wuth
CNN
Grayscale
uGAN
cINN couns)
cVAL
25

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Gayscalo
<CTAN
cINN (ours)
eVAF
Ground truth
CNN
scale
vGAN
cINN couns)
eVAE
CNN
chokk
eGAN
cINN
CNN
Gnyscale
UGAN
cINN couns)
AVAF
Ground wuth
CNN
Grayscale
:GAN
cINN couns)
eVAL
26

---

Under review as a conference paper at ICLR 2020
Ground tuuth
CNN
Glayscalo
oTTAN
cINN (ours)
eVAF
Ground truth
CNN
Gcayscale
cGAN
GINN couns)
eVAE
CNN
eGAN
CNN
Gnyscale
cGAN
cINN couns)
eVAF
Ground wuth
CNN
scaye
:GAN
cINN couns)
cVAL
27

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Glayscalo
6CTAN
cINN (ours)
eVAF
Ground tuuth
CNN
Scayscale
vGAN
cINN couns)
eVAE
CNN
CNN
Gnyscale
cGAN
cINN couns)
CVAF
Ground tuth
CNNN
SEE
Grayysall
UGAN
cINN couns)
CVAL
28

---

Under review as a conference paper at ICLR 2020
Ground muth
CNN
Grasscalo
6TIAN
cINN (ours)
eVAF
Ground truth
CNN
Grayscale
=GAN
cINN couns)
eVAE
CNN
muth
CNN
Gnyscale
eGAN
cINN (ouns)
AVAE
Ground wuth
CNN
Grayscale
:GAN
cINN couns)
CVAL
29

---

Under review as a conference paper at ICLR 2020
Ground muuhh
CNN
Grasscaoo
<TIAN
cINN (ours)
eVAF
Ground truth
CNN
Scayscale
vGAN
GINN
eVAE
CNN
CNN
Gnyscale
cGAN
cINN couns)
<VAE
Ground wuth
CNN
Gray
cGAN
cINN couns)
eVAL
30

---

Under review as a conference paper at ICLR 2020
B.3.2 HUMANS
We find that the cINN often has difficulties generating convincing skin colors, shown below. Clothing is colored in diverse ways, but not always with the correct connectivity and consistency.
Ground cuth
CNN
Grayscale
oGAN
GINN COunS)
CVAE
Ground tuth
CNN
Gray scale
cGAN
cINN
CVAE
Ground
CNN
Gayyccale
6GAA
aNNN
Gmyscale
Cours)
31

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Grasccaoo
<TAAN
cINN (ours)
eVAF
Ground truth
CNN
scale
vGAN
cINN
eVAE
CNN
eGAN
tuuh
CNN
Gnyscale
vGAN
cINN couns)
eVAF
Ground tuth
CNN
Gray scale
cGAN
cINN Couns)
CVAL
32

---

Under review as a conference paper at ICLR 2020
Ground truhh
CNN
Grasscalo
<TTAN
cINN (ours)
eVAF
Ground truth
CNN
Gray scale
=GAN
cINN
eVAE
CNN
Gmaycall
muth
CNN
Gnyscale
cGAN
cINN couns)
AVAE
B.3.3 LACKING CONSISTENCY
The following failure cases exhibit a lack in consistency, in occluded objects, multi-pan objects, or reflections.
33

---

Under review as a conference paper at ICLR 2020
Ground muth
CNN
Gryssalo
<TAAN
cINN (ours)
eVAF
Ground truth
CNN
Scayscale
vGAN
GINN couns)
CVAE
CNN
eGAN
CNN
Gnyscale
AGAN
cINN couns)
<VAE
Ground wuth
CNN
Gcaye
:GAN
cINN couns)
eVAL
34

---

Under review as a conference paper at ICLR 2020
B.3.4 COLOR IGNORES SEMANTIC CONTENT
In the following examples, the semantic content of the image was recognized, and the generated colors are clearly incorrect.
Ground tuuth
CNN
Grayscale
cGAN
cINN Couns)
eVAE
Ground wuth
CNN
Scayscale
cGAN
cINN
CVAE
Grouad
CNN
Glayscale
oGAA
Gmascale
cours)
35

---

Under review as a conference paper at ICLR 2020
Ground truth
CNN
Goasscalo
6TAAN
cINN (ours)
eVAF
Ground tuuh
CNN
Grayscale
eGAN
GINN
eVAE
CNN
eGAN
CNN
Gnyscale
cGAN
cINN couns)
eVAE
Ground wuth
CNN
Graysclle
UGAN
cINN couns)
CVAL
36

---

Under review as a conference paper at ICLR 2020
Ground muth
CNN
Grasscalo
oTTAN
cINN (ours)
eVAF
Ground truth
CNN
Scayscale
vGAN
GINN
CVAE
CNN
cGAN
B.3.5 OUTRIGHT FAILURES
For the following images, the cINN fails completely, and generates colors with seemingly little or no connection the grayscale image.
Giayscale
(ouns)
37

---

Under review as a conference paper at ICLR 2020
Ground muthh
CNN
Gayscalo
<TAAN
cINN (ours)
eVAF
Ground wuth
CNN
Scayscale
vGAN
GINN
eVAE
CNN
CNN
Gnyscale
oGAN
cINN couns)
AVAE
Ground wuth
CNN
Grayscale
:GAN
GINN Couns)
CVAL
38

---

Under review as a conference paper at ICLR 2020
Groan mruhh
CNN
Grasscalo
<TAAN
GINN (ours)
eVAF
Ground tuuh
CNN
Grayscale
vGAN
GINN couns)
CVAE
CNN
eGAN
(unn)
39


Reviews:
[{"review_id": "r1xHGWLRKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an invertible generative network, for conditional image generation.  The model is an extension of Real NVP with a conditioning component. Experiments are performed for image generation on two tasks: class conditional generation on MNIST and image colorization conditioned on a grey scale image (luminance). Comparisons are performed with a conditional VAE and a conditional GAN (Pix2Pix). An ablation study motivates the importance and role of the different components.\nThe model itself is a relatively simple extension of Real NVP, where a condition vector is added to the initial model as an additional input to the NN components of the invertible blocks. In the experiments conditioning may be a simple class indicator (MNIST) or a more complex component corresponding to a NN mapping of an initial conditioning image (colorization). The experiments show that this model is able to generate good quality images, with an important diversity, showing that the conditioning mechanism works well. The quantitative comparison also shows that the proposed model is competitive with two baselines taken in the VAE and GAN families. The model works well for the non-trivial task of colorization.\nThe authors claim is that they are the first to propose conditional invertible networks. The main contribution is probably the implementation of the model itself. They make use of several “tricks” that improve a lot on the performance as demonstrated by the ablation study.  As such more details and motivations for these different ideas that improve the performance and stability of the model would be greatly helpful.  It looks like these are not details, but requirements to make the whole thing work. The Haar component for example should be better motivated. There is no comparison in the ablation study with an alternative, simpler decomposition.\nThe baselines are probably not the strongest models to date, and better results could certainly be obtained with other VAE or GAN variants. For example, there have been several works trying to introduce diversity for GANs. This is not redhibitory, but this should be mentioned. Besides a short description of the two baselines, would make the paper more self-contained.\nThe quantitative comparison with the VAE baseline, shows that the two models are quite similar w.r.t. different measures. This could be also commented.\nThe notations for the Jacobian do not integrate the conditioning, this could be corrected.\nConcerning the interpretation of the axis for the MNIST experiment, it is not clear if they are axis in the original space or PCA axis. If this is the first option, more details are needed in order to understand how they were selected.\n\n\n------post rebuttal -----\n\nThe authors clarified several of the raised points. I keep my score.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxHRDIKFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes conditional Invertible Neural Networks (cINN), which introduces conditioning to conventional flow-based generative models. Conditioning is injected into the model via a conditional affine coupling block, which concatenates conditioning with the input to the scaling and shifting sub-networks in the coupling block. Other small modification are proposed to improve training stability at higher learning rates, including soft clamping of scaling coefficients, and Haar wavelet downsampling, which is proposed to replace the squeeze operation (pixel shuffle) that is often used in flow-based models. The invertibility of the cINN allows for style transfer by projecting images into the latent space, and then reconstructing the image with a different conditioning. The performance of the cINN is evaluated empirically on the task of colorization, where it is shown to outperform other techniques in terms of nearness to the true colours, as well as sample diversity.\n\nOverall, I would tend to vote for accepting this paper. The base method for integrating conditioning into the flow is simple and intuitive, and additional modifications which allow for stable training at higher learning rates, such as soft clamping and Haar wavelet downsampling, appear to be very effective. Conditional models often lend themselves to a wide variety of useful applications, so I think this work could be of interest to many. \n\n\nMy primary concerns with this paper are related to the comparison of image colorization methods. Specifically:\n1) I would like to see a comparison to Probabilistic Image Colorization (PIC) [1], which was mentioned in the related work section but not included in the comparison of colorization models. PIC has been shown to outperform VAE-MDN in terms of diversity, so it would be good to include it. Code is available online with pretrained ImageNet models (https://github.com/ameroyer/PIC), so it should not be difficult to add.\n\n2) Pix2pix is known to have very bad sample diversity. A more useful comparison would be to evaluate one of the newer variants of Pix2pix that emphasizes sample diversity, such as BicycleGAN [2] or MSGAN [3]. Code is also available for each of these models (https://github.com/junyanz/BicycleGAN, https://github.com/HelenMao/MSGAN), although you would need to train models from scratch.\n\n3) Pixel-wise metrics such as MSE are bad at measuring perceptual similarity. While these pixel-wise metrics are still useful for comparison to prior work, better metrics are available for evaluating image colorization. I would recommend the use of Learned Perceptual Image Patch Similarity (LPIPS) [4] in place of pixel-wise distance measures for evaluating image similarity and diversity.\n\n\nThings to improve the paper that did not impact the score:\n5) I was somewhat disappointed by how little attention was spent on the Haar wavelet downsampling method. It seems like a very neat idea, but it is only briefly explored in the ablation study. It would be nice to include a more in-depth study of how it compares to the conventional pixel shuffle downsampling, perhaps in terms of stability with learning rates, and final model performance. \n\n6) There is some potentially related work on conditional adversarial generative flows [5] that could be added to the literature review if deemed relevant enough.\n\n\nReferences:\n[1] Amelie Royer, Alexander Kolesnikov, and Christoph H. Lampert. Probabilistic image colorization. In British Machine Vision Conference (BMVC), 2017.\n\n[2] Zhu, Jun-Yan, et al. \"Unpaired image-to-image translation using cycle-consistent adversarial networks.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[3] Mao, Qi, et al. \"Mode seeking generative adversarial networks for diverse image synthesis.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[5] Liu, Rui, et al. \"Conditional Adversarial Generative Flow for Controllable Image Synthesis.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n### Post-Rebuttal Comments ###\n\nThank you for including the PIC results. I did not realize that they would take so long to run, but I think it important to include comparison with the current SOTA methods so that we have some reference. It is also nice to see the addition of the LPIPS metric. My overall opinion of the paper is not much changed from before, so I will retain the same score.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hygm93nTqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use a normalizing flow architecture to tackle the structured output problem of generalization.\nThey propose:\n- a conditioning architecture: they use a convolutional feature extractor (similar to a U-Net architecture), and (on top of the common architectural details of models like Glow - Kingma and Dhariwal, 2018) uses Haar wavelets for downsampling;\n- they train their architecture stably using the maximum likelihood principle;\n- they demonstrate interesting properties of their model coming from the bijectivity. \n\nThis is an interesting application of the architecture to colorization. The diverse and consistent colorization results are compelling (with comparison with previous methods), while clearly showing the failure cases where the model should be improved. Ablation studies are done to show the importance of different components (e.g. the conditioning network). The paper is clearly written. \n\nA few remarks:\n- arctan soft-clamping seems very similar to the scalar times tanh soft-clamping of Real NVP (Dinh et al., 2016). Why was arctan adopted?\n- the choice of the car image (the biggest one in Figure 10) for the colorization transfer is questionable. I'm not able to tell from this figure if there was any segmentation happening in the model. The pose of the cars are similar, the car in the back is mostly black. The colorization transfer result gives me the impression that the segmentation is not done properly, e.g. the red color from the red car image seems to spill outside of the confine of the car in the colorization transfer.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "H1xjIDIp5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors provide an extension to the invertible generative models, by fashioning them into conditional invertible generative networks. The networks are conditioned at the input with a feed-forward network initialized with a VGG based classification network. Conditioning is implemented within the coupling layers (Dinh et. al. 2016) of the invertible model by simply concatenating the hidden layer output of the VGG encoder. The model is learned using an MAP objective along with some modifications to the original training procedure (described in sec 3.4). The model is evaluated qualitatively on \"style transfer\" on MNIST digits and image colorization. The technical contribution of this paper is the somewhat straight-forward extension of the cINNs to conditional generative networks. The actual implementation of conditioning seems quite trivial (sec 3.1). Although the results on colorization are claimed to be good, the baselines they compared to are not very recent (e.g. cGANs). Overall, I believe there is very less novelty, technical sophistication and performance improvements in this paper.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an extension of flow-based invertible generative models to a conditional setting. The key idea is fairly simple modification of the original architecture, but authors also propose techniques for down-sampling with Haar wavelets. The experimental results on class-conditional MNIST generation and colorization are promising. However, in terms of weakness, the technical novelty seems somewhat limited although it's a reasonable extension. In addition, the experimental results lack evaluation on general conditional image generation tasks with more widely used benchmarks (e.g., class-conditional generation setting for real images, such as CIFAR and ImageNet; attribute-conditional or image-to-image translation settings; etc.). In other words, colorization seems like a niche task. The baselines compared are not the strongest models. For example, the diversity of 
cGANs can be significantly improved by simple plug-in modifications (e.g., DSGAN) to any existing GAN architectures, and those methods were demonstrated on broader benchmarks. So I view the experimental validation somewhat limited in scope and significance. While this work presents a reasonable extension of conditional invertible generative models with promising results, I believe that more work needs to be done to be publishable at a top-tier conference.

Diversity-Sensitive Conditional Generative Adversarial Networks
https://arxiv.org/abs/1901.09024

Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis
https://arxiv.org/abs/1903.05628
* exactly the same idea as DSGAN above.

Decision: Reject