Learning Likelihoods with Conditional Normalizing Flows 
Authors: Christina Winkler
Daniel Worrall
Emiel Hoogeboom
Max Welling
Venue:  (2020)
Keywords: Likelihood learning
conditional normalizing flows
generative modelling
super-resolution
vessel segmentation

Abstract: Normalizing Flows (NFs) are able to model complicated distributions p(y) with strong inter-dimensional correlations and high multimodality by transforming a simple base density p(z) through an invertible neural network under the change of variables formula. Such behavior is desirable in multivariate structured prediction tasks, where handcrafted per-pixel loss-based methods inadequately capture strong correlations between output dimensions. We present a study of conditional normalizing flows (CNFs), a class of NFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x). CNFs are efficient in sampling and inference, they can be trained with a likelihood-based objective, and CNFs, being generative flows, do not suffer from mode collapse or training instabilities. We provide an effective method to train continuous CNFs for binary problems and in particular, we apply these CNFs to super-resolution and vessel segmentation tasks demonstrating competitive performance on standard benchmark datasets in terms of likelihood and conventional metrics.

Under review as a conference paper at ICLR 2020
LEARNING LIKELIHOODS WITH CONDITIONAL NOR- MALIZING FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
Normalizing Flows (NFs) are able to model complicated distributions py (v) with strong inter-dimensional correlations and high multimodality by transforming a simple base density (z(2) through an invertible neural network under the change of variables formula. Such behavior desirable in multivariate structured pre- diction tasks, where handcrafted per-pixel loss-based methods inadequately cap- ture strong correlations between output dimensions. We present study of con- ditional normalizing flows (CNFs), class of NFs where the base density to out- put space mapping is conditioned on an input x, to model conditional densities Prxx (y)æz). CNFs efficient in sampling and inference, they can be trained with a saelihood-baseee objective, and CNFs, being generative flows, do not suf- fer from mode collapse training instabilities. We provide an effective method train continuous CNFs for binary problems and in particular, we apply these CNFs to super-resolution and vessel segmentation tasks demonstrating competitive per- formance on standard benchmark datasets in terms of likelihood and conventional metrics.
INTRODUCTION
Learning conditional distributions Py (y|x) is one of the oldest problems in machine learning. When the output y high-dimensional this is a particularly challenging task, and the practitioner is left with many design choices. Do we factorize the conditional? If not, do we model correlations with, say. a conditional random field (Prince 2012) Do we use a unimodal distribution? How fat should the tails be? Do we use explicit likelihood at all, use implicit methods (Mohamed & Rezende 2015) such as a GAN (Goodfellow et al. 2014)? Do we quantize the output? Ideally, the practitioner should not have make design choices at all, and the distribution should be learned from the data.
In the field of density estimation normalizing flows (NFs) are a relatively new family of models (Rezende & Mohamed, 2015). NFs model complicated high dimensional marginal distributions (y) by transforming simple base distribution or prior pz(z) through learnable, invertible mapping fo and then applying the change of variables formula. NFs are efficient in inference and sampling. are able to learn nnterdiimeniionl correlations and multi-modality, and they are exact likelihood models, amenable to gradient-based optimization.
Flow-based generative models (Dinh et al., 2016) are generally trained on the image space, and are in some cases computationally efficient in both the forward and inverse direction. These are advantageous over other likelihood based methods because i) sampling efficient opposed to au- toregressive models (Van Oord et al. 2016). and ii) flows admit exact likelihood optimization in contrast with variational autoencoders (Kingma & Welling 2014).
Conditional random fields directly model correlations between pixels, and have been fused with deep learning (Chen et al.) (0166). However, they require the practitioner to choose which pixels have pairwise interactions. Another approach uses adversarial training (Goodfellow et al., 2014). A downside is that the training procedure can be unstable and they are difficult to evaluate quantita- tively.
We propose to learn the likelihood of conditional distributions with few modeling choices using Con- ditional Normalizing Flows (CNFs). CNFs can be harnessed for conditional distributions Py|x (y x)

---

Under review as a conference paper at ICLR 2020
by conditioning the prior and the invertible mapping on the imput In particular, apply condi- tional flows to super-resolution (Wang et al. 2018) and vessel segmentation (Staal et al. 2004). We evaluate their performance gains multivariate prediction tasks along side urchitecturally-matchh factored baselines by comparing likelihood and application specific evaluation metrics.
BACKGROUND
In the following, we present the relevant background material on normalizing flows and structured prediction. This section covers the change of variables formula, invertible modules, variational dequantization and conventional likelihood optimization.
2.1 NORMALIZING FLOWS
A standard NF in continuous space is based on simple change of variables formula. Given two spaces of equal dimension 2 and y; a once-differentialle, parametric, bijective mapping S& y 2, where @ are the parameters of f; and a prior distribution pz Pz(z), we can model a complicated
distribution py (y) as
( Qy
py =y))))p((6(>))
(I)
The term (y) /Oyl is the Jacobian determinant for evaluated at y and accounts for volume changes induced by fe- The transformation fø introduces correlations and multi-modalle in py- The main challenge in the field normalizing flows designing the transformation fa. It has to be i) bijective, have efficient and tractable Jacobian determinant, iii) be from 'flexible' model class. In addition, iv) for fast sampling the inverse needs to be efficiently computable. Below we briefly state which invertible modules are used in our architectures, obeying the aforementioned points.
Coupling layers Affine coupling layers (Dinh et al., 2016) are invertible, nonlinear layers. They work by splitting the input z into two components Zo and z and nonlinearly transforming as function of Z1, before reconcatenating the result. If [Zo z and y [Yo. y1] this is
Yo s(1) 2n + ((21)
Zo (Zo ((y1))/8(11)
Y1 Z1
21 Y1
Where the scale s(-) and translation t(-) functions can be any function, typically implemented with CNN. Similar conditioning with normalizing flows has been done in previous works by Mohamed Rezende (2015) and Kingma et al. (2016b).
Invertible 1 x 1 Convolutions Proposed in Kingma & Dhariwal (2018), invertible con- volutions help mix information across channel dimensions. We implement them as regular convolutions and for the inverse, we convolve with the inverse of the kernel.
Squeeze layers Squeeze layers (Dinh et al. 2016) are used to compress the spatial resolution of activations. These also help with increasing spatial receptive field of pixels in the deeper activations.
Split Prior Split priors (Dinh et al. 2016) work by spliting a set of activations z into two com- ponents 2g and Z1. We then condition 21 Zn using simple base density e.g. p(z, (z)) ((21; ((z)), o² (zu)), where ((-) and o²(-) are neural networks. The component Zg can be mod- eled by further flow layers. This prior, is useful for modeling hierarchical correlations between dimensions, and also helps reduce computation, since 2n is reduced in size.
Variational dequantization When modeling discrete data, Theis al. (016)) introduced the cept dequantization. For this, they modeled the probability mass function over as a latent variable model
P Poodal (y) = P(ylv)p(v)dv== ((r,, v) dv
We also injective mappings
2

---

Under review as a conference paper at ICLR 2020
where the latent variables v V are catiinuuu--vluued This is convenient model to use, since the marginal p(v), living on a continuous sample space, can be modelled with a continuous NF. The indicator function P(ylv) I[v
distribution P(y(v) is known as the quantizer and is typically y + [o, P). Other works (Hoogehoom et al. 2019a Tran et al. 2019) directly model (y) with discrete-valued flow, but these known be difficult optimize. As extension of dequantization, Ho et al. (2019) introduced variational distribution g(vly), called a dequantizer, and write a lower bound on the data log-likelihood using Jensen's 's inequality follows
Pau((y) log (y) (rr log /
v)
(3)
g(vly) ((vly)
ply, v) 2 E / ((vly) log p(y, v) (vvly)
= Pauully (y) log
dv
(4)
that the dequantizer distribution q(vly) must be defined such that v E y + [0, otherwise ply, v) = O and the lower-bound is undefined. Restricting g(vly) to satisfy this condition, results in the following variational dequanti- zation bound
Noting that the joint p(y; v) T[v E y (0.1)_p(vv)
Paunty) log (y) Brau(r)) / ((vll)) p(v) q(vly)
dv.
(5)
2.2 STRUCTURED PREDICTION
Structured prediction tasks such as image segmentation or super-resolution, can be probabilistically framed as learning an unknown target distribution p" [x), with an input x and a target E ). In practice with deep learning models, the unknown distribution is often learned by factored model:
p(yxx) II p(y)))),
where ya represents the dth dimension y. Several loss-based optimization methods are a special case of this factored model. The mean squared error is equivalent to product of normal distributions with equal and fixed standard deviation. Other examples are: cross entropy, equivalent to a product of log categorical distributions, and binary cross entropy, equivalent to product of log Bernoulli distributions.
With factorized independent likelihoods, individual dimensions of are assumed to be conditionally independent. As a result, sampling leads to results with uncorrelated noise over the output dimen- sions. In the literature, fix for this problem is to visualize the mode of the distribution and interpret that as a prediction. However, because the likelihood was optimized assuming a conditionally inde- pendent noise distribution, these modes tend to be blurry and lack crisp details.
METHOD
In this section we present our main innovations. i) learning conditional likelihoods using CNFs and ii) a variational dequantization framework for binary random variables.
3.1 CONDITIONAL NORMALIZING FLOWS
We propose to learn conditional likelihoods using conditional normalizing flows for complicated target distributions in multivariate prediction tasks. Take an input x X' and a regression target E y. We learn complicated distribution Py x x) using conditional prior Pzix (z)x(z|x) and mapping fo y x X + Z, which bijective y and 2. The likelihood of this model is:
Oz
Ø O θy
(ylx = (z|x)
= PzIx (fa [x(y,x)|x))
θy
Notice that the difference between Equations and 7 is that all distributions are conditional and the flow has conditioning argument of x.
3

---

Under review as a conference paper at ICLR 2020
The generative process from x y (shown in Figure be described by first sampling Pz|x (z/x) x) from a simple base density with its parameters conditioned on x (for us this is a diagonal Gaussian) and then passing it through a sequence of bijective mappings fg (): This allows for modelling multimodal conditional distributions in y, which is typically uncommon.
For the training procedure, the process runs reverse. We begin with label y and conditioning input x. We 'flow' the label back through la to yield z fo(y; x), and then we evaluate the log-likelihood of the parame- ters of the prior, given this transformed label z. The flow and prior parameters can be optimized using stochastic gradient descent and training in minibatches in the usual fashion. Note that this style of training conditional den- sity model prlx (y(x) differs fundamentally from tradi- tional models, because we compute the log-likelihood in z-space and not y-space. As a result, we are not biasing our results with arbitrary choice of output-space likeli- hood or in the case of this paper, handcrafted image loss. Instead, one could interpret this method as learning the correlational and multimodal structure of the likelihood or simply put loss-learning
Training
Sampling
2
y
p(z|x)= +X
p(z[x) +
Figure 1: Diagram of our model in the train and sampling phases. Solid lines represent deterministic mappings and dashed lines represent sumpling. The conditioning variable enters net- work in base density p(z)x) and the bi- jective mappings fly.x, x).
Conditional modules In our work, the conditioning is introduced in the prior, the split priors, and the affine coupling modules. For the prior, we set the mean and variance as functions of For the split prior, we add as conditoning argument to the conditional p(z) x). And for the affine coupling layers, we pass x to the scale and translation networks so that
Conditional Prior Conditional Split Prior Conditional Coupling
p(zxx) ((x),
p(z, 22. x) = (Z (Z x), (z), x))
Yo ,(1, 2o ((21, x): Y1 =1
In practice these functions are implemented using deep neural networks. First the conditioning term transformed into rich representation h = g(x) using a large network g. Subsequently, each function in the flow is applied to concatenation 1., of h and the relevant part of z. For example, the translation of conditional coupling is computed ((21, x)
NN((21, h))-
3.2 VARIATIONAL DEQUANTIZATION FOR BINARY RANDOM VARIABLES
We generalize the variational dequantization scheme for the binary setting. Let y E {o. 11 be multivariate binary random variable and v E Ie'' its dequantized representation. In Ho et al. (2019) the bound is not guaranteed to be tight, since there is domain mismatch in the support of p(v and the variational dequantizer g( n(v Technically, if p(v) is modeled as a bijective mapping from a Gaussian distribution where the mapping only has finite volume changes, then the support of p(v) is unbounded. On the otherhand, the support of the dequantizer is bounded and so we have to redefine either the dequantizer to map to all of R or restrict the support of the flow to bounded volume inside D We resolve this by dequantizing with half-infinite noise, where
vly, z = 0.5 + sign(y 0.5) softplus(NN(1)) (z)).
The softplus guarantees that samples from the neural network NN are only positive. If y is the term sign(y - 0.5) outputs positive-valued noise and if y is 0 the noise is negative-valued.
4 RELATED WORK
Normalizing flows were originally introduced to machine learning to learn a flexible variational posterior, conditional distribution, in VAEs (Rezende & Mohamed| 2015 Kingma al. 201
van den Berg et al. 2018 Flow-based generative models (Dinh et al., 2016 Papamakarios et al,
2017 Huang al. 2018 Kingma & Dhariwal 2018 Hoogeboom al. 2019b Grathwohl et al.
2019 Cao et al. 2019 Chen et al. (019) are typically trained directly in the data space. Several of
these are designed to be fast to invert, which makes them suitable for drawing samples after training.
4

---

Under review as a conference paper at ICLR 2020
Different versions and applications of conditional normalizing flows include Agrawal & Dukkipati (2166) who utilize flows in the decoder of Variational AutoEncoders Kingma & Welling (0014).
which are conditioned on the latent variable. 'Trippe & Turner (2018) who utilize conditional flows for prediction problems in a Bayesian framework for density estimation, Atanov et al. (2019) intro-
duce semi-conditional flow that provides efficient way to learn from unlabeled data for semi- supervised classification problems. Very recently, Ardizzone et al. (2019) have proposed conditional flow-based generative models for image colorization, which differs from our work training objec-
tive, architecture and applicabiliity to binary segmentation. Autoregressive models (Van Oord et al., 2016) have also been studied for conditional image generation van den Oord al. (2016) but are generally slow to sample from.
Adversarial methods (Goodfellow al. 2014 have widely been applied to (conditional) image density modeling tasks (Vu et al.] 2019; Sajjadi et al. 2017b) Yuan et al., 2018 Mechrez et al.) 2018), because they tend to generate figh-fidlitty images. Disadvantages of adversarial methods are that they can be complicated to train, and it is difficult to obtain likelihoods. For this reason, it can be hard to assess whether they are overfitting or generalizing.
5 EXPERIMENTS
Here explain our experiments into super-resolution and vessel segmentation. All models were implemented using the PyTorch framework.
5.1 SINGLE IMAGE SUPER RESOLUTION
Single Image Super Resolution (SISR) methods aims to find high resolution image Zhr given single (downsampled) low resolution image Ttr. Framing this problem as learning a likelihood, we utilize CNF learn the distribution p( [x,)). To compare our method also train factor- ized baseline likelihood model with comparable architectures and parameter budget. The factorized baseline uses product of discretized logistic distributions (Kingma K al.) [2166a] [Salimans al.) 2017). All methods are compared on negative logz-likelihood if available, which has the informa- tion theoretic interpretation bits per dimensions. In addition, we evaluate using SSIM (Wang al. 2004) and PSNR metrics.
Implementation Details
The flow is based on the (Dinh et al. 2016 Kingma & Dhariwal 2018)
multi-scale architectures. Each step of flow consists of K subflows and 1. levels. One subflow consists of an activation normalization, 1 x convolution, and our conditional coupling layer. After completing level, half of the representation is factored-out and modeled using our conditional split prior. After all levels have been completed, our conditional prior is used to model the final part of the latent variable.
The conditioning variable 2i, is transformed into the feature representation h using Residual-in Residual Dense Block (RRDB) architecture (Wang et al. 20188), consisting of 16 residual-in -n-rsiuuaal blocks. To match the parameter budget, the channel growth Is 55 for the baseline and the growth is 32 for the CNE
Data The models are trained on natural image datasets, Imagenet32 and Imagenet64 (Chrabaszez) al. 2017). Since the dataset has no test set, we use its validation images as test set. For validation we take 10000 images from the train images. The performance is always reported on the test set unless specified otherwise. We evaluate our models on widely used benchmark datasets Set5 (Bevilacqua et al. 2012). Set14 [eydee et al. 2012) and BSD100 (Huang et al. 2015). At test time, we pad the test images with zeros at right and bottom so that they are square and compatible with squeeze layers. When evaluating on SSIM and PSNR, we can extract the patch with the exact image shape. For all datasets the LR images are obtained using MATLABs bicubic kernel function with reducing aliasing artifacts following Wang et al. (2018). For these experiments, the pixel values dequantized by adding uniform noise (Theis et al. 2016).
Training Settings We train on ImageNet32 and ImageNet64 for 200, 000 iterations with mini batches of size 64, and learning rate of 0.0001 using the Adam optimizer (Kingma & Bal 2015).
5

---

Under review as a conference paper at ICLR 2020
(a) Low resolution
(b) Ground truth
(b) CNF sample
(c) Baseline mode
Figure 2: Super resolution results on the 1magenet64 test data. Samples are taken from the CNF The p( (itr) and the mode is visualized for the factorized baseline model. Best viewed elec- tronically.
The high-resolution image The either the original 32 32 or 64 x original input images. The flow architecture is build with L = levels and K 8.
5.1.1 EVALUATION
In this section the performance of CNFs for SISR is compared against a baseline likelihood model on ImageNet32 and ImageNet64. Their performance measured in logz-likelihood (bits per dimension) is shown in Table [ which show that the CNF outperforms the factorized baseline in likelihood. Recall that the baseline model is factorized and conditionally independent. These results indicate that it is advantageous to capture the correlations and multi- multi-modality present in the data.
Table 1: Comparison of likelihood learning with CNFs and factorized discrete baseline on Ima- geNet32 and ImageNet64 measured in bits per dimensions.
Dataset
CNF
factorized LL
ImageNet32
3.01
4.00
ImageNet64
2.90
3.61
Super resolution samples Thr p( from the 1magenet64 test data are shown in Figure The distribution mode of the factorized baseline is displayed in Figure [ panel c). We show that the baseline is able to learn a relationship between conditioning variable x and y, but lacks crisp details. The super-resolution images from the CNF are shown in panel b). Notice there are more high-frequency components modelled, for instance in grass and in hairs. Following Kingma & Dhariwal (2018), we sample from the base distributions with a temperature 7 of 0.8 to achieve the best perceptual quality for the distribution learned by the CNF.
As there are no standard metric for measuring perceptual quality, we measure performance results on PSNR and SSIM between our predicted image and the ground truth image in Table We compare CNFs to other
per-pixel loss based methods and the factorized baseline for a 2x upsampling task standard super-resolution benchmarks. If available, we report negative log2- likelihood (bpd) (computed as an average over 1000 randomly cropped 128 x 128 patches). Note that without any hyperparameter tuning or compositional loss weighting, is typical in SISR, the CNF performs competitively with state-of-the-an art super-resolution methods by simply optimizing the likelihood. The SSIM scores for the baseline perform on par or better than the adversarial methods and the CNF on all benchmarks. On PSNR scores however, the CNF beats the factorized discrete baseline. Samples shown Figure3Jsow that the CNF predictions have more fine grained texture details. Comparing this finding with the baseline that outperforms every method SSIM, show that metrics can be misleading.
Notice how samples from independent factorized likelihood model have lot of color noise, whereas samples from the CNP do not have such problems. Increasing temperature increases high- level detail, where find that can be attributed to the property of flows to model pixel correlations among output dimensions for high-dimensional data such as images.
0.5 strikes balance between noise smoothing and detail. This

---

Under review as a conference paper at ICLR 2020
Factorized Baseline
CNF r 0
CNF 0.5
CNF 1.0
Ground truth
Figure 3: Conditional samples from the CNF (ours) for sampling temperatures {0., 0.5, 1.0) and the factorized discrete baseline for 2x upscaling. Conditioning image is a baboon from Set14 test set. Both models were trained on ImageNet64. Best viewed electronically.
Table 2: CNF compared to factorized discrete baseline and adversarial, pixel-wise methods (Dong et al. 2015 Sajjadi et al. 2017a Prez-Pellitero et al. (016) based on negative logz-likelihood (bits per dimension or bpd), PSNR, SSIM and for 2x upscaling. Our methods were trained on ImageNe164.
Set5 PSNR
Set14 PSNR
BSD100
Model Type
bpd
SSIM
bpd
SSIM
bpd
PSNR
SSIM
Bicubic
33.7
0.930
30.2
0.869
29.6
0.843
SRCNN
36.7
0.954
32.4
0.906
31.4
0.888
PSyCO ENet
36.9
0.956
32.6
0.898
31.4
0.890
37.3
0.958
33.3
0.915
32.0
0.898
LL Baseline
2.34
32.5
0.958
3.23
31.0
0.917
3.20
30.6
0.900
CNF (ours)
2.11
36.2
0.957
2.51
32.5
0.911
2.33
31.4
0.893
5.2
VESSEL SEGMENTATION
Vessel segmentation is important, long-standing, medical imaging problem, where we seek to segment blood vessels from pictures of the retina (the back of the eye). This is a difficult task, because the vessels are thin and of varying thickness. A likelihood function used in segmentation is
Input
DRIU
Factored model (Us)
CNF (Us)
Ground truth
Figure 4: Example of retinal segmentations using DRIU, our likelihood baseline trained with the same loss, and our CNF. For the CNF, the mean of 100 samples is visualized. Notice that our seg- mentations more accurately capture the vessel width, which is overdilated in the DRIU and factored models.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Numerical results on the DRIVE dataset. We see that the CNF is in the range of the SOTA
model DRIU. SE: Structured Forests (Dollár & Zitmick 2013). LD: Line Detector (Ricci & Perfetti,
2007). Wavelets (Soares et al., 2006 Human (Staal al. 2004), HED: Holistic Edge Detector
Xie & Tu. 2015). KB: Kernel Boost Becker et al. (2013). No N Fields (Ganin & Lempitsky,
2014). DRIU: Deep Retinal Image Understanding (Maninis et al. 2016). Our answers are shown in
mean + 1std form, where statistics are taken over 5 runs.
SE
LD
Wavelets
Human
HED
DRIU
Factored (ours)
CNF Uniform (Ouns)
CNF (ours)
bdd F-Score
0.0847 0.0015
0.3366 =: 0,0290
0.0254 0.0008
0.658
0.692
D.762
0.791
0.794
0,800
0.805
0.821
0.815
0.001
0.762 0.002
0.819 0.001
weighted Bernoulii distribution
L Pj 8-3s (1 - -p;) (1))118)-(113) 123 + (1 p2)
where P, p(y, 1|x) is the prediction probability that pixel vj is positive (vessel class) and /3 is a class balancing constant set to ~ 10% for us. This loss function is preferred, because accounts for the apparent class imbalance in the ratio of vessels to background. In practice, the numerator of this likelihood is used as a loss function and the normalizer is ignored. The resulting loss is called weighted cross-entropy. In our experiments literature), but we report likelihood values including the normalizer, for a meaningful comparison.
train using the weighted cross-entropy (as in the
Dataset and comparisons We test on the DRIVE database Staal et al. (2004) consisting of 584 x 565, 8-bit RGB images, split into 20 train, and 20 test images. To compare against other methods, we plot precision-recal curves, report the maximum F-score along each curve (shown as a dot in the graph), report the bits per dimension, and plot distributions in PR-space. The main CNN-based contenders are Deep Retinal Image Understanding (DRIU) (Maninis et al.] 2016) and Holistically- Nested Edge Detection (HED) (Xie & Tu. 2015). both of which are instances of Seeply-Supervised Nets (Lee et al. 2015). The main difference between DRIU and HED is that DRIU is pretrained on ImageNet Krizhevsky et al. (2012); whereas, HED is not. Other competing methods are reported with results cited from (Maninis et al.) 2016). For faimess, we also train a model which we call the likelihood baseline, which uses the exact same architecture as the flow but run feedforward model and trained with the weighted Bernoulli loss.
Implementation The flow is identical to the flow used in the previous section, with some key differences. i) instead of activa- tion normalization, we use instance normalization (Ulyanov et al.) 2016). äi the conditional affine coupling layers do not contain scaling component s(-) but just the translation t(-), hence it is volume preserving, and tii) we train using variational dequanti- zation. Since the data is vanury, we dequantize accord- ing to the Flow++ scheme of Ho et al.) modified to binary vari- ables (see Section 3.2). using a CNF at just a single scale for the dequantizer. The CNF is conditioned on resolution- resolution-matched matched fea- tures extracted from VGG-like network (Simonyan & Zisserman (2015). This model is composed of blocks of the form block InstanceNorm, RELU, conv], and 2x2 max-pooling layer, shown in Table 4
Table 4: Feature extractor ar- chitecture for retinal vessel segmentation. RES. abbrevi- ates resolution. Outputs are layer 4 and 7. LAYER
TYPE
RES.
0
input block
1024
1024
2
max-pool
512
block
512
4
block
512
5
max-pool
256
block
256
All filter sizes are 3x3. The outputs are at layers 4 and 7. These used to condition the resolution 512 and 256 levels of the CNF, respectively.
7
block
256
Training/test settings We train using the Adam optimizer at learning rate 0.001, and a minibatch size of 2 for 2000 epochs. All images are padded to 1024x 1024 pixels, so that they are compatible with squeeze layers. We use 360 rotation augmentation, isotropic scalings in the range [0.8. 1.2], and shears drawn from normal distribution with standard deviation 10a. At test time we draw samples from our model and compare those against the groundtruth labels. This contrasts with other methods, that measure labels against thresholded versions of factorized predictive distribution. To
8

---

Under review as a conference paper at ICLR 2020
create the PR curve in Figure 5 we take the average of 100 samples and threshold the resulting map (example shown in Figure 4) While crude, this mean image is useful in defining PR-curve, since there is not great topology change between samples.
Evaluation The results of our experiments are shown in Table and Figure 5 with visualization in Figure We see in the table that the CNF trained with our binary dequantization achieves the best bits per dimension, with comparable F-score to the state of the art model (DRIU), but our model does not require pretraining ImageNet. Interestingly, tization slightly unstable and the results were far from satisfactory. In the PR-curve Figure 5 we show comparable curve for our binary dequantized CNF the DRIU model. These results, how- ever, say nothing about the calibration of the probability outputs, but just that the various probability predictions are well ranked. To gain insight into the calibration of the probabilities, we measure the distribution of precision and recall values for point samples drawn from all models, including a second human grader, present in the original DRIVE dataset. We synthesized samples from fac- tored distributions (all except ours and 'human'), ). by sampling images from a factored Bernoulli with mean as the soft image. We see the results in the right hand plot of Figure which shows that while the other CNN- recall. On the other hand, the CNF drops in precision a little bit, but makes up for this terms of high recall, with PR distribution overlapping the human grader. This indicates that the CNF has learned well-calibrated distribution, compared to the baseline methods. Further evidence of this is in the visualization in Figure 4 which shows details from the predicted means (soft images). This shows that the DRIU and likelihood baseline overdilate segmentations and the CNF does not. This can be explained from the fact that in the weighted Bernoulli it is cheaper to overdilate than to underdilate. Since the CNF contains no handcrafted loss function, we circumvent this pathology.
found training a flow with uniform dequan-
methods such DRIU or HED have good precision, they suffer in term of
1.0
1.0
0.9
0.9
0.8
0.8
0.7
Precision
0.7
Precision
0.6
0.5
0.6
DRIU
0.4
Kemel Boost
0.5
Wavelets
0.3
HED
LineDctector
0.2
0.4
N4. Fields
DRIU
HED
SE
SE
Boast
LineDetector
CNF (Us)
CNF (Us)
Wavelets
No Fields
Human
0.3.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
0.0
0.1
0.2 0.3 0.4
0.5 Recall
0.6
0.7 0.8
0.9 1.0
Recall
(a) PR curve
(b) PR seatter plot
Figure 5: Here we show two visualizations of the same data. LEFT: We show the PR-curves gen- erated from sweeping threshold soft images output by each listed method. Maximal F-scores for each curve are shown as circles with the green lines indicating constant F-score. We see that our method beats all traditional methods and is on par with DRIU, which unlike was pretrained on Imagenet. RIGHT: We show scatter plot in PR-space of samples drawn from each model. To draw samples from the all factored models, we sample images from a factored Bernoulli with mean as the soft image. We see that the DRIU and HED models, while having good precision, have poor recall this regime. This indicates that while the output of their networks produce good ranking of probabilities, the values of the probabilities are poorly calibrated. For we drop in precision slightly, but gain greatly in terms recall, indicating that our samples are drawn from a better calibrated distribution, overlapping significantly with the Human distribution.

---

Under review as a conference paper at ICLR 2020
CONCLUSION
In this paper we propose to learn likelihoods of conditional distributions using conditional normal- izing flows. In this setting. supervised prediction tasks can framed probabilistically. In addition, we propose generalization of variational dequantization for binary random variables, which is useful for binary segmentation problems. Experimentally we show competitive performance with competing methods in the domain of super-resolution and binary image segmentation.
REFERENCES
Siddharth Agrawal and Ambedkar Dukkipati. Deep variational inference without pixel-wise recon- struction. CoRR, abs/1611.00 1e2266110052000 .05209, 2016.
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided im- age generation with conditional invertible neural networks. CoRR, abs/1907.00232 2019. URL
bttp://arxiv.orgg hbwpgg/abs/1997/ 02392
Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semi- conditional normalizing flows for semi-supervised learning. 05 2019.
Carlos J. Becker, Roberto Rigamonti, Vincent Lepetit, and Pascal Fua. Supervised feature learn- ing for curvilinear structure segmentation. In Medical Image Computing and Computer-Asiisted Intervention MICCA/ 2013 16th International Conference, Nagoya, Japan, September 22-26, doi:
2013, Proceedings. Part I, pp. 526-533 2013.
URL
https://doi. org/ 10. 1007/970 978 3 642 10811-8813-00
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding, In BMVC, 2012.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, pp. 511, 2019.
Liang-Chie -hangCChieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR, hbss/606.00915. 2016. URL httt:://txxiii www.s/xww.wrg.abb//oao wrgp.bs/..........
00915
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jörn-Henri Jacobsen. Residual flows for invertible generative modeling. CoRR, abs/ 090//09960735.... 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as alternative to the CIFAR datasets. CoRR, Abs//107.08819 2017. URL /ttp:tt/://axxii... /
abs/1707. 08819
Laurent Dinh, Jascha Sohl- Sick-Dickstein, and Samy Bengio. Density estimation using real NVP. CoRR, abs/1605. 60..08800,, 2016. URL httt::tttt//iii www. wtwpg//b///eee
08803
Piotr Dollár and C. Lawrence Zitnick. Structured forests for fast edge detection. In IEEE Inter- national Conference Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pp. 1841-1848, 2013. doi: 10. 1.1109///CCV.201..23.1 URL
1109/
TCCV 2013.22
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolutton using deep convolutional networks. CoRR, abs/1501.00092, 2015. URL http://aaxii..rttttte// abs/ 1501 00092
Yaroslav Ganin and Victor S. Lempitsky. N4-fielss: Neural network nearest neighbor fields for image transforms. CoRR, abs/1406.6553 2014. URL
arxix.iiiiv..... http::/aaxxv..org/abb///..
6558
10

---

Under review as a conference paper at ICLR 2020
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference Neural Information Processing Systems Volume 2, NIPS' 14, pp. 2672-2680, Cambridge, MA, USA, 2014. MIT Press. URL http://dl acm. org/ citat cfffid==26990000 2969125
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In 7th In- termational Conference on Learning Representations, ICLR 2019, 2019.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long mlr press/
Beach, California, USA, pp. 2722-2730, 2019. v97/ho19a.htm. html
Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer discrete flows and lossless compression. CoRR, his/1995.00376. 2019a. URI.
1905. 07376
Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 2019b.
Chin-Wei Wei Huang, David Krueger, Alexandre Lacoste, and Aaron C. Courville. Neural autoregressive flows. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholnsmässann Stockholm, Sweden, July 10- 15, 2018, pp. 2083-2092, 2018
J. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self-exemplars. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5197-5206, June 2015. doi: 10. (009//VPR.2015.72999
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations, 2014.
Diederik Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverse autoregressive flow. CoRR, abs/1606 606404434, 2016a. httpt::://axxil..oo//oab////////
04934
Diederik P. Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverse autoregressive flow. CoRR, abs/1600.04990 04934, 2016b. URL wwt.xxvv.or//b//660000o
04934
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10236- 10245. Curran Associates. Inc., 2018. URL http:://papeet nips wttp:cccppppppcpcp...cecppppppee pdf
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Process- ing Systems 25: 26th Annual Conference on Neural Information Processing Sys- tems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pp. 1106-1114, 2012. URL /
http://appprrnisss...c.eeeccceep 4824 104--mmggenc magemet--elllle coanet-classe cat on- th whw.reeee-----ooooooiiiiooaaalle

---

Under review as a conference paper at ICLR 2020
Chen- Yu Lee, Saining Xie, Patrick W. Gallagher. Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Proceedings of the Eighteenth International Conference on Artificial Intelli- gence and Statistics, AISTATS 2015, San Diego, California, USA, May 9-12. 2015, 2015. URL
httt:://proceedi: ngs. m1 prwss vv8// 11/38//eee eeelaa. ht.ml
Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Andrés Arbeláez, and Luc Van Gool. Deep reti- nal image understanding. In Medical Image Computing and Computer-Assisted Intervention MICCAI 2016 19th International Conference, Athens, Greece, October 17-21, 2016, Pro- ceedings, Part 11, pp. 140- 148, 2016. doi: 10.
17. URL https:
//doi.org/ 10 1007/
Roey Mechrez, Itamar Talmi, Firas Shama, and Lihi Zelnik-Manor Learning to maintain natural im- age statistics. CoRR, abss/800300l.m 803.04666, 2018.
04626
Shakir Mohamed and Danilo J. Rezende. Variational information maximisation for intrinsically mo- tivated reinforcement learning. In Proceedings of the 28th International Conference on Neural In- formation Processing Systems Volume 2, NIPS' 15, pp. 2125-2133, Cambridge, MA, USA, 2015. ctt.:tiitattee wttp:ecccccmmmmmeeeem htt.::aa................22......
MIT Press. URL http: //dl acm.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for den- sity estimation. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gameett (eds.), Advances in Neural Information Processing Systems 30, pp. Curran Associates, Inc., 2017. URL http::// 6828 2828-masked's -e--mmeeddautttrerreesssee flow- fow.::ofrreeeeesiie-eeetttteeeee
Simon JD) Prince. Computer vision: models, learning, and inference. Cambridge University Press, 2012.
Eduardo Prez-Pellitero, Jordi Salvador, Javier Ruiz-Hidalgo and Bodo Rosenhahn. Manifold span reduction for super resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)., June 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 1530-1538, 2015. URL http: URL htttp:://proeeeeiggg..... .mw.eediiiiiimmmmmm...mmmmm.mm.e mll press/ v31 rwtp/dele.d.....mmm html
Elisa Ricci and Renzo Perfetti. Retinal blood vessel segmentation using line operators and support vector classification. IEEE Trans. Med. Imaging, 26(10):1357-1365 2007. doi: 10. 1009/TMM. 2007.898551. URL
org./.rg//11 http://1.11111..111 1109/ 2007. 898551
M. M. Sajjadi, B. Schölkopf, and M. Hirsch. EnhanceNet: Single image super- resolution through automated texture synthesis. In Proceedingy IEEE International Con- ference on Computer Vision (ICCV), pp. 4501-4510 Piscataway, NJ. USA, October 2017a. IEEE. URL http://openaccess www.s/cccccssstteeeeeoeoe thecvf com content ICCV 2017/papers/
Sa jadí SirhanceNet_Sinne Ticiegggl__mmaeeecccc. Baceccvvo11__apppe htwpaag__cc___11__ppprrr___
M. S. M. Sajjadi, B. Schölkopf, and M. Hirsch. EnhanceNet: Single image super- resolution through automated texture synthesis. In Proceedings IEEE International Confer- ence on Computer Vision (ICCV), pp. 4501-4510, Piscataway, NJ, USA, October 2017b.
IEEE. URL httt:::/ppeaacceesssssts///////e thecvf com/ coom/cootent...mm wttpoootttnntoooooooooeeoooooeee 2tt.22001ppprr////// Sa 1 jjdii EnhanceNet Siig_iinnnnlleeeee Image ICC_ 2011__ppper_pee
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: Improving the
pixelenn with discretized logistic mixture likelihood and other modifications. 5th International Conference on Learning Representutions, ICLR 2017, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
1556
12

---

Under review as a conference paper at ICLR 2020
João V. B. Soares, Jorge J. G. Leandro, Roberto M. Cesar, Herbert F. Jelinek, and Michael Cree. Retinal vessel segmentation using the 2-d gabor wavelet and supervised classification. 1EEE Trans. Med. Imaging, 25(9):1214-122 1222, 2006. doi: 10. 10.1099TMM.2206.879997 URLhttps:
//doi.
109/TMI. TMI 6. 879967
Joes Staal, Michael D. Abråmoff, Meindert Niemeijer, Max A. Viergever, and Bram van Ginneken. based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging, 23 (4):501-509, 2004. doi: 10.1 10/1 TMI 00004625662
URL https:://ci.ooo/1100000
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL httt:://aaxxtt. orq/ abs//ab//ta 1511. 01844
Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. In Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop. New Orleans, Louisiana, United States, May 6, 2019, 2019. URL
Brian L. Trippe and Richard E Turner. Conditional density estimation with bayesian normalising 04908
flows. 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022, 2016. URL http:://rxiy: org/ abs bbas1600..0l 08022
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester nor- malizing flows for variational inference. In Proceedings of the Thirty-Fourty Conference on Un- certainty in Artificial Intelligence, UAI 2018, Monterey, Califormia, USA, August 6-10, 2018, pp. 393-402, 2018.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances Neural In- formation Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, pp. 4790 4798, 2016.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016.
Thang Vu, Tung Luu, and Chang Yoo. Picception-Enhaneee Image Cuppe-Reeoluttion via Relativistic Generative Adversarial Networks: Munich, Germany, September 8- 14, 2018, Proceedings, Part 7.
V. pp. 98-113. 01 2019. ISBN 978-3-030-11090 1020-8.
doi:
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. ESRGAN: enhanced super-resolution generative adversarial networks. CoRR, abs/ 0ts/18809002211 2018. URL
httpg/g/bbs/8/// 00219
Zhou Wang. Alan Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: From error visibility structural similarity. Image Processing, IEEE Transactions on, 13:600 612, 05 2004. doi: 10.11009/1P.2033.. 119861.................
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In 2015 IEEE International Con- ference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1395-1403) 2015.
2015. doi: 10.1109/CCC.2205..10 164. URL http:////oo..cc11.111111
64
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 814-81409, 2018.
13

---

Under review as a conference paper at ICLR 2020
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Proceedings of the 7th International Conference on Curves and Surfaces, pp. 711-730, Berlin, Heidelberg, 2012. Springer- ISBN 978-3-642-27412-J1 doi: 10.1007/
978-3-642-27413-88 47. URL https: /tt.://......///000000000033322-
47
14

---

Under review as a conference paper at ICLR 2020
A ARCHITECTURES
This section describes architecture and optimization details of the conditional normalizing flow net- work, low-resolution image feature extractor, and shallow convolutional neural network in the con- ditional coupling layers.
The conditional coupling layer is shown schematically in Figure This shows that conditioned on an imput we are able to build relatively straight-forward invertible mapping between latent representations and y, which have been partitioned into vectors of equal dimension.
Z:
y. y,
Z,
y. y,
Z,
Z,
(a) Forward
(b) Reverse
Figure The forward and reverse paths of the conditional coupling layer. our experiments we concatenate an embedding of the conditioning input to the latent which is fed through another neural network output the affine transformation parameters applied Zo- This operation is invertible in and y, but not in
Details for the CNFs are given in Table and the details of the individual coupling layers in Table ① and 7 The architecture of the feature extractor is given in Table [ The architecture has levels
and subflows, following (Dinh al. 2016 Kingma Dhariwal 2018). All networks are optimized using Adam (Kingma & Ba 2015) for 200000 iterations.
Table 5: Configuration of the CNF architecture for the super-resolution task.
DATASET
MINIBATCH SIZE
LEVELS
SUB-FLOWS
LEARNING RATE
ImageNet32
64
8
0.0001
ImageNet64
64
8
0.0001
DRIVE
2
0.001
15

---

Under review as a conference paper at ICLR 2020
Table 6: Architecture details for single coupling layer in the super resolution task. The variable Cout denotes the number of output channels. The first two convolutional layers are followed by ReLU activation.
LAYER
INTERMEDIATE CHANNELS
KERNEL
Conv2d
512
3
Conv2d
512
X 1
Conv2d
Cout
X 3
Table 7: Architecture details for a single coupling layer in the DRIVE segmentation task. The
variable Cout denotes the number of output channels for the The first two convolutional layers followed by a ReI.U activation.
LAYER
INTERMEDIATE CHANNELS
KERNEL SIZE
Conv2d
32
3 x 3
InstanceNorm2d
32
ReLU
Conv2d
Cout
3 x 3
Table 8: Architecture details for the conditioning network in the super-resolution task. Residual- in-residual denseblocks Wang et al. (2018) are utilized. The channel growth is adjusted so that the CNF and the factorized baseline have an equal number of parameters.
MODEL TYPR
RRDB BLOCKS
CHANNEL GROWTH CONTEXT CHANNELS
CNF Factorized LL.
16
32
128
16
55
128
16

---

Under review as a conference paper at ICLR 2020
B CONDITIONAL IMAGE GENERATION
In this section, larger versions of the ImageNet64 samples are provided, sampled at different tem- peratures T.
Figure 7: Super resolution results CNF trained on Imagenet64 sampled at temperature 0.
17

---

Under review as a conference paper at ICLR 2020
Figure 8: Super resolution results for the CNF trained on Imagenet64 sampled 0.5
Figure Super resolution results for the CNF trained on Imagenet64 sampled at 0.8
18


Reviews:
[{"review_id": "rJeN7RdTYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the conditional normalizing flow for structured prediction. The idea is to use conditioning variables as additional inputs to the flow parameter forming networks.  The model was demonstrated on image superresolution and vessel segmentation.\n\nI find the contribution of this paper minimal. The idea of conditioning has extensively been used during recent years because it is the most natural thing to do (e.g., [1], [2] and numerous other papers). Their's nothing new about the flows used in this paper. The results in table 2 are not convincing; I see no benefit of using the proposed flow model for image super-resolution instead of the SOTA super-resolution methods. This also applies to other experiments.\n\n[1] van den Oord et al., Conditional Image Generation with PixelCNN Decoders, 2016.\n[2] Papamakarios et al., Masked Autoregressive Flow for Density Estimation, 2017.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklyFUW-9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented the conditional normalizing flows (CNFs) as a new kind of likelihood-based learning objective.  There are two keys in CNFs. One is the parametric mapping function f_{\\phi} and the other is the conditional prior. This paper assumed the conditional prior as Gaussian distribution of x. The mapping function is invertible with x as a parameter. The prior parameter and \\phi are updated by stochastic gradient descent. The latent variable z is then sampled from conditional prior. The output targe y is obtained with dependency on x and f_{\\phi}. \n\nStrength:\n1. This study adopted the flow-based model to estimate the conditional flow without using any generative model or adversarial method.\n2. This method obtained the advanced results on DRIU dataset without the requirement of pretraining.\n3. This paper proposed an useful solution to train continuous CNFs for binary problems.\n\nWeakness:\n1. It is required to address how to design the function f_{\\phi} which depends on x. In particular, the property invertibility should be clarified.\n2. Why the issues of mode collapse or training instability in flow are considerable in the experiments?\n3. It will be meaningful to evaluate this method by performing the tasks on text to image or label to image.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxfq8jg5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper: \n\nThe paper proposes an extension of Normalizing flows to conditional distributions. The paper is well written overall and easy to follow. Basically the conditional prior z|x = z=f_{\\phi}(y,x), where x is the conditioning random variable, and we apply the change of variable formula to get the density of y|x . For example in super resolution y is the high res image and  x is the low res. image.  To sample from the models authors propose to use f^{-1}_{\\phi}(z;x). \n\nThe conditional modules are natural extensions of invertible blocks used in the literature (coupling layers, split priors, conditional coupling, 1x1 conv), where the conditioning is done on some hidden representations of the conditioning variable x (i.e one or multiple layers of NN).\n\nAuthors propose a dequantization for binary random variables (useful for segmentation applications), where they give an implicit model for the dequantizer (obtain a continuous variable from a discrete binary variable).\n\nAuthor apply the method in two applications super-resolution and vessel segmentation. the method is compared to supervised learning of the corresponds between x and y and to others competitive methods in the literature and shows some advantage. \n\nMinor comments : \n\n- Formatting the bibliography is messed up and needs some cleaning , Figure 5 is also making formatting issues of the paper. \n- Figure 1 for sampling it should be f^-1_{\\phi } and not f_{\\phi}\n\nReview: \n\n- Figure 2 is hard to get any idea of the sample quality would be good also to put the low resolution input to the algorithm . Also did you use a temperature sampling for the baseline ? otherwise the comparison is not fair.\n\n- The Drive database is too small 20 training samples and 20 testing only? can the model be just overfitting?\n\n- In the vessel implementation why do you drop the scaling modules? \n\n- The conditioning for the vessel implementation on x is on two layers , would be great to put all architectures of the models in details , and to show both sampling and training paths \n\n- It would be great to add the details of the skip connection used from the network processing x, and how ensure that the flow remains invertible.  \n\nOverall this is a well written paper and a good addition to normalizing flows methods , some discussion of related works on conditional normalizing flows and more baselines with other competitive methods based on GANs for example would be helpful but not necessary. \n\nIt would be great to add details of the architectures and on skip connections and how to ensure invertibility for this part in the model . \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a conditional normalizing flow approach to learning likelihoods. While reviewers appreciated the paper, in its present form it lacked a clear champion, and there were still some remaining concerns about novelty and clarity of presentation. The authors are encouraged to continue with this work and to account for reviewer comments in future revisions. Following up on the author response, a reviewer adds:
"Thanks for your clarification. I still disagree that the conditional flow architecture proposed should be considered as a novel contribution. The reason why I mentioned [1] or [2] was not because they follow the exact setting (coupling based conditional flow model) discussed in this paper. I wanted to highlight that the idea to use conditioning variables as an input to the transforming network (whether it is an autoregressive density function, autoregressive transforming network, or coupling layers) is quite universal (as we all know many of the existing codes implementing flow-based models includes additional keyword arguments 'context' to model conditioning). I'm not sure why the fact that the proposed framework is conditioning on high-dimensional variables makes a contribution. There seems to be no particular challenge in doing that and novel design choices to circumvent that (i.e., we can just use existing architectures with minor modifications).

I agree that the binary dequantization should be considered as a contribution, but as significant as to change my decision to accept. Thanks for the clarification on experiments. Considering this, I raise my rating to weak reject...

Another previous work I forgot to mention in the initial review is "Structured output learning with the conditional generative flow", Lu and Huang 2019, ICML 2019 invertible neural network workshop. This paper discusses the conditional flow based on a similar idea, and attacks high-dimensional structured output prediction. I think this should be cited in the paper."

Decision: Reject