Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presented the conditional normalizing flows (CNFs) as a new kind of likelihood-based learning objective.  There are two keys in CNFs. One is the parametric mapping function f_{\phi} and the other is the conditional prior. This paper assumed the conditional prior as Gaussian distribution of x. The mapping function is invertible with x as a parameter. The prior parameter and \phi are updated by stochastic gradient descent. The latent variable z is then sampled from conditional prior. The output targe y is obtained with dependency on x and f_{\phi}. 

Strength:
1. This study adopted the flow-based model to estimate the conditional flow without using any generative model or adversarial method.
2. This method obtained the advanced results on DRIU dataset without the requirement of pretraining.
3. This paper proposed an useful solution to train continuous CNFs for binary problems.

Weakness:
1. It is required to address how to design the function f_{\phi} which depends on x. In particular, the property invertibility should be clarified.
2. Why the issues of mode collapse or training instability in flow are considerable in the experiments?
3. It will be meaningful to evaluate this method by performing the tasks on text to image or label to image.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary of the paper: 

The paper proposes an extension of Normalizing flows to conditional distributions. The paper is well written overall and easy to follow. Basically the conditional prior z|x = z=f_{\phi}(y,x), where x is the conditioning random variable, and we apply the change of variable formula to get the density of y|x . For example in super resolution y is the high res image and  x is the low res. image.  To sample from the models authors propose to use f^{-1}_{\phi}(z;x). 

The conditional modules are natural extensions of invertible blocks used in the literature (coupling layers, split priors, conditional coupling, 1x1 conv), where the conditioning is done on some hidden representations of the conditioning variable x (i.e one or multiple layers of NN).

Authors propose a dequantization for binary random variables (useful for segmentation applications), where they give an implicit model for the dequantizer (obtain a continuous variable from a discrete binary variable).

Author apply the method in two applications super-resolution and vessel segmentation. the method is compared to supervised learning of the corresponds between x and y and to others competitive methods in the literature and shows some advantage. 

Minor comments : 

- Formatting the bibliography is messed up and needs some cleaning , Figure 5 is also making formatting issues of the paper. 
- Figure 1 for sampling it should be f^-1_{\phi } and not f_{\phi}

Review: 

- Figure 2 is hard to get any idea of the sample quality would be good also to put the low resolution input to the algorithm . Also did you use a temperature sampling for the baseline ? otherwise the comparison is not fair.

- The Drive database is too small 20 training samples and 20 testing only? can the model be just overfitting?

- In the vessel implementation why do you drop the scaling modules? 

- The conditioning for the vessel implementation on x is on two layers , would be great to put all architectures of the models in details , and to show both sampling and training paths 

- It would be great to add the details of the skip connection used from the network processing x, and how ensure that the flow remains invertible.  

Overall this is a well written paper and a good addition to normalizing flows methods , some discussion of related works on conditional normalizing flows and more baselines with other competitive methods based on GANs for example would be helpful but not necessary. 

It would be great to add details of the architectures and on skip connections and how to ensure invertibility for this part in the model . 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes the conditional normalizing flow for structured prediction. The idea is to use conditioning variables as additional inputs to the flow parameter forming networks.  The model was demonstrated on image superresolution and vessel segmentation.

I find the contribution of this paper minimal. The idea of conditioning has extensively been used during recent years because it is the most natural thing to do (e.g., [1], [2] and numerous other papers). Their's nothing new about the flows used in this paper. The results in table 2 are not convincing; I see no benefit of using the proposed flow model for image super-resolution instead of the SOTA super-resolution methods. This also applies to other experiments.

[1] van den Oord et al., Conditional Image Generation with PixelCNN Decoders, 2016.
[2] Papamakarios et al., Masked Autoregressive Flow for Density Estimation, 2017.
review_title: Official Blind Review #1

Meta-review: The authors propose a conditional normalizing flow approach to learning likelihoods. While reviewers appreciated the paper, in its present form it lacked a clear champion, and there were still some remaining concerns about novelty and clarity of presentation. The authors are encouraged to continue with this work and to account for reviewer comments in future revisions. Following up on the author response, a reviewer adds:
"Thanks for your clarification. I still disagree that the conditional flow architecture proposed should be considered as a novel contribution. The reason why I mentioned [1] or [2] was not because they follow the exact setting (coupling based conditional flow model) discussed in this paper. I wanted to highlight that the idea to use conditioning variables as an input to the transforming network (whether it is an autoregressive density function, autoregressive transforming network, or coupling layers) is quite universal (as we all know many of the existing codes implementing flow-based models includes additional keyword arguments 'context' to model conditioning). I'm not sure why the fact that the proposed framework is conditioning on high-dimensional variables makes a contribution. There seems to be no particular challenge in doing that and novel design choices to circumvent that (i.e., we can just use existing architectures with minor modifications).

I agree that the binary dequantization should be considered as a contribution, but as significant as to change my decision to accept. Thanks for the clarification on experiments. Considering this, I raise my rating to weak reject...

Another previous work I forgot to mention in the initial review is "Structured output learning with the conditional generative flow", Lu and Huang 2019, ICML 2019 invertible neural network workshop. This paper discusses the conditional flow based on a similar idea, and attacks high-dimensional structured output prediction. I think this should be cited in the paper."

Decision: Reject