Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary:
 
This paper uses Variational Inference to query pre-trained flow-models. If the flow variable is $x$, then querries are either conditioned on the part of $x=(x_1,x_2)$, or on a differentiable transformation of $x$. Authors first show that it is not trivial to conduct such queries exactly or approximately for a general class of flow-models. The paper then proposes a framework that affords such querries by working in the latent space--empirical evidence favors the proposed approach over contemporary methods.  

### Strength:

The problem is well-motivated. The authors outline several instances where one would want to work with a pre-trained model and use it to query over new data. Further, I appreciate the use of proof that the problem is hard to motivate the solution. I also commend the author's effort to collect empirical evidence for their method. I especially like the "Why Ambient VI fails" explanation and find the contour plot beautiful. 


### Concerns:

One primary with paper is the lack of clarity and overload of notation. This is especially true for section 3.

The equations 1 and 2, use the distribution $p_{f\circ \hat{f}}(x_2)$ and  $p_{\hat{f}}(x_2)$. The preceding section uses $p_{f\circ \hat{f}}(x)$ and  $p_{\hat{f}}(x)$ for $x = (x_1, x_2)$; it is possible that the authors are referring to the marginal for $x_2$. However, it is not immediately clear from the discussion. Further, there is reference to $y$ which is undefined till that point. More so, if we are talking about marginals, then this necessiates the need to evaluate these marginals.  The authors offer no discussion on this. 

In my understanding of the work, $p_f (x_2| \tilde{x_1} = x_1^*)$ is approximated with $p_f (x| \tilde{x_1} = x_1^*)$. Thereafter, we can use $p_f (x| \tilde{x_1} = x_1^*) \approx p_f (x_2,x_1) p_\sigma  (x_1^*|x_1)$ to calculate the eq 1. However, this still leaves me uncertain about the marginal distribution $p_{f\circ \hat{f}}(x_2)$. I also believe if the above explanation is true, then it is bit of a leap.

However, these concerns vanish for the section with differentiable transformations--here, we do not talk about the partitioning of x, so the expressions are straightforward to evaluate. 

The easiest way to convince me would be to offer a clear explanation of the method and to straighten out the notation. It will be great to get an algorithm--like the one in Appendix C--for the partitioning case. 

#### Minor concerns and suggestions:

In section 2, the last paragraph, the formulation is not unique to the authors' framework. It is the fundamental idea of VI to use ELBO over KL divergence; however, the current presentation makes it feel that this is a novel observation made by the authors. 

Section 1, third paragraph: "VI allows for likelihood evaluation and.."--I believe the term likelihood has been used to refer to $\log q$--this is confusing as the term is often reserved for $\log p(x)$ or $\log p(x|z)$. I will suggest being unambiguous with terminology.

Section 1, fourth paragraph: "Specifically, we use variational inference to
learn a distribution in the latent space ..."--I find this sentence hard to parse. How is a distribution "fed" to the pre-trained model? (this is more of a writing concern than anything else.)

Authors can consider using less left margin for bullet points under the heading "our contributions."

### Update after the rebuttal

I think that the ideas in this paper are interesting and can inspire new uses. All of us agree that the problem presented here was important, and there is a lot of work to be done in this domain. However, after reading the discussion with other reviewers and their reviews, I believe the manuscript can benefit from another review round. Specifically, the authors can benefit from a thorough revision of the claims in the paper. Further, I would encourage authors to at least investigate how naive amortization approaches fair (irrespective of the result, authors will develop a stronger case for their line of work.) 

review_title: I like the motivation of the problem and the solution. There is also a commendable effort to collect empirical evidence. Some reservations; can benefit from a second review round.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper proposes to solve the conditional inference problem by performing a relaxed version of variational inference in the prior space of the flow-based model. The model p(x) is pretrained, and one is interested sampling from p(x|observation). The observation could be some subset of x (inpainting), gray scale representation of an image (colorization), lower-res representation (super resolution), or noisy version of the data (compressed sensing). The paper proposes to perform inference in the prior space, by composing the post-hoc trained latent flow with the trained invertible decoder, as the conditional distribution in that space is believed to have a better geometry.  

Contributions and novelties: (A) propose to perform inference in the latent space of a latent variable model to side step the bad geometry, (B) propose to replace hard constraint with stochastic relaxation (placing an additional likelihood term to model the dummy variable). (C) Applications seem interesting for testing the quality of an unconditional flow-based model.

Flaws: missing several important references in the discussion to prior works. These include [1], which describes a more general framework to post-hoc perform sampling from a conditional distribution of a learned latent variable model by fitting a distribution in the latent space; [2,3,4] which propose to mitigate the bad geometry of the learned data energy (in this case, the density model itself) by transforming it into a space where it’s more Gaussianized. Similar idea has also been incorporated in [5] to enable the training of a residual EBM (no need to cite). The key novelties (A and B) are incremental in nature given the above related works that are not cited.


[1] Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models, 2017
[2] Transport map accelerated markov chain Monte Carlo, 2014
[3] NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport, 2019
[4] Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC, 2020
[5] VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models, 2020

Additional details:

For experiments, please report the bits-per-dim (BPD) of the pre-trained flow model, as well as unconditional samples for reference and better comparison. 
“Ambient” langevin dynamics is a very weak baseline, since one can easily improve the mixing via deriving a kernel in the latent space (as per [3]). It wouldl be a more fair comparison since the proposed method also composes with the learned decoder flow. Same goes to the naive implementation of the PL-MCMC which only uses a random walk kernel, which is a very weak baseline.  
Are the rows of Fig 4 independent? Why do they have the same stroke style if they are all conditioned only on the label only? Is it an indicator of the mode seeking behavior of the reverse KL?   
In the qualitative results (tab 2, fig 3 and fig 5), the generated samples do not really satisfy the hard constraints that they are conditioned on (e.g. the subsets of x for inpainting), possibly due to the relaxation via the dummy variable. This is not desirable. Is it possible to anneal the $\sigma$ while training the latent flow so that it will concentrate on the (potentially degenerate) solution that satisfies these constraints? 
The authors claim the hardness result is surprising. It has been long shown that sampling from a general Bayesian Belief Network is NP-hard [6]. Can’t the same conclusion be derived from that, with the main difference being the explicit parameterization via a coupling flow? 

[6] The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks, Cooper, 1990



Additional questions about the hardness result (I didn’t read the proof):

Generality of the hardness result: the statement is about the hardness of sampling from the conditional distribution. What is the conditional distribution in this context, is it referring to p(x2|x1) (i.e. observation is a subset of x)? Please be precise. If this is the case it is consistent with the presentation of the previous section (VI). However the proposed method seems to require a more general treatment to take account of the other tasks, e.g. inverse problems. 

The discussion of hardness seems to be used to motivate the relaxation of the hard constraints (of the givens). It doesn’t seem that relevant when the observation is not a deterministic function of x (e.g. inpainting, coloration, etc). For compressed sensing for example, the likelihood p(observation|x) naturally exists and is non-degenerate (by assumption). The presentation seems a bit confusing. 


--- POST REBUTTAL ---

Modified my score after the rebuttal, since (1) I believe the re-purposing achieved by this work can potentially broaden the applicability of flow-based generative model (2) the authors have toned down the abstract and clarified the contributions in the intro, which now better reflects the value of the work. 

I am still leaning towards rejection at the end given the limited originality of the proposed method and the lack of a more comprehensive discussion of different possible approaches, but as means to the same end. For example, the relaxed inference problem can be solved with an MCMC method. These should all be discussed and compared if the contribution is about repurposing a joint likelihood model using flows.
 

PS. the last line (the references) of the last page might have been a mistake. 


review_title: Solving interesting inference problems with existing tricks

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors propose a novel method of estimating conditional distributions over arbitrary partitions of variables $x = [x_1,x_2]$ using an existing pre-trained flow model for $p(x)$. Their method fits a new *pre-generator* flow $\hat{f}$ for each observation which maps from a base distribution $\epsilon \sim N(0,\mathbb{I})$ to the latent variables $z \sim N(0,\mathbb{I})$ where the mapping $z \leftrightarrow x$ is learned by the pre-trained "base" flow $f$. The result is a mapping $\hat{f}$ which learns to shift probability mass to regions of the latent space which correspond to the conditional distribution of the given observation. The authors present comprehensive experimental results which show a clear improvement over existing methods for conditional inference but with the drawback of needing to re-train the pre-generator for each individual observation.

Pros:
- Very well written, clear, easy to understand
- The proposed method is intuitive and well defined
- Placement of the method relative to recent work is very clearly explained
- Comprehensive theoretical analysis including an interesting hardness result, which is uncommon for the deep learning literature
- Comprehensive and convincing empirical analysis with clear results

Cons:
- There is very little discussion on how the construction of the base generator $f$ affects the results of the proposed method
- The proof of hardness is somewhat opaque and feels contrived; but this is often the case with hardness proofs!
- The method has a clear weakness in needing to be retrained for each observation. However, this is clearly stated by the authors and left open as a direction for future work.

Overall, I think this is an exceptional paper which makes a significant contribution to the field. I think it is suitable to accept as-is with only a few minor adjustments which I will enumerate below.

1. It would be nice (but not absolutely necessary) to see some discussion regarding the construction of the base generator, as I mentioned in the Cons above; e.g. does the performance of this method depend significantly on the user's choice of base model? Intuitively, I would think so.
2. A few notes on the proofs:
- The variable lower-case $m$ shows up in several spots in the hardness proof but is never defined. Perhaps these are typos and you meant to write $M$?
- In the proof for equation 3, the notation for expectations (i.e. $\mathbb{E}$) is inconsistent in a few places. Presumably just typos.
- I may be missing something, but it's not immediately clear why $y=T(x)$ can be substituted for the conditioner $x$ in $p_{\sigma}(\tilde{y}=y^*|x)$. My immediate intuition is that this would only be valid if $T$ is injective, otherwise this may change the underlying conditional density. Please correct me if I am wrong, and preferably add a clarification to the proof as to why this is justified.

Congratulations to the authors on a job well done!
review_title: A solid contribution to the field of normalizing flows and conditional generative models

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 
This paper is concerned with the approximation of conditional densities in trained normalizing flow models. The author use a variational Bayesian approach to estimate the conditional density of a set of variables given another set. As use cases, the authors present results in image inpainting and colorization. The paper also contains a theoretical result showing that exact conditioning is NP hard in additive normalizing flow models.

Pros:
- Relevant problem.
- Possibly interesting theoretical result.
- Interesting range of experiments with careful analysis of both quantitative and qualitative results. Large and appropriate list of performance metrics

Cons:
- Low originality, the main idea to perform inference in the latent space is obvious
- The relevance of the theoretical result in justifying the rest of the paper is not clear and not enough space is given to its explanation.
- The smoothing of the likelihood using a very rough noise model is not required
- More baselines are needed. In particular, it would be useful to have baselines without likelihood smoothing and also comparison with (simulation based) forward KL methods such as [1,3].

Relevance:
The problem of conditioning in generative models is highly relevant in our current ML environment as it allows to convert generators into very flexible inference machines capable of solving a large variety of problems. 

Originality: With the possible exception of the NP-hardness theorem, the original contributions of this work are very limited with some questionable elements (see below).  Conditioning deep differentiable models is a standard domain of application of variational Bayesian inference and the authors use a very standard and natural  approach. The main idea behind the method is to perform inference in the latent space, this in my opinion is not a noteworthy contribution as it is just the obvious way to do inference in this setting. The second methodological trick is to use a Gaussian smoothing of the likelihood. To my understanding, this procedure is neither well motivated nor supported experimentally. 

Major concerns:
I could be missing something but I do not understand why the authors are smoothing the likelihood since the flow already give a perfectly well-behaved joint model. The authors also seem to motivate this choice using their theorem showing that exact conditioning is NP hard. However this is only a valid motivation for using approximate inference, not for adopting an approximate likelihood. It seems that the authors want to get a least-square loss component in the pixel (or add-hoc feature) space. Without smoothing you will get a least square loss in the latent space which is likely to be much more appropriate. Therefore, I am not convinced that this work better in practice and it should at least be tested experimentally. 

Paper structure:
In general, the paper is well structured with a clear narrative and an appropriate amount of background material. 
However, the treatment of the theorem in section 3 should be expanded. The experiment section is very well structured and the analysis of the results is very good and definitely above average. I am pleased that the authors took the time to analyze the bad performance of the native VI method.

Writing:
The paper is very clearly written. 

Literature
The coverage of the literature is appropriate. However, the author should also discuss methods based on synthetic sampling and forward KL divergence as they are a very viable approach to generative model conditioning in the latent space. For example:

[1] Papamakarios, George, and Iain Murray. "Fast ε-free inference of simulation models with bayesian conditional density estimation." Advances in Neural Information Processing Systems. 2016.
[2] Le, Tuan Anh, Atilim Gunes Baydin, and Frank Wood. "Inference compilation and universal probabilistic programming." Artificial Intelligence and Statistics. PMLR, 2017.
[3] Ambrogioni, Luca, et al. "Forward amortized inference for likelihood-free variational marginalization." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

review_title: A standard use of stochastic variational inference with a not motivated use of likelihood smoothing

Meta-review: This paper proposes a method for conditional inference with arbitrary conditioning by creating composed flows. The paper provides a hardness result for arbitrary conditional queries. Motivated by the fact that conditional inference is hard the paper therefore suggests a novel relaxation where the *conditioning* is relaxed.

There were various concerns from the reviewers regarding notation, comparison algorithms, and how the hardness result motivates the smoothing operation introduced. After careful study of the paper and all the comments I find that I am most concerned about the hardness result and how it motivates the smoothing operation that is done. Novel computational complexity results *as such* are not really in the scope of ICLR. There's nothing wrong with having such a result in a paper, of course, but a paper like this should be evaluated on the basis of the algorithm proposed.

Like R4, I do not follow how this hardness result is meant to motivate the smoothing that's applied. The paper is unambiguous that the goal is to do conditional inference. A hardness result is presented for conditional inference, and so a relaxed surrogate is presented. This has a minor problem that it's not clear the relaxed problem avoids the complexity boundary of the original one. There's a larger problem, though. The hardness result has not been sidestepped! The goal is still to solve conditional inference. The algorithm that's presented is still an approximate algorithm for conditional inference. R4 suggests that other approximation algorithms should be compared to. The authors responded to this point, but I am not able to understand the response. For the same reason, I think it is valid to ask for comparison to other approximate inference algorithms (e.g. without smoothing)

None of the above is to say that the smoothing approach is bad. It may very well be. However, I think that either the existing argument should be clarified or a different argument should be given.

Finally here are two minor points (These weren't raised by reviewers and aren't significant for acceptance of the paper. I'm just bringing them up in case they are useful.)

Is Eq. 3 (proof in Appendix B.1) not just an example of the invariance of the KL-divergence under diffeomorphisms?

Proof in appendix B.2 appears to just a special case of the standard chain rule of KL-divergence (e.g. as covered in Cover and Thomas)
Decision: Reject