Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:
This paper conjectures that normalizing flows are fundamentally limited due to the architecture assumption that the generative function g is continuous in x. It is argued that this constraint makes maximum likelihood estimation difficult in general. Localised generative flows are proposed as a solution and consist in modeling the generative model as a continuous mixture of bijections. Experiments suggest an improvement over MAF.

Decision:
The observation that continuity imposes a hard constraint on the network is sound, and the proposed solution appears to show some improvement. However, in its current state, this work appears to be quite fragile both from a theoretical and experimental point of view. First, it is only conjectured that this constraint poses actual problems. Second, the experimental evaluation is weak and insufficient. It omits comparisons with more recent generative flows that have shown to be able to model discontinuous densities. For this reason, I do not recommend the paper for acceptance.

Further arguments:
- The whole paper rests on intuition without strong theoretical backup. 
- The experiments are quite poor and results frankly oversold. It is said the method "improves performance across a variety of common density benchmarks". While we see improvements in Table 1 over MAF, the comparison omits all recent architectures based on Normalizing Flows, such as TAN (Olivia et al, 2018), NAF (Huang et al, 2018), B-NAF (De Cao et al, 2019) or SOS (Jaini et al, 2019). All of those methods have reported better results than those provided in Table 1. They have also been shown empirically to work for discontinuous densities. While I understand that LGF can be combined with any flow architecture, the question remains whether using a continuous mixture translates into significant improvements for those baselines as well. The experimental benchmarks also omit datasets such as BSDS300, for which the higher dimensionality is usually challenging. The same goes for Table 2 which omits recent and better results, such as Glow or FFJORD.
- Closer to LGF, a proper experimental comparison to RAD (Dinh et al, 2019) would be appreciated.
- The proposed architecture supposedly enables better generative models. However, this comes at the price that the density can no longer be evaluated exactly and analytically. Since normalizing flows are also typically slow for sampling, this makes the benefits of the proposed architecture quite limited. In particular, it is not clear why generative models that are good at sampling only (e.g., GANs) should not then be preferred?
- As a result of the point above, the experimental results are reported only in terms of approximated negative log-likelihood. I do not think this is fair, since models like MAF do provide exact values. It also makes the comparison with previous methods more difficult.

Further feedback:
- As per ICLR policy, higher standards should be applied to papers with 9 or more pages. I am confident the paper could be written within 8 pages only.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose to extend flow-based density models by replacing a single bijection with a hierarchical mixture of bijections. Each component in the mixture is then required to only push the prior onto a local region; this helps improve the coverage of $\mathcal{X}$. This is motivated by the conjecture that in many cases the topology of $\mathcal{X}$ might be overly complicated to be effectively captured by a single bijection. Formally, this is achieved by introducing a conditional random variable $U|Z$. In doing however, the log-likelihood is rendered intractable and a variational approximation must instead be resorted to. A recursive formula for computing the ELBO is introduced in this vain. 

Overall, I think this is a generally interesting contribution to the normalizing-flow literature that I expect to spark further research. However, there are some rough edges to this paper. The initial motivation is well-presented and relatively easy to follow, though a diagram would serve to cement the intuition regarding the support mismatch. The issue mentioned in footnote 2 deserves further discussion. At the same time, while well-reasoned, their justifications are nonetheless largely conjectural and further theoretical or empirical evidence would be welcome, both for characterising the pathologies they aim to redress and their proposed solution.

For the experiment section, I would have liked to have seen comparisons not only to the simplest baseline but also to some of the other methods mentioned in related works. In general, the experiment section is quite short and I didn't get a very good sense of how well this method performs.

The following should be addressed:

- provide more evidence for the conjectures surrounding the motivation and derivation
- supply more varied baselines (e.g. RAD model)

Minor comments:

- at the top of page 4, you refer to $p_{U|X}$ several times, but I think you mean $p_{U|Z}$
- in the paragraph after equation (2), the $\theta$ superscript seems to be missing from the $p_X^\theta$
- in the first sentence of section 3.1, you refer to "the single $g$ used in equation 2", but equation 2 mentions no $g$
- in the first line on page 3, you talk about some region of supp $p_X^\theta$ being pushed out of supp $p_X^*$, shouldn't this be the other way round since the KL is infinite only if the the support of $p_x^*$ is not contained within the support of $p_X^\theta$?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper introduces a straight-forward way to expand the flow models by considering mixture of flow distributions. The idea is not very novel since several previous work have tried the mixture of flow such as the mentioned RAD and Deep Mixture. The paper studies some further improvements such as using the continuous auxiliary variable and stacking multiple mixture layers. 

The major concerns are the following:

1) The paper tries to solve a “problem” build upon intuition. The paper explains as “the normalizing flow places global constraint on the bijection”, “it need to match the topology of  X to the topology of Z ”, “continuous functions necessarily preserve topology”. What kind of topological properties are referred to here? Are all topological properties preserved under continuous function? It needs to be more accurate when using such terminologies. The intuition of the paper is weak and heuristic. The example in Figure 1 can potentially be easily solved with a two component Gaussian mixture of input Z to a vanilla flow model.

2) For the mixture p(X), will the proposed method generate samples concentrated on one or some of the components? Why or why not?

3) Considering there are a plenty of improvements of flow models, it is neccesary for the proposed method to compare with, at least for some methods explained in the Related Work section.

4) Since the proposed methods inevitably lose the advantage of analytic density property of flow methods, it is better to show some advantage over implicit or semi-implicit methods. For example, (https://arxiv.org/abs/1805.11183) also uses a hierarchical model with continuous auxiliary variables and a marginalization similar to Eq.(4) in this paper. How does the proposed method related to or compared to these methods?

5) In experiment section, in Table 1, the proposed method is worse than MAF for 2 datasets out of 4? In Table 2, what are the numbers refer to?

6) On page 4, it should be p(U|Z) not p(U|X)?

In sum, I think though the paper makes contribution on exploring better flow models but the novelty is relatively weak, the discussion and comparison of related work is insufficient and the experiments are not convincing or have mistakes. I think a modification is necessary before publishing.

################

I have read the author's feedback.


review_title: Official Blind Review #2

Meta-review: This paper proposes to overcome some fundamental limitations of normalizing flows by introducing auxiliary continuous latent variables. While the problem this paper is trying to address is mathematically legitimate, there is no strong evidence that this is a relevant problem in practice. Moreover, the proposed solution is not entirely novel, converting the flow in a latent-variable model. Overall, I believe this paper will be of minor relevance to the ICLR community.
Decision: Reject