Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to bridge the concept of the normalizing flow model with the structured layers that encode the domain knowledge (also called inductive bias here). The general idea is to first notice that many existing probablistic programs (such as univariable RV) can be converted to flow under some special $f_{\phi}$.  Furthermore, the authors proposed a gated layer to allow the model to switch between user-specified model and the learned MAF. Extensive experiments in toy multimodality distributions, hierarchical gaussian, timeseries models and variational inferences are conducted to compare the proposed method with other baselines.


main_review: The strengths and weaknesses are pretty much discussed in the paper. Specifically,

The advantage of the method lies in the ability to control the flow model by injecting inductive bias. The whole model can be considered as a mixture of freely trainable MAF layers and fixed probabilistic programs, connected by the gated layers. The idea is novel and inspiring to me.  The whole idea looks natural, it could be considered as an augmentation of existing flow models that work well for small-sized data.

Several weaknesses that I've noticed:
1. The paper overall is easy to follow, however, I find the algorithm detail hard to read. Especially Figure 2, I encountered some difficulties in understanding the algorithm details as python. For instance, how as_bijector(gated=True) is implemented?
2. There are some typos in Section 3.1, when $C_{\phi}$ is introduced, it should be $\int_a^x$ right? and $a$ is undefined.
3. The gated layer requires root finding as mentioned, so exactly how much overhead is introduced? does this algorithm scalable to high dimensional data?
4. I noticed other methods such as Glow (Kingma et al.) not mentioned here, could authors explain why?
summary_of_the_review: Overall, I think this paper advances the research in this area, although more careful ablations are required to ensure the empirical results are indeed promising.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new type of normalizing flow for incorporating inductive biases into the model architecture. To this end, the authors introduce a so-called “structure layer”, aiming to transform a spherical Gaussian variable into a pre-defined probabilistic program. A modified version, called “gated structured layer” is proposed in order to skip problematic parts of the model. The model is evaluated on a variety of different datasets and outperforms the masked autoregressive flow baseline.
main_review: I do not count this as a weakness of the paper (but it should be changed): It seems that the paper is using the ICLR 2021 (last year) format because the page headers say “Under review as a conference paper at ICLR 2021”.
### Strengths
* Incorporating inductive biases into flow architectures is a very interesting and important problem to deal with.
* The method presented in section 3.1 of converting a distribution into a uniform distribution and the uniform distribution into another one sounds very interesting and applicable in other settings.
* The presented model outperforms the baselines in the presented experiments.

### Weaknesses
* The paper claims outperforming the state-of-the-art. I do not think that this conclusion can be drawn from the presented experiments:
* The paper compares the suggested architecture with two flow models, Inverse Autoregressive Flows (IAFs) and Masked Autoregressive Flows (MAFs), which were published in 2016 and 2017, respectively. Both baseline models are dated and do not constitute the current state of the art. In a more recent survey on normalizing flows ([1], p. 13), MAFs did not achieve state-of-the-art performance on any of the covered datasets.
* The datasets that the paper uses for evaluation are not common for assessing the quality of normalizing flow models. None of the datasets is used in [1]. In particular, neither the IAF paper nor the MAF paper use these datasets for evaluation. I recommend evaluating the presented model on some (simple) image datasets, like MNIST and CIFAR10, which are very commonly used in the normalizing flow literature.
* The conclusion states “We showed how, by choosing appropriate inductive biases, EMF can improve over generic normalizing flows on a range of different domains, with only a negligible increase in complexity …”
This conclusion should only be drawn if it can be supported by some concrete numbers. To the best of my knowledge, the paper does not mention anywhere how the complexity of the proposed EMF compares to the complexity of the baseline model. Indeed, presenting training and sampling times of the proposed and the baselines models would strengthen the contribution.
Clarity
* I have really tried to understand the methodology in the main section 3.2, but the section is still not clear to me. I understand the univariate case in section 3.1, but I cannot make sense of the main section 3.2.
In Figure 2, I do not understand what expressions like model_generator, gen.send and d.as_bijector mean. Moreover, the function forward_and_log_det_jacobian seems to be called within itself, but with only one argument instead of the 3 used in the definition. (The same applies to inverse_and_log_det_jacobian.)
What is the input and the output of the structured layers? What do the epsilon variables in e.g. equation (5) mean? Shouldn’t the epsilons only come from the flow latent space and not be present in intermediate layers? 
A toy example where each step of the proposed layers is explained for some easy problem would be really helpful. This could either be put into the appendix or replace Figure 1, which I did not find very illuminating.

**Minor and typos**
1. x_0 is not present on the left hand side of equation (2), so it should not be on the right hand side either.
2. On p. 3, in the sentence “The probability integral transform theorem states …”: The upper limit of the integral should be x and not y.
3. p. 2: “We call these architectures embedded-model flow s (EMF).” -> *flows*

[1] Normalizing Flows: An Introduction and Review of Current Methods, Kobyzev, Prince, Brubaker.

summary_of_the_review: Incorporating inductive biases into normalizing flow architectures is a very relevant and important problem. Some of the techniques presented in the paper seem interesting, but I really struggled to understand the main methodology part in the way it was presented. The paper claims to outperform the state-of-the-art normalizing flows. For the reasons stated above, I do not agree with this. The paper would benefit from further experiments on more common datasets and from a comparison with more recent state-of-the-art flow models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper first proposes to take a probabilistic program and compile it into a sequence of invertible transformations. Applying this invertible transformation to a standard normal distribution produces a sample from the probabilistic program.
This "compiled" invertible transformation is then used as a layer with learnable parameters, called a structured layer, in a larger flow model which can also have unstructured layers.
The paper argues that this enables the ability to incorporate domain knowledge via these structured layers while enabling deviation from this domain knowledge via the unstructured flow layers.
The paper performs experiments on multimodal, hierarchical, time series and variational inference data.

main_review: **Strengths:**
- Proposes a generic method to compile probabilistic programs into invertible functions given an ordering of the variables from the probabilistic program.

- Introduces a gating layer that allows the model to select whether to use the "prior" or not.  This is essentially a neural architecture search idea where the model is free to choose the prior or not.

- Shows better performance on structured VAE problems (particularly for the Lorenz system).

- Paper will provide code to actually do the compilation automatically from a probabilistic program. This could be generally useful to the community.

**Weaknesses:**
- There seems to be a disconnect between the probabilistic program (i.e., the domain knowledge) and flows.  As soon as the flow parts (i.e., MAF) modify the input to the EMF layer, the EMF layer no longer represents the probabilistic program, but rather than some distorted version of the probabilistic program. The gated part allows the model to completely ignore the prior if better. Thus, a key question remains: Why not just use a more state-of-the-art flow which can handle more arbitrary distributions? Why go through the trouble of carefully specifying a probabilistic model?

- Limited novelty in compilation approach: The idea of using conditional CDF functions is generally known as the "Rosenblatt transform".  Thus the construction of an invertible transformation to or from a Gaussian distribution via conditional CDF functions is not particularly novel.
[1] M. Rosenblatt, "Remarks on a multivariate transformation," Annals of Mathematical Statistics, vol. 23, no.3, pp. 470-472, 1952.

- The empirical results seem relatively weak (especially because the baselines seem weak).  It is unclear that the added work of incorporating domain knowledge significantly improves performance compared to just using more state-of-the-art flows. More details below.

    - Why not just use a more state-of-the-art normalizing flow (Neural Spline Flow, Residual Flow, Flow++, etc.)?  It is unclear what is the real advantage of EMF layers.  It seems that they add some modeling power but this extra modeling power could be added in different ways.

    - The multimodal part can be easily captured by Neural Spline Flows or more recent flow models than MAF as they have complex non-linear transformations that can model multi-modality.  Thus, it is unclear that the "multimodality" introduced by the EMF is caused by the "prior knowledge" about multimodality or merely because MAF is a very limited model and EMF is more complex. This would be significantly more convincing if this "prior knowledge" could be incorporated into models that already can model multimodality.  Overall, the experimental setup seems unfair as EMF introduces complex non-linearities that are unavailable to MAF (even with more parameters).

    - In the hierarchical experiment, again, it seems that even though the experimental setup favors hierarchical distributions, the MAF with more parameters does better.  Again, it's not clear the added benefit of EMF layers since they are not directly comparable to MAF (which has natural limitations).

    - In all these experiments, it seems that EMF should be the clear winner as the EMF incorporates domain knowledge that the baselines do not (i.e., it should have an unfair advantage because it uses external domain knowledge).  Yet, except for the Lorenz experiments, other simple baselines seem quite close.

- Limitation: The autoregressive transformation can be quite slow to compute in one direction.  This is why coupling layers have become quite useful in comparison to autoregressive invertible transformations.  The construction seems to inherit this computational issue with an autoregressive structure---i.e., if you want to use structured layers, you will be limited computationally in one direction.

**Other comments or questions**
- The gated architecture is useful for model power as the model can choose not to use the prior specification (i.e., the "bad" model).  However, this seems to be in partial opposition to the idea of incorporating prior knowledge---i.e., the goal of incorporating prior knowledge is to improve performance by biasing the model towards more appropriate solutions.  Thus, can these domain-specific or prior-specific models just be considered "hints" to help the algorithm find the best fitting model?

- Could the interpretability of the model be better because of these structured layers?  This could be an added benefit but it is unclear if that is true if the input and output of the structured layer can be arbitrarily manipulated by unstructured flow layers.

- Why was the dataset augmented in the hierarchical experiment?  Shouldn't the hierarchical EMF layer be able to learn the hierarchy automatically from the data?

- The Lorenz system seems to show the most improvement.  The BR and OU time series only show a marginal improvement of negative log likelihood.  Why is that?

- Regarding time series experiment, while the base distribution of B-MAF follows the continuity equation, why not also make the masked flow layers obey this ordering of the variables (as I believe MAF generally assumes a random order)? This would be a simple and straightforward way to incorporate the ordering of variables in the flow part of MAF.

summary_of_the_review: **Update since author response**:
I thank the authors for their very thoughtful response.  I greatly appreciated the new experiments and more discussion on why the structured layers are simple and have fixed parameters for essentially simple inductive biases.  Also, I appreciate the new NSF experiments.  Overall, I felt these responses answered most of my major concerns.   I have updated my score accordingly.  I would encourage the authors to incorporate some of this discussion in the paper if accepted.

----
Overall, I think the idea of the paper seems reasonable (i.e., to compile known domain structures into flows). However, the compilation technique using conditional CDFs is relatively well-known as the Rosenblatt transform, and it is unclear why a practitioner would not just want to use a more state-of-the-art flow instead of spending time to construct a probabilistic model.  The empirical results are unconvincing that a SOTA flow (even with similar number parameters) wouldn't outperform the current approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors describe how to translate a probabilistic program into a normalising flow layer which maps samples from a unit Gaussian into samples from the probabilistic program. They propose a "gating" mechanism for this layer so that it can learn to interpolate between this transformation and the identity transform. They then show how this layer can be combined with generic normalizing flow layers, and in this way add an inductive bias to a normalising flow without sacrificing its expressivity. In the experiments, they provide various examples using probabilistic programs in this way to construct normalising flows with helpful inductive biases.
main_review: ### Novelty
Although it is well-known that a probabilistic program (with deterministic control flow and continuous random variables) is reparameterisable with samples from a unit Gaussian, I believe that using this insight to construct a normalising flow layer is a novel and interesting idea. The "gated" layers introduced add further novelty (while being demonstrably useful). I am therefore confident that this paper makes sufficiently novel contributions.

### Strengths
- Incorporating the known structure/properties of complex distributions into usually black-box normalising flow architectures is a well-studied problem, which I suspect is important for many real-world applications. This paper presents a general solution applicable whenever a probabilistic model for the distribution can be written, even in the common case where it is only approximately correct (or e.g. only the prior is known but we are modelling a posterior).
- Experiments demonstrate the method on various distribution-modelling tasks, as well as its use for variational inference. Results are generally positive for the proposed method.
- The paper is mostly well-written.

### Weaknesses
- The experiments are all fairly small-scale with in the order of 10s of random variables. It would be interesting to see larger scale applications but I don't think this is necessary for publication.
- The method has some limitations (correct me if I'm wrong): (1) All variables must be continuous. (2) The probabilistic program must sample the same random variables on every execution. (3) The probabilistic program must be executed whenever the normalising flow is run (which could be a problem if the probabilistic program is slow to execute). There are not critical issues (and are shared by much related work), but are probably worth mentioning in the discussion.

### Minor
- In Section 5.2, it is misleading to say that "GEMF-T outperforms the baselines on all the problems", since the baselines obtain the best performance on two of the results in Table 5 in the appendix.
- Confusing equation: my understanding from Section 3.2 is that you set $x_{t+1} := f_{t+1, \sigma}(\epsilon_{t+1}; x_t)$. Why do you replace $\epsilon$ with $x_{t+1}$ in $f_{t+1, \sigma}(x_{t+1}; x_t) = x_t + \sigma x_{t+1}$ on the line below equation 13?
summary_of_the_review: I'm recommending acceptance because this paper makes a novel, yet simple and generic, way to impose inductive biases on normalising flows. It is validated by experiments in various domains.
review_title: null

Meta-review: This paper proposes a method for incorporating inductive biases into the model architecture of normalizing flows through a suitable probabilistic program. All reviewers agree the paper makes an interesting contribution to the growing normalizing flow literature. The paper is well written and the idea is novel. Additionally, the experimental results are promising and the additional experiments and baselines added during the rebuttal further strengthen the paper. I recommend acceptance.
Decision: Accept (Poster)