Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper uses the energy loss for training normalizing flows. This produces a consistent estimator that doesn't require computing determinants of the Jacobian of the transformation, making it more flexible in terms of the choice of architecture. The paper also introduces a few tricks to further improve scaling and/or performance: slicing the data using 1D projections, and a semi-autoregressive flow model that interpolates between a full autoregressive model and a completely parallel model (i.e., a typical normalizing flow). Experiments on UCI and handwritten digit datasets show favorable results compared to baseline flows, VAEs, and autoregressive models, while allowing for fast sampling and exact posterior inference.
strength_and_weaknesses: Strengths
- The energy loss is straightforward, faster to compute, and produces good samples.
- Slicing provides an effective way of scaling the training.
- Semi-autoregressive flows elegantly trade-off computation and expressivity.
- Exact posterior calculation is possible, allowing for things like interpolation that is not easy to do in autoregressive models.

Weaknesses
- The experiments are only done on UCI and handwritten digit datasets. At the very least, CIFAR-10 would have been good.
- Comparisons to larger-scale flow models like GLOW would be more convincing.
- There are key differences between the behavior of the energy loss and NLL (they are not strictly correlated). More exploration of where they differ in terms of allocation of probability mass would be nice.
clarity,_quality,_novelty_and_reproducibility: 
The paper is reasonably easy to follow, at least on a high level. The paper is very heavy on the use of acronyms, which makes it a little difficult to follow for one who is not intimately familiar with all of these acronyms and their variations. A section on each acronym used in the tables and a brief description of them would be helpful. Also, in the case of CRPS vs U-CRPS, a mathematical definition would be useful as opposed to just a textual description.

Minor nitpicks:
- Table 5 should be lower down, the current draft has it above Table 4.
- Appendix typo: "the above **clam** also holds"
summary_of_the_review: The use of the energy loss is compelling, and the background behind it is nicely written. This approach frees normalizing flows from their previous architectural constraints, and this paper explores this idea through the introduction of a semi-autoregressive flow model that interpolates between faster sampling and better sample quality.

I think that the ideas certainly have merit, however it feels like there is a bit of a gap between the promise of the paper and the actual demonstration of the approach. Specifically, the experiments are very small-scale for 2022, only operating on UCI datasets and MNIST. I'm surprised that CIFAR-10 wasn't included, as it is commonly used as a baseline for generative models. For example, one method referenced and compared with (Papamakarios et al., 2017) uses CIFAR-10, and it is now 5 years old. Normalizing flows (e.g., GLOW),  PixelCNN, and VAEs have all been scaled to more challenging datasets many years ago.

I certainly appreciate a careful and detailed study of the estimator and its effects on small-scale data, however the conspicuous lack of larger-scale experiments tell me that either the authors have not yet run these experiments, or there is a severe limitation that is keeping the model from working on those datasets.

I personally appreciate new and elegant ideas, and so all I'm asking for is an attempt on something a more challenging baseline. If the results are negative, then a discussion about reasons for this, and avenues for improvement would be necessary. The intention is to ensure that this paper is carried forward and does not languish because other researchers find it cannot scale well.

If there is a strong reason why the existing experiments are sufficient, then I would be happy to discuss this with the authors.

Minor comments:
I think you should add additional references for normalizing flows.
Laurent Dinh, David Krueger, Yoshua Bengio, "NICE: Non-linear Independent Components Estimation", ICLR 2015
Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, "Density estimation using Real NVP", ICLR 2017
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel strategy to train invertible neural networks that model a distribution, a.k.a. normalizing flows. In particular, the authors propose to use proper metrics as the objective function to train flows. This approach relaxes the requirement of computing the determinant of the Jacobian of the flow and unlocks architecture for which computing this determinant would be too computationally demanding. 

Authors strongly motivate their approach with theoretical arguments for the soundness of proper metrics. They also validate their approach experimentally, showing improvements in the sample quality compared to likelihood-based training. 
strength_and_weaknesses: # Strengths
- The idea presented in this paper is exciting and somehow novel. It efficiently trains invertible architectures while keeping the potential advantage of their invertibility at test time. The generalisation of the ideas from Si et al. (2022) to non-autoregressive architectures is well-exposed and has a practical interest.
- The empirical results demonstrate that using the energy metric to train the flow is a good objective function if one is interested in good sample qualities (although it is not always clear how we define good samples) or by the marginal densities (as well represented by the CRPS score).

# Weaknesses
Although I think the ideas presented in the paper make sense and could have a real interest for some practical use cases, I find the paper's organisation and way of presenting ideas confusing. For instance, the tables' order does not follow the structure of the paper and the captions are uninformative, which makes it hard for the reader to follow the results at first glance. In addition, the discussions in section 3 go in many different directions and are missing a link between them. It is only at a second read that we can more or less grasp what each paragraph brings to the paper.

I am also a bit confused by the argument from the authors that autoregressive architectures lead to "good sample quality". They can be good indeed. Still, I do not see why they are better than other generative models in terms of sample quality. 

The authors say that MAF leads to gaussian distribution, which is only valid for a 1-step flow but is immediately relaxed if there are dependencies between variables and one uses more than one step of the flow. Indeed, MAF allows non-linear interaction between the variables. Thus the corresponding joint and marginal distribution can be non-Gaussian.

I also think the author should mention that CRPS only reflects the quality of the marginal distribution and cannot say whether the model captures dependencies well.

At some point, the authors say that using the sliced objective (which I find a neat idea) reduces the computational complexity. Of course, this is true. However, it is unclear to which extent this is necessary, I would have liked to see numbers for that.

I recommend that the authors create a better structure and story for this paper. I do think the ideas are excellent and (probably) well-motivated theoretically. However, the article in its current form is not pleasant to read, which I find sad for the quality of the ideas presented.


clarity,_quality,_novelty_and_reproducibility: See my comments above.
summary_of_the_review: Although the idea and experiments presented in this paper are interesting. I find the presentation too poor to argue for acceptance. Hence I argue for a weak reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes training normalizing flows using sample-based objectives. 
strength_and_weaknesses: 
Weakness:
  - The use of IPM are standard in the GAN literarture. From what I understand, the main thing separating this work from those training invertible generative models with a discriminator (e.g. [1]), is that there are no parameters in this objective. In fact, the CPRS objective (Eq 3) has the same gradient as MMD, so the only reason this isn't adversarial is because unlike the MMD-GAN objective, this work does not make use of a neural network-defined kernel function.
  - Related to the above, the only experiments are in tabular and MNIST, where I imagine simple kernels and distance metrics suffice. But for training on more complex distributions like colored images, a neural network discriminator is crucial. 
  - Furthermore, this work does not convincingly show that training without a discriminator is sufficient, since Flow-GAN [1] is not compared against.
  - In terms of evaluation metrics, it would be better to use an evaluation metric that is different from the training objective for SAEFs, which heavily biases towards SAEF. Have the authors considered metrics such as MMD, NLL, or a kernel-based density estimator [2]?

[1] "Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models." Grover et al. (2018)

[2] "On the Quantitative Analysis of Decoder-Based Generative Models." Wu et al. (2016)
clarity,_quality,_novelty_and_reproducibility: Novelty:
  - Using IPMS to train normalizing flow has been explored before [1]. 
  - The use of slicing seems interesting, but it isn't clear if this was used in the experiments. Furthermore, this seems similar in spirit to using random features for kernel estimation [3]. 

Clarity:
  - This paper discusses multiple objectives for training. Which objective(s) were used for training the models in the experiments? Were the baselines trained with the same objective (CPRS) or maximum likelihood?
  - I feel like NLL values should be reported in Table 5, since all of these are invertible and can compute log-likelihood values. 
  - The NLL values should be negated in Table 4.

Comments:
  - Can the inverse of SAEF be computed? Perhaps not analytically, but an iterative procedure [4] should work.
  - SAEF's log probability can be computed by computing the determinants of each block (which I assume is how the NLL estimates are computed in Table 4). Have the authors tried training with maximum likelihood and comparing it to the sample-based objectives? 

[3] "Random Features for Large-Scale Kernel Machines." Ali Rahimi and Ben Recht. (2007)

[4] "Mintnet: Building invertible neural networks with masked convolutions" Song et al. (2019)
summary_of_the_review: Since the paper focuses on sample quality and sample-based metrics, I feel the experiments on only tabular and MNIST are not sufficient. Furthermore, I'm not convinced this is better than the existing variety of GAN objectives such as MMD-GAN. The main difference seems to be the lack of a neural network for defining a kernel function; however, human-designed metrics tend to scale poorly to more complex data sets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author's establish a novel method to train normalizing flows without the need to compute the likelihood. Thereby, the computation of the Jacobian determinant of the flow is avoided, which can be expensive or requires a restricted architecture to be cheap to compute. They propose objectives inspired by two-sample tests, such as the energy loss, and show how they can be estimated and used for training. This objective can be used to train a variety of flows, and the authors focus on applying it to flows having Jacobian determinants that are expensive to compute. Moreover, they introduce a novel flow architecture, called semi-autoregressive flow, that can only be trained with their energy loss.
In their experiments, the authors demonstrate that their approach is competitive with other training procedures on UCI and MNIST datasets, while being often faster to train and evaluate.
strength_and_weaknesses: ### Strengths

Computing the Jacobian determinant of normalizing flows is a challenge, but is required for most existing training procedures. Often, the architecture is designed such that the computation becomes simple or approximations are used, but this can limit expressivity and accuracy. The authors address this challenge in a creative way by proposing a novel training procedure based on two-sample test objectives.
Furthermore, they demonstrate that such a training procedure unlocks new opportunities in terms of flow design by introducing a novel flow model that can only be trained with their method.
They apply their method to flows of various architectures and, thereby, demonstrate its effectiveness. Important is the runtime comparison, since the superior runtime is one of the main arguments for using their method.

### Weaknesses

My main criticism concerns the experimental evaluation. The authors apply their method to various normalizing flows for which alternative training procedures exist, but the baseline models trained via maximum likelihood have a different architecture. There should also be experiments with the same flow architecture but different training methods being used. Moreover, the use of the CRPS and the NLL as evaluation objectives make it difficult to compare the work with other articles in the field, where the likelihood (UCI) or bits per dim (MNIST) are the main evaluation metrics being used. Computing them might be expensive for some of the models, but this is important information to evaluate the method.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and well organized. The method is novel and the argumentation correct. My main concern regarding the quality of the paper is that the experimental evaluation of the method has flaws that I mentioned above.
summary_of_the_review: Due to the importance and the novelty of this method, I tend towards accepting this paper. However, I ask the author to address the concerns I voiced regarding the experimental evaluation

Edit after rebuttal:

I thank the reviewers for their response and additional experiments. I am still inclined to accept the article.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to use energy loss for training normalizing flows, avoiding the computation of determinants of the Jacobian of the transformation. 
 
Main strengths:
- Well-motivated approach for using energy loss for training normalizing flows.
 
Main weakness:
- The experiments do not sufficiently validate the promise of the paper. The evaluated datasets are of a very small scale. Even results from CIFAR-10 are not included. 

The reviewers appreciate the authors' responses but still find the missing larger-scale evaluation concerning. 

The AC agrees with the reviewers that the paper is not yet ready for publication. The AC thus recommends to reject.


justification_for_why_not_higher_score: The paper does not include sufficient experimental validation to validate the premise of the proposed approach.
Decision: Reject