Semi-Autoregressive Energy Flows: Towards Determinant-Free Training of Normalizing Flows
Authors: Phillip Si
Zeyi Chen
Subham Sekhar Sahoo
Yair Schiff
Volodymyr Kuleshov
Venue: Submitted to ICLR 2023 (2023)

Abstract: Normalizing flows are a popular approach for constructing probabilistic and generative models. However, maximum likelihood training of flows is challenging due to the need to calculate computationally expensive determinants of Jacobians. This paper takes steps towards addressing this challenge by introducing objectives and model architectures for determinant-free training of flows. Central to our framework is the energy objective, a multidimensional extension of proper scoring rules that admits efficient estimators based on random projections. The energy objective does not require calculating determinants and therefore supports general flow architectures that are not well-suited to maximum likelihood training. In particular, we introduce semi-autoregressive flows, an architecture that can be trained with the energy loss, and that interpolates between fully autoregressive and non-autoregressive models, capturing the benefits of both. We empirically demonstrate that energy flows achieve competitive generative modeling performance while maintaining fast generation and posterior inference.

SEMI- SEMI-AUTOREGRESSIV ENERGY FLOWS: TOWARDS DETERMINANT-FREE NORMALIZING FLOWS
Anonymous authors Paper under double-bling review
ABSTRACT
Normalizing flows popular approach for constructing probabilistic and generative models. However, maximum likelihood training of flows is challenging due the need calculate computationally expensive determinants of Jacobians. This paper takes steps towards addressing this challenge by introducing objectives and model architectures for determinant-frre training of flows. Central to our framework is the energy objective, a multidimensional extension of proper scoring rules that admits efficient estimators based on random projections. The energy objective does not require calculating determinants and therefore supports general flow architectures that are not well-suited to maximum likelihood training. In particular, we introduce semi-auttoregressivv flows, architecture that can be trained with the energy loss, and that interpolates between fully autoregressive and non-autoregressive models, capturing the benefits of both. We empirically demonstrate that energy flows achieve competitive generative modeling performance while maintaining fast generation and posterior inference.
INTRODUCTION
Normalizing flows are one of the major families of probabilistic and generative models (Rezende and Mohamed, 2015; Kingma al., 2016; Papamakarios et al., 2019). They feature tractable inference and maximum likelihood learning, and have applications in areas such as image generation (Kingma and Dhariwal, 2018; Dinh et al., 2014), anomaly detection (Nalisnick al., 2019), and density estimation (Papamakarios et al., 2017; Dinh et al., 2017). However, flows require calculating computationally expensive determinants of Jacobians order to evaluate their densities; this either limits the range of architectures compatible with flows, or makes flow models with highly expressive neural architectures slow to train.
This paper seeks to question the use of maximum likelihood for training flows and instead explores an approach for determinant-free training inspired by twopsample testing and the theory of proper scoring rules. See Appendix G for more detailed motivation. Si et al. (2022) recently showed that normalizing flows can be trained using objectives derived from proper scoring rules (Gneiting and Raftery, 2007a) that involve only samples from the model and the data distribution (hence do not require computing densities and departs from log-likelihood based training of autoregressive models as (Papamakarios et al., 2017)). Although quantile flows (Si al., 2022) are determinant- they are also necessarily autoregressive due to the CDF only existing in one dimension, and therefore inherit various limitations, such as slow sampling speed.
Here, we extend the sample-basse based proper scoring rule framework of Si et al. (2022) to models that are not fully autoregressive. Central to our approach the energy objective, multidimensional extension of proper scoring rules that only requires model samples, and not densities. We complement this objective with efficient estimators based on random projections and compare against alternative sample-based objectives that serve strong baselines. We examine the theoretical properties of our approach, draw connections to divergence minimization, and highlight benefits over maximum likelihood training.
Our framework enables training model architectures that are more general than the ones compatible with maximum likelihood learning (e.g., densely connected networks). In particular, we propose semi- autniiautoreressivve flows, an architecture trained with the energy loss that integrates the speed of feed-forward architectures with the sample quality of autoregressive models. Across number

---

Table I: Energy Flows and Semi-Auttreggressvve Energy Flows (SAEFs) are invertible generative models that feature expressive architectures, exact likelihood and posterior evaluation, and their training does not require computing -og-determinants, in contrast to VAEs (Kingma and Welling,
2014), MAFs (Papamakarios et al., 2017), NAFs (Huang et al., 2018), AQFs (Si et al., 2022), GMMNets (Li et al., 2015), and CramerGANs (Bellemare et al., 2017).
Method
Likelihood
Posterior
Sampling Feedforward
Representation
Objective
VAE
Approx.
Approx.
Gaussian
ELBO
MAF
Exact
Exact
Autoregressive
Gaussian
Likelihood
NAF
Exact
Exact
N/A
Neural
Likelihood
AQF
Exact
Exact
Autoregressive
Neural
Quantile Loss
GMMNet
N/A
N/A
Feedforward
Neural
MMD
CramerGAN
N/A
N/A
Feedforward
Neural
Diser+Ennergy
Energy Flow (Ours)
Exact
Exact
Feedforward
Neural
Energy Loss
SAEF (Ours)
Exact
Exact
Autoregressive
Neural
Energy Loss
of generative modeling tasks, our method produces high quality samples, while supporting exact posterior inference, which we demonstrate via an exploration of the models' latent space. Table 1 compares our approach to existing methods.
Contributions. In summary, this work (1) questions the use of maximum likelihood for training flows and proposes an alternative approach based on proper scoring rules and two-sample tests that extends quantile flows (Si et al., 2022) to multiple dimensions. We (2) introduce specific two-sample objectives, such as the energy loss, and derive efficient slice-based estimators. We also (3) provide theoretical analysis for the proposed objectives as they are consistent estimators and feature unbiased gradients. Finally, we (4) introduce semi-autoregressive architecture that features high sample quality and speed on generation and posterior inference tasks.
BACKGROUND
Normalizing Flow Models Generative modeling involves specifying probabilistic model p(y) A (RRd) over a high-dimensional y E IRd (Kingma and Welling, 2014; Goodfellow et al., 2014) normalizing flow is generative model p(y) defined via an invertible mapping f Rdd + R
between a noise variable z E R2 sampled from a prior z ~ p(z) and the target variable y (Rezende
and Mohamed, 2015; Papamakarios et al., 2019). We may obtain an analytical expression for the
likelihood p(y) via the change of variables formula p(y) af(x)
8z p(z). where af(())
denotes
Oz
the determinant of the inverse Jacobian of f. Computing this quantity often expensive, hence we
typically choose f to be in class of models for which the determinant is tractable (Rezende and Mohamed, 2015), such as in autoregressive models (Papamakarios et al., 2017).
Proper Scoring Rules Consider score loss f ARR)) Rd IR+ over probabilistic forecast E A (IRd) and a sample y E RRd The loss is proper if the true distribution G E arg minF CE F, y) (Gneiting and Raftery, 2007a). A popular proper loss is the continuous ranked probability score (CRPS), defined for two cumulative distribution functions (CDFs) F and G as CRPS(F, G) (F(g) - G(v))² dy. When we only have samples from G, we can generalize this score to obtain the following loss for single sample y': CRPS, 3/) J, (F(2) 1yy 3/))² dy. where denotes the Heaviside step function. The above CRPS can also be written an expectation relative to the distribution F:
CRPS (PPS(F.))) v/) = EE - Y' E v/1.
-
1
where Y are independent copies of random variable distributed according to F. Recently. Si al. (2022) proposed autoregressive quantile flows, which are trained using the CRPS and are determinant-frre. We seek to extend the approach of Si et al. (2022) beyond autoregressive flows.
Two-Sample Tests and Integral Probability Metrics Two-sample tests compare distributions F, G based their respective sets of samples Dr {m( )m and De {x(0)m j Specifically,
2

---

wwo-sample test defines statistic T I R; determine whether Dp, De originate from identical or different distributions F, G based on differences T across Dp, Dg. Two-sample tests motivate objectives for generative models such generative moment matching networks (GMMNs; (Dziugaite et al., 2015; Li et al., 2015)) and generative adversarial networks (GANs; (Goodfellow al., 2014)). Two-sample tests also attractive training objective for flows: because they density-free, they do not require computing determinants (Grover et al., 2018).
More modern approaches include integral probability metrics (IPMs) (Müller, 1997), which take the form I Ey [T[v]] - Eny-c (v)], where 7 is a family of functions. A special case of IPMs maximum mean discrepancy (MMD) (Gretton et al., 2008), in which T {TT ||T||x x 1) the
set of functions with bounded norm in reproducing kernel Hilbert space (RKHS) with norm || ||
the CRPS objective can be shown to be form MMD (Gretton et al., 2008).
EXPLORING DETERMINANT-FRES TRAINING OF NORMALIZING FLOWS
We propose training normalizing flows using objectives inspired by two-sample tests, which do not require computing densities. This idea poses two sets of challenges: (1) most classical two-sample tests (e.g., Kolmogorov-S (orrov---mrrv are defined in one dimension and do not have simple multivariate extensions; (2) modern two-sample tests (e.g., IPMs) extend to high dimensions, but typically require solving a costly optimization problem. Here, we derive two-sample tests that form good learning objectives, and use the theory of proper scoring rules to justify their validity.
3.1 HAMPLE-BASER TRAINING OF NORMALIZING FLOWS AND THE ENERGY OBJECTIVE
We seek to extend autoregressive quantile flows (Si et al., 2022) to general structure without the limitations of autoregressivity (e.g.. slow sampling). Specifically, we leverage a generalization the sample-based form of the CRPS objective (1) by Szekely (2003); Gneiting and Raftery (2007a):
version called the energy score
CR, = - -F --
- Y'1 + E y'1l.
(2)
where 3 E 2). || l12 denotes the Euclidean norm, and Y, Y' E TRd are independent copies of vector-valued random variable distributed according to F. The rightmost term EF Y - y'
promotes samples Y from F that are close to the data point y': the leftmost term _ encourages the model to produce diverse samples and not concentrate all probability mass on one
Y Y'
The Kernelized Energy Objective As generalized extension the CRPS, the Kernelized Energy Objective extends the Euclidean norm kernel function:
CRPS (F.y') = KYY, Y') EyK(YYY'),
(3)
The kernelized energy loss can be shown to be a proper loss (Gneiting and Raftery, 2007a), and thus represents valid training objective for generative model. The flow objective consists of Ey.=D(CPPSS (F,y)) and also reveals the connection to two-sample tests between and D.
Two-Sample Baselines In Appendix D.I, We define two classical statistical tests as baselines and to illustrate examples of alternative methods that can be derived from our two-sample-based approach. In brief, Hotelling's two-sample test uses the statistic H2(Dr; Da) (m - -ma) 5 (mp mc). where mF, mg sample means, Sp, SG sample variances and S SS)25 The Fréchet distance uses the objective R(Dp, Dc) = || ma 12 + tr(SF + Sc - 2(8F where we are using the same notation.
3.2 THEORETICAL PROPERTIES
Divergence Minimization is precisely equivalent the Cramér divergence ( G) f (P(y) dly) between distributions F, G. Szekely (Szekely, 2003) showed that the one-dimensional version of the energy
When the variable y E R is one-dimensional, the objective CRPS G)
3

---

loss (2) is precisely equivalent lg. Its kernelized version can be shown to also represent valid divergence between distributions (Gneiting and Raftery, 2007a). The connection to divergence minimization lends additional support to using (2). (3) as principled objectives- -f G is the data distribution, minimizing (2), (3) over a space of models will produce a model F that is close to G.
Unbiased Gradient Estimation Our objectives have the property that given a sequence YK K of K samples Y1, .... YK from G, the gradient of the empirical distribution over these samples yields an unbiased estimate of the gradient of the expected loss: VoEx ((Fo, Gx) = Vol(Fa, G), where l is one of objective functions, Gx is the empirical distribution over Y K and Fe is model with parameters that we are optimizing. The above fact follows directly from the fact that both the energy and the CRPS objectives proper scoring rules (Bellemare et al., 2017).
Why Energy Objectives? Consider the set of objectives (FF,,) = [ P for p 2 over y E IR; these are also known as Wasserstein p-metrics (Kantorovich, 1939). The energy objective corresponds
B3, and it is the only l objective support unbiased gradients (Bellemare et al., (). In high dimensions, IPMs are general-purpose two-sample tests; popular IPMs include the Kantorovich metric (Kantorovich and Rubinstein, 1958), Fortet-Mourier metric (Fortet and Mourier, 1953), the Lipschitz (or Dudley) metric (Dudley, 1966), and the total variation distance. In general, IPMs are defined terms of potentially costly optimization problem; out of the aforementioned IPMs, only the energy objective has a known analytical (optimization-frey free)) solution, and it also features a faster statistical convergence rate (Sriperumbudur al., 2009).
Overall, we summarize the above facts part of the following formal result.
Theorem 1. The energy objectives (2) and (3) consistent estimators for the data distribution and feature unbiased gradients.
This follows from properties of proper scoring rules and MMD; see the Appendix for full proof.
3.3 SCALING SAMPLE- SAMPLE-BASED FLOW OBJECTIVES USING PROJECTIONS
The framework of IPMs provides a wide range of high-dimensional sample-based objectives (Müller, 1997). However, most of these objectives involve costly optimization problems, with the energy loss being a rare exception. At the same time, there exist many popular one-dimensional two-sample tests that have appealing statistical and computational properties and can yield training objectives.
We propose further improving our objective via random projections, specifically slicing, which projects data into one dimension (Kolouri et al., 2019; Song et al., 2019; Nguyen et al., 2020). Formally, define sampling probability p(v) over one-dimensional vectors Rdd We define sliced version of a ane-dimensional loss function L(x, y) R R
R as
(x( y) = Evepp(v)
[L(v x, v y)]
We approximate the expectation with a number of Monte- Carlo samples.
Sliced Energy Objectives The sliced energy objective applies Equation to the projected data. In practice, we find that the number of slices needed for good performance lower than the dimensionality of the data, resulting in favorable computational profile. Furthermore, we can formally prove that the resulting objective has appealing statistical properties.
Theorem 2.
The sliced versions of the energy objectives (2) and (3) are consistent estimators for the data distribution and feature unbiased gradients.
Intuitively, the first part of the theorem is true because the CRPS objective is related to the MMD. At the same time, for each v the objective remains proper score; a weighted combination of proper scores is also a proper score, hence the second part holds. See the Appendix for a full proof. Recall also that in one dimension, the energy loss reduces to the CRPS, which is equivalent to the Wasserstein-2 2 distance. Wasserstein distances have more favorable convergence properties (Arjovsky al., 2017) than maximum likelihood training, which lends further support choice of objective.
4

---

Slicing also allows to use univariate two-sample tests objectives. We describe several objectives in Appendix D.2. In brief, these include Kolmogoroy-Smirnon a popular statistical test defined for two CDFs F and G (S G) sup, [ Hotelling's 2 test (DFF PG) (mp=mea' sliced version of Hotelling's objective; the sliced version of the Fréchet objective Ry, DG)
Sliced Two-Sample Baselines
(mF mG)² (5) s²
ARCHITECTURES FOR ENERGY FLOWS
Next, we introduce energy flows, class of models trained with our proposed determinant-fre objectives. An energy flow is defined by an invertible mapping between 2 and y and is trained using the energy loss. As result, energy flows improve over classical flow models by, among other things, supporting flexible architectures and by simultaneously providing fast training and sampling.
Previous work on normalizing flows involved constrained architectures with tractable determinants, such as autoregressive models. In contrast, our model supports flexible feedforward architectures, which we outline in Appendix E. In brief, our loss supports dense invertible flows (DIFs), sequences of fully connected layers constrained to be invertible, invertible residual networks (Behrmann et al., 2019), as well as rectangular flows (REFs), in which the dimensionality of y and z is not equal (Nielsen et al., 2020; Cunningham and Fiterau, 2021; Caterini et al., 2021).
4.1 SEMI-AUTOREGRESSEE FLOWS
We also introduce remi-autoregressive flows (SAEFs), an architecture trained with the energy loss that combines the speed of feed-forward architectures with the sample quality autoregressive models. The SAEF model divides d-dimensional the data into B blocks and generates samples blockwise; as a result, sampling time reduced by factor of (d/ B) relative to an autoregressive model.
Formally, SAEFs define an invertible mapping between latent variable and observed variable y and require choosing a partition of y, z into ordered blocks (yt) and (z,) (e.g.. 4x4 blocks of pixels in an image). They induce probabilistic model p(y) over y that factorizes as p(y) IT ((ybly<s)) where each p(ysly <t) is defined via invertible mapping
Yb ===7(2s>
TZzs; h2)
h cy()))),
(5)
where r(zs; hy) is an invertible transformer mapping the b-th latent block 2, to the b-th observed block yo. and c is the b-th conditioner which outputs transformer parameters hs. Any invertible feed-forward energy flow can be used to parameterize -- provide specific examples below. The
entire SAEF is trained via a sum of energy losses applied to each block EywD [Ei= E(Fy,
where D is training set, y 'D is datapoint sampled from D, Fy, is the distribution over Yb induced by T., hy())))), and & is one of our two-sample losses, such as (2) or (3).
When blocks one-dimensional, this reduces standard autoregressive architecture that features high sampling quality but slow sampling speed. When blocks are full-dimensional, this reduces to a non-autoregressivv energy flow with fast sampling but possibly worse quality. our experiments, we show that SAEFs can trade-off between these two regimes and obtain the best of both worlds. Note also that SAEFs are hard to train using maximum likelihood, as they require specifying invertible non-ane non-autonn autoregressive mappings 7 between possibly high- -igh-iimensional blocks ys. Zu the SAEF architecture is only trainable using the energy objective. See Appendix FF for pseudocode.
EXPERIMENTS
We evaluate our framework on range of UCI datasets (Dua and Graff, 2017) as well datasets of handwritten digits (Pedregosa et al., 2011; 10 Deng, 2012).
Classical Normalizing Flows maximum likelihood and that two types of architectures: autoregressive and -on-autoregressive. Our autoregressive models are based on basclines from earlier work (Papamakarios et al., 2017; Si al., 2022) and include Neural Autoregressive Quantile Flows (NAQF), and Masked Autoregressive
We benchmark our models using normalizing flows trained using
5

---

Table Objective Results and Complexity.
Metrics
KS ID Hotelling
Hotelling 0.717 d
ID-FD 0.558 n
FD 0.559 d
ID-Energy 0.545 bn
Energy 0.548
CRPS
1.53
0.57 72
Complexity
nloog
P(yylı< 2y), where each p(ys [y is probability conditioned the previous variables and 2j. In MAFs, the p(uslve (v are Gaussian; in NAQFs they are parameterized by flexible quantile flow (Si et al., 2022), In our UCI experiments, this we used densely connected networks; in our image experiments, used variant of the PixelCNN architecture.
Flows (MAF-LL). These models assume parameterization p(y|z)
=IT p(ylz)=1I[/-
Our non autoregressive models consist of variational auto-encoders (VAEs) trained using the evidence lower bound (ELBO) on the maximum likelihood and based on fully-comnected architecture (see below for details). In order to understand the benefits of our objective, we fit VAE model with the same invertible architecture for the generator as the one used by our energy flow models; we refer to the resulting method as dense invertible flow trained using maximum log-likelihood (DIF-LL). We additionally compare against two oate-of-ttheart flow models, FFJORD (Grathwohl et al., 2018) and Invertible Resnets (Behrmann et al., 2019), using their open-source codebase.
Energy Flow Models We constructed flow models using dense invertible layers, referring the resulting model as a Dense Invertible Flow trained with an energy loss (DIF-E). The DIF-E model consists of three feedforward invertible layers and Leaky ReLU activation functions, and is trained using the kernelized energy loss 3 with a mixture RBF kernels with bandwidth in (2,5, 5, 10, 20, 40, 80]. We also use the energy score train non-invertible rectangular flows trained with the energy loss (REF-E). In particular, rectangular flows are parametrized by layers of size [d/8,d/4 d/4, d/2, as compared to DIF-E which requires all layers be of size for invertibilityy
We also compared DIF-E to autoregressive models trained with a Jacobian-free objective. Specifically, we trained our autoregressive MAF models with the quantile loss (Si al., 2022) addition the NAQF models. We denote these as MAF-QL and AQF-QL respectively.
Metrics We evaluate the models in terms of log-likelihood (when available and appropriate) and using variants of the CRPS metric. For VAF-rype models trained using the ELBO, we report the ELBO as a lower bound on the log-likelihood. We use two CRPS-style metrics which have the following structure: the first is sample-based version in Equation 2. The second, marked as univariate CRPS (U-CRPS), the sum of one-dimensional CRPS measured for each output dimension and estimates the quality of marginal distributions (Si al., 2022). Both versions use the & norm.
In our image datasets, we are also interested in estimating in a quantitative way the quality of the generated samples and their similarity to the data distribution. In order do that, define metric called the D-loss. The D-Loss measured by the accuracy of a discriminative model to determine whether an image is generated. Details be found in the appendix.
5.1 UNDERSTANDING THE SAMPLE-BASED OBJECTIVES
We start with experiments that analyze the properties of the sliced energy objectives and compare them to the baseline two-sample objectives on the UCI and image datasets.
Energy Objective vs. Two-Sample Baselines We claim that the energy objective is particularly favorable training criterion for flows; we empirically establish this fact by comparing it against the other two-sample objectives, which we see as strong baselines. We train an invertible flow model on the Miniboone UCI dataset. Complexity is written where b denotes the batch size, d denotes the dataset dimension, and n denotes the number of projections made.
Flows trained using the energy objective achieve the best performance in Table 2, outperforming the strong baselines; we focus on the energy objective in subsequent in-depth experiments.

---

Table 3: Slicing on MNIST.
Improvements in Scalability From Slicing Next, we seek to understand the scalability improvements from slicing. We train projection based model
400
200
100
50
U-CRPS
0.088
0.088
0.088
0.091
on MNIST using projected energy loss. The model consists of dense layers of size 784 with leaky
CRPS
0.191
0.191
0.192
0.195
Block-n
100
20
10
relu activation functions for the first three layers and sigmoid activation function for the last. When we calculate the loss, we take n projections into a single dimension, which we denote in the top half of Table We additionally conduct slicing experiments Slicing conducted separately for each block in a similar manner to the fully-feedforward flow, so we're projecting a 49 dimensional block down to single dimension. The number of projections block denoted in Table as Block-n. In general, even for the block parameterization, the CRPS tends to stay stable when using fewer projections.
U-CRPS
0.084
0.085
0.084
0.084
CRPS
0.086
0.087
0.086
0.087
for the projection parameters. the SAEFs, in particular the 7x7 block size variant.
see in Table 3 that sliced objectives perform comparably to non-sliced objectives with a fraction of the dimension, while having improved computational complexity.
On Likelihood vs. Non-Likelihood Based Losses Our work questions the use of maximum likelihood for training flow models. Table 4, we observe that a PixeIMAF model trained with likelihood has low CRPS scores, and conversely it achieves poor likelihoods when trained with an energy loss (samples have high quality in both scenarios). An con-autoregressive DIF-E model achieves the highest quality samples and the worst likelihood when trained with the energy loss, suggesting that the energy loss is an attractive alternative to the likelihood, featuring high quality without the instability of GAN-type objectives (Grover et al., 2018).
5.2 UCI EXPERIMENTS
Table 4: Metries on MNIST.
Non-autoregressivve energy flows can capture joint dependencies without the slow generation process of autoregressive models. We implemented energy flows five UCI datasets used previously benchmarks by Papamakarios et al. (2017) and Si al. (2022), which are: BSDS 300, Miniboone, Gas, Power and Hepmass. The size of the datasets are noted in Table 5. We used an LSTM architecture for all autoregressive models and the SAEFs. The SAEFs use an appropriate block size b which divides the dimension of the data; Miniboone has extra dummy dimension appended to make it divisible by in this case.
Models
NLL
CRPS
FID
PixeIMAF LL PiXelMAF-OU
-3142
.279
100.15
-2617
.215
85.08
DIF-E
2388
.186
22.76
Table Model performance on UCI datasets measured by CRPS.
Dataset
d
MAF-II
MAF-QI.
DIF-E AQF-QL
DIF-E Proj
b
SAEF
BSDS 300
63
.044
.036
.033
.039
.040
3
.037
Miniboone
43
.567
.561
.525
.524
.545
2
.521
Gas
8
.645
.565
.513
.548
.551
2
.530
Power
.542
.506
.502
.451
.454
2
0.443
Hepmass
21
.617
.614
.523
.589
.587
3
.559
Results. As shown in Table 5, energy flows perform comparably to the neural AQF-QL baseline. Both methods are trained using variants the CRPS and obtain top performance across the five datasets. However, the DIF-E model is non-autoregressive, hence provides advantage in terms of sampling speed. Our experiments thus illustrates that non-autoregresssve models can match the performance of autoregressive models trained with log ligeelikohood versions of the CRPS objective. SAEFs, which use a variant of the same loss as DIF-E. further improve upon its results, giving slightly better CRPS scores across the board, while still being reasonable in terms of sampling speed.
7

---

Table 6: Digits Generation Experiments
Method
U-CRPS
CRPS
D-Loss
Training (sec)
Sampling (sec)
PiXeIMAF-LI.
0.136
0.206
0.974
0.15
14.00
PixeIMAF-QL
0.131
0.204
0.883
0.15
14.00
PixcIAQF-QL
0.127
0.199
0.681
0.13
14.00
DIF-LL (VAE)
0.138
0.207
0.941
0.07
0.01
REF-E
0.127
0.201
0.823
0.12
0.06
DIF-E
0.126
0.197
0,807
0.14
0.07
UIF-E-Proj
0.127
0.199
0.815
0.06
0.08
SAEF-1
0.126
0.198
0.795
0.12
10.54
-
0.126
0.199
0.754
0.07
2.73
SAEF-4
0.127
0.198
0.772
0.05
0.86
5.3 IMAGE GENERATION DIGITS
Next, we test our methods on a standard generative modeling task: digit generation (Pedregosa al., 2011) in Table We use a PixeICNN architecture for the autoregressive models, which we denote PixeIMAF-LL, PiXeIMAF-QL, and PixeIAQF-QL. The PixeICNN maps from (N(0, 1) 1 distribution (PixelMAF) and (U 1))² distribution (PixelAQF-QL. to the target distribution. SAEF models also utilize the same PixelCNN architecture, but with its generation sectioned off into different blocks, each which is evaluated by our energy loss. We provide ELBO loss as an upper bound on the NLL for VAE-type models, and training speed is given by seconds per epoch, while sampling speed is given by seconds per 1000 samples generated.
Results. The proposed DIF-E and SAEF models perform comparably on the U-CRPS and CRPS metrics to the AQF model, although the latter is more discriminable (has a better D-loss), On the other hand the samples generated by the DIF-E and SAEF outperforms those of any of the other autoregressive architectures. as well as the samples from non-autoregressive DIF-LL -L- model, which is trained with maximum log likelihood.
Inversion and Interpolation. A key feature of the DIF-E model exact posterior inference (despite not being trained with log-liielihood). To demonstrate this, we create intermediate representations between pairs of digits through which we can smoothly interpolate (Figure 1).
Figure 1: Interpolated digits with Energy Flow
By taking the inverse of the decoder model by inverting the activation functions and weight matrices, we are able to create an encoder similar to that of the VAE. Like the VAE, the energy flow can generate interpolated samples, with the added advantage of exact posterior inference.
5.4 MNIST
For this of experimental baselines, architecture, with a receptive field of 7 and residual blocks. Our SAEF models use the same architecture, but with the corresponding block sizes and trained using the block energy loss. In addition, we trained a rectangular flow model using the ELBO approximation of the log-likelihood (REF-LL), which results in model equivalent VAE. We computed the same metrics did for the experiments on the digits dataset for all methods.
again trained autoregressive models with the PixelCNN
Results. Though the fully feedforward DIF-E and DIF-E-Proj variants still showed decent performance with fast training and generation speed and show great results in CRPS, SAEF-4 in particular greatly outperforms them in FID without losing much ground in terms of CRPS. and in general by introducing degree of autoregressivity, the sampling time and complexity still very reasonable in comparison to the fully autoregressive models, and medium size SAEFs improve upon the sampling time by order of magnitude.
much improved results. In addition.
8

---

6
7
9
7
3
5
9
7
3
0
7
7
P
/
8
4
5
5
7
7
5
7
/
V
3
(a) PiXeIMAF-LL
(b) PixelMAF-QL
4
2
2
5
9
6
3
6
9
0 3
6
4
7
3 0
0
6
3
6
7
9
4
6
5
O
4
4
7
3
6
2
DIF-LL
(d)
DIF-E
7
2
4
9
6
7
4
6
3
8
1
7 8
1
0
2
9
<
1
6
0
O
S
0
2
3 0
5 7
6
7 0
6
3
7
2
6
6
7
1
6
O
2
4
O 4
8
2
(c) FFJORD (64)
(f) SAEF-4
Figure 2: MNIST samples from six methods
Table 7: MNIST Generation Experiments
Method
U-CRPS
CRPS
D-Loss
FID
MMD
Training (sec)
Sampling (sec)
PixeIMAF-LI
.128
.799
1.00
100.55
0.296
35
195.38
PixeIMAF-QL
.099
.215
.93
85.08
0.287
35
195.38
PixeIAQF-QL
.119
.288
.966
61.27
0.232
29
195.38
REF-LL (VAE)
.090
.189
.903
44.55
0.140
0.25
DIF-LL (VAE)
.089
.190
.855
36.91
0.131
10
0.48
FFJORD (16)
.101
208
.650
24.78
0.103
540
48.88
FFJORD (64)
.002
209
.633
9.69
0.087
3100
155.69
iResNet
100
206
.642
41.47
0.111
840
2.43
Flow-GAN
.055
.187
0.608
43.67
0.068
15
0.40
GLOW
0.090
0.197
0.983
63.06
0.600
1400
190.63
REF-E
.085
117
.778
41.04
0.052
0.21
DIF-E
.084
.186
.701
22.76
0.051
0.40
DIF-E-Proj
.085
186
.819
22.55
0.056
3
0.40
SAEF-2
.855
.188
.675
9.86
0.167
32
31.22
SAEF-
.085
.187
.567
7.05
0.081
12
8.19
SAEF-7
.085
.187
.608
14.91
0.088
6
2.17
SAEF-14
.085
.187
.650
19.57
0.068
0.93
6 PREVIOUS WORK, DISCUSSION, AND CONCLUSION
Comparison to Other Generative Model Families We provide complete comparison to other models in Table Our approach contrasts against VAE style models (Kingma and Welling, 2014) by providing exact inference and likelihood evaluation. Unlike MAFs (Papamakarios et al., 2017), our models are not fully autoregressive in nature and do not make any Gaussianity assumptions and instead use a fully neural parameterization of the output probabilities, which contributes to improved performance and modeling flexibility. Approaches like AQF (Si et al., 2022) and NAF (Huang al., 2018) also provide neural approximators, but their fully autoregressive architecture makes for slow sampling. Closely related feedforward models include GMMNets (Li et al., 2015; Dziugaite al., 2015) and CramerGANs (Bellemare al., 2017), but they do not offer likelihood evaluation and posterior inference. Our framework is closely related to Maximum Mean Discrepancy (MMD) (Gretton al., 2008) and its generalizations (Li al. (2015), Dziugaite et (2015)). However, unlike models optimizing MMD (Li et al., 2015; Dziugaite et al., 2015), ours provide latent variable inference and exact density evaluation using the change of variables formula.
Conclusion We proposed training normalizing flows using the energy objective, proper scoring rule that does not require computing determinants during training. We then proposed cuni-eautorerrsssvve energy flows, which feature fast sampling and high sample quality. Using an invertible flow architecture also allows us to retain exact posterior inference in energy flows. We work questioning the of likelihood for training normalizing flows, and first step towards determinant-Ff flows based on novel learning objectives and architectures.

---

7 REPRODUCIBILITY
Datasets All datasets that we use are open-source. We have described in detail the datasets used in our experiments. The UCI dataset is used without preprocessing, and the digits and MNIST datasets are cast to floats and scaled to [0, I] range from their original range.
Models For each of our baselines as well as the main code, we have given implementational details in the Appendix A including layers, learning rate, and epochs. For our SAEF, we additionally provide a pseudocode in Appendix F.
10

---

REFERENCES
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2015.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743 4751, 2016.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.00762,, 2019.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation, 2014. URL https: www.aa org/ aabss////////// 01110.11.000 8516.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features, 2019.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint arXiv:170S.07057 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 2017.
Phillip Si, Allan Bishop, and Volodymyr Kuleshov. Autoregressive quantile flows for predictive uncertainty estimation. 2022.
T. Gneiting and A. E. Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, (10):359-378, 2007a.
Diederik P Kingma and Max Welling. Stochastic gradient vb and the variational auto auto-encoder. In Second International Conference Learning Representations, ICLR, 2014.
Chin-War Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference Machine Learning, pages 2078-2083 PMLR, 2018.
Yujia Li, Kevin Swersky, and Richard Zemel. Generative moment matching networks, 2015.
Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance a solution to biased wasserstein gradients, 2017.
Ian Goodfellow, Jean Pougct-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Gintare Karolina Dziugaite, Daniel Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization, 2015.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaii.v311 1221..1119, URL https://ojs aaai. org/ bogg/nnex...m php AAAI aatticeeeviiiieeeaaeeee werie///112......
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):29-4433, 1997. ISSN 00018678. URL http: http:://www//.oooo..//p/o org/ stable/ /4480011
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alexander Smola, kernel method for the two-sample problem, 2008
Gabor Szekely. E-statistics: The energy of statistical samples. 2003.

---

Leonid Kantorovich. Mathematical methods of organizing and planning production. 1939.
Leonid Kantorovich and Gennady S. Rubinstein. On space of totally additive functions. Vestnik Leningrad. Univ, 13:52-59, 1958.
Robert Fortet and Edith Mourier: Convergence de répartition empirique vers la répartition théorique. Annales scientifiques de I' É'École Normale Supérieure, 3e série, 7063)2667-255, 1953. doi: 10.24033/ asens. 1013. URL http: www. numdam org/ wtt.aaattiileeeeeeeeeeeeeeeeeeee 10. 24033 00/00333300 aaeess.ss.......... 1013/.
R. M. Dudley. Weak convergence of probabilities nonseparable metric spaces and empirical measures on Euclidean spaces. Illinois Journal of Mathematics, 10(1):109 126, 1966. doi: 10.1215/jj//1556055200 URL https doi. www//og//// 1215 1mm/12560552066.
Bharath K. Sriperumbudur. Arthur Gretton, Kenji Fukumizu, Gert R. G. Lanckriet, and Bernhard CoRR, abs/0901 12269.
Schölkopf. A note on integral probability metrics and SSphiš-divergene 2009. URL http: / www.iii.ggggaaaee. 00901.200000 2699..
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo K. Rohde. Generalized sliced wasserstein distances. CoRR, abs/1902.00434, 2019. URL 1902. 00434.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation, 2019. URL https:tttt.... http://axvvv...oo attp/aass//90 /05. 07088.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional slised-wassersstein and applications to generative modeling, 2020. URL ht https:/ http///xxivv..r/ ort/bbs/////////.ro wttp:or////b////........ 2702.07377
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017. URL https:
//arxivv org/ abt///bs//// 07875.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks, 2019.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing 2020.
Systems,
Edmond Cunningham and Madalina Fiterau. A change of variables method for rectangular matrix-vector products. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2755-2763. PMLR, 13-15 Apr 2021. URL https: htap:::/prrccee http///oooeiiocccooooooo ngs ml wwr.s..rssss.... / vv v130 cunn mowniinghmm111 a. html.
Anthony L. Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John Cunningham. Rectangular
flows for manifold learning, 2021.
Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URL http://archive. ics. wwi.cciieoo edu/m1.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916)1111112, 2012.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models, 2018. httpst
//arxiv.. wwg./rgg//oooo. abs 1136.01367.
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and rahs-based statistics in hypothesis testing. The Annals of Statistics, pages 2263-229 2013.
12

---

T. Gneiting, F. Balabdaoui, and A. E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243-266 2007.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation.
Journal of the American Statistical Association, 102(477):359-37, 2007b.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks, 2021.
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural importance sampling, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, 2019.
Hadi M. Dolatabadi, Sarah Erfani, and Christopher Leckie. Invertible generative modeling using linear rational splines, 2020.
Francois Chollet et al. Keras, 2015. URL httttp:://ggthuub.coo//ootee
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference, 2016. URL btttps://arxiv.ort abs / 610./6666..00 00704.
Jeff Donahue, Philipp Krähenbühl,, and Trevor Darrell. Adversarial feature learning, 2016. URL https://arxiv.e wwwpt/i..rraoooo..oooe httpg//bss//1...... 6666009822.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders, 2015. URL
httpg//bs/11111 05644.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autocncoders and generative adversarial networks, 2017. URL org/abs/ /170 04722.
13

---

Table 8: Abbreviations for models (upper) and losses (lower) used throughout the text.
Abbreviation
Full Name
Models
AQF
Autoregressive Quantile Flow
DIF
Dense Invertible Flow
MAF
Masked Autoregressive Flow Rectangular Flow
REF
SAEF
Semi-Autoregrressvve Energy Flow Invertible Residual Network PixeCCNN
iResNet
Pixel
Losses
E
Energy Loss
LL
Log Likelihood Quantile Loss
QL
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 ARCHITECTURE DETAILS
Abbreviations and Losses We have included a table of abbreviations in Table 8, where the upper half consists of the abbreviation for the model names, and the bottom half consists of the abbreviations for losses.
Throughout the text we use loss abbreviations appended to model abbreviations to indicate which objective particular model was trained with (e.g.. PixeIMAF-LL corresponds to PiXelMAF model trained with log likelihood). For the remaining baseline models which do not follow this convention (FFJORD, GLOW, iResNet) we note that they are trained with bog-likelihoo, and note that Hoow-GNN uses an adversarial loss combined with a log likelihood term.
Slicing For our slicing experiments with the various multidimensional and single-dimension losses, the invertible feedforward model consists of layers of size 43 (to match Miniboone dimension), and each objective is trained for 200 epochs with a learning rate of 1e-3. Each sample is projected onto 200 random normal vectors, and the resulting two-sample loss is summed across the 200 projections.
UCI Experiments For the PixeIMAF and PixelAQF models, the LSTM architectectures composed of two LSTT layers with hidden size equal to the dimensionality of the data. All models were trained with a batch size of 200 and learning rate of lc-3 using the Adam optimizer for 200 epochs for the smaller datasets (Miniboone, Hepmass) and 20 epochs for the larger ones (Gas, Power, BSDS 300).
Image Generation Experiments For the PixeICNN architecture, we used receptive field 7 for digits, and a receptive field of 15 for MNIST. We chose to switch the autoregressive architectures here from LSTM based models which are more sequential to the convolutional PixeICNN architecture to retain local features. The PixelMAF-LL and PiXelMAF-QU autoregressively transforms pixels from the image into a distribution of the next pixel, except that PixeIMAF-LL uses log likelihood loss while PixelMAF-LL uses quantile loss. PixelAQF-QQ model takes in samples from uniform distribution, and predicts the corresponding quantile loss for each pixel.
Our DIF model are parametrized with layer size of 64 on the digits dataset, and 784 MNIST, with invertible activation functions (Leaky ReLU activations up until the last layer, and Sigmoid for the final activation). The images flattened out prior being passed through the feedforward model. On MNIST, each model was trained for 300 epochs with a learning rate of 1e-3, while on digits, each model was trained for 2000 epochs.
The additional baselines are constructed as such: Each model is adapted from the code from their original papers. FFJORD models two blocks of stacked CNF layers composed of ODE Nets. We
14

---

tested with both a hidden size of 64 and hidden size of 16, and trained for total of 100 epochs with the optimizer at a learning rate le-3. The Invertible ResNet has three scale blocks, each having 32 Invertible ResNet blocks, consisting of 32 filters with three convolution types with an ELU activation function (Behrmann et al., 2019). It used an AdaGrad optimize with batch size of 128 trained for 70 epochs at learning rate of 0.003. Glow was trained with 3 levels and depth of on 8 GPUs for 250 epochs.
A.2 D-LoSs
The D-Loss is derived from an ensemble of SVM discriminators (with RBF kernels having bandwidth parameter 7 of 0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000, and 10000) used to differentiate the real data versus the sampled data from the model. More details are included in the appendix. The D-Loss for single discriminator model D _=" - {0, }} is measured the accuracy with which D can discern the difference between generated samples and true samples, which are labeled 0 and respectively. Given validation with m data points and labels Y {0, 1}, we have that
1 D-Loss = Dz + ())((=))) - y(
The final D-Loss is measured by the validation accuracy achieved by the best SVM, which gotten through 80:20 train-validation split on the 300 real and 300 fake images: D-Loss max, D-Loss,
A.3 DEFINING CRPS AND U-CRPS
For simulated samples X1,..., Xm and true data we define CRPS as
1 X
CRPS((X1,..., Xm).
y/1
|X
m
2m
and U-CRPS
77L
U.
X;
X,I
2m
1=1
B THEORETICAL ANALYSIS AND PROOFS
Multi-Variate Objectives In this section, make the assumption that the kernels K used to define our objectives are measurable and bounded by K. Under these conditions, when using a kernelized objective with kernel K, each distribution F' can be represented by mean embedding Pr in the reproducing kernel Hilbert space (RKHS) induced by K (Gretton et al., 2008). We also assume that there exists unique mapping between μe and F for every F in the class of model distributions F. Note that this can be satisfied for any Borel probability measure if the kernel K is chosen to be universal or characteristic (Gretton al., 2008). Alternatively, we may satisfy this claim by choosing our F to be a restricted set of distributions for which the above claim is truc.
Theorem. The energy objectives (2) and (3) are consistent estimators for the data distribution and feature unbiased gradients.
Proof. First, we seek to establish the consistency of the minimize of our objectives as an estimator of the data distribution. Our argument uses the fact that the (kernelized) energy score is closely connected to the maximum mean discrepancy (MMD; (Gretton et al., 2008)). Observe that
EyDD (V/WDCRPxx(FFy))) EpK(Y, Y') EyrK(Y,,')
Rr Y' Kr,,K(Y,Y)) B , (y
MMD (F, D) + const;
15

---

hence, by maximizing the CRPS over set of possible models F, minimizing monotonic transformation of the MMD. Since (2) a special case of (3) with the distance kernel (Sejdinovic et al., 2013), the above claim also holds for the non-kernelized energy objective (2).
We would like to establish that minimizing objectives (2) and (3) over a data distribution D, of samples from the true data distribution P yields model F,. that is similar to what would obtain we searched for the best using the full data distribution P; in other words:
(F P)) = inf L(F, P) +)+ o(n),
PEF
where L(F, P) a metric or pseudo-metric" that we will instantiate shortly, the hypothesis class for the model F, Fn is the empirical risk minimization solution (from our method) over dataset Dn.. and the additive o(n) term decays to zero as we increase n. Note that if the model is well-specified (i.e., P F), we have (F, P)] o(n), and we have consistent estimator.
To establish this fact, we will derive a version of the above identity for a modified version of the MMD and under the assumption of this section; we will also argue that the kernelized energy estimate satisfies that identity.
MMD (F, P). By the properties of MMD and kernels, know that MMD(F, P) = Pr - Mlll, and MMD is a pseudo-metric Note that we have by the triangle inequality
First, let L(F,P)
L(F P) S L(Fn, P.) L(D., P),
where we overload notation and use D, to also denote the empirical distribution. Note that because our objective is monotonic transformation MMD2 + const) of the MMD, the F, minimizes the MMD within F. Thus we can write for any F FF
L(Fn, P) < I(F,D.) L(D.,,) S L(F,P) + {L P)
where we have used once more the triangle inequality in the last line. Taking expectations on both sides and using the fact that F F was arbitrary, we find that
EL(Fn, P) S inf L(F,P) + 2BL(Da,,) FEF
S inf L(F,P) + 2 EL(D,,,))) FEF
To establish our claim, we need to bound the last term. Let x; denote the i.i.d. samples from D,, let denote the embedding induced by the kernel in RKHS H. and note that we have
1 n
EL(D,,, P)² E
)(x,) Eδ(x)l
Var - ((I)))
1 Var()(x1))
n
2
Blld(x)))
n
2x
Thus, main claim follows with
2k
EL(F,, P) inf L(F,P) + 2/ FEF
n
'A metric d(x, y) satisfies four properties: symmetry, triangle inequality, d(x, x) = 0 and d(x, v) = 0 + = y. A pseudo-mette metric satisfies only the first three properties. The MMD objective is a metric if its kernel is characteristic or universal (or more generally if there is one-to-one mapping between (r and F; otherwise, it pscudo-metric.
16

---

Thus the estimated model Fn satisfies the above inequality and if the data distribution P E F, our consistency claim holds.
We can establish that the gradients unbiased by leveraging properties of proper scoring rules. Recall from the background section that loss L (RR)) x I + strictly proper (Gneiting and Raftery, 2007a) if G arg minp (y y). In the context of the CRPS objective L, we have by definition of a proper loss
L(P,G) E (F,y') E Gy:),
where Gy is the empirical distribution derived from y'.
Let P denote the true data distribution, D,, dataset of size drawn from P, and Gn the resulting empirical distribution. Then we have:
VolD DEP..PL(FFo G,)) Nply ((Fo.y')
VolBy-pl(Po..)) VoL(Fe, P),
which is equivalent the statement that we wanted prove.
Alternative Approaches to Showing Consisteney The fact that consistency holds also follows from properties of the MMD for general classes F; for example shown in Dzuigaite et al. (Dziugaite al., 2015) (Theorem I),
E E[MMD P)] S inf MMD2 o(n), PET
JF satisfies fat-shattering condition. The desired consistency claim with L(F, G) being our kernelized energy objective CRPSk & then follows directly from our earlier derivation by applying an affine transformation on each side of the above equation.
Alternative Approaches to Showing that Gradients are Unbiased Note that special case of the unbiased gradient property for the non-kernalized objective (1) has been established using techniques discussed in Bellemarre et al. (Bellemare al., 2017) (Proposition 3). This result also follows from our aforementioned connection to the MMD and Lemma 6 in Gretton et al. (2012) (Gretton et al., 2008).
Sliced Objectives
Theorem. The sliced versions of the energy objectives (2) and (3) are consistent estimators for the data distribution and feature unbiased gradients.
Proof. We establish the first part of the claim by observing that the sliced version of the energy objective
CRPS(F,y') = Em--p(w) F Ew Y. w Y EpK(w Y, w wTyy
(6)
where K is a kernel in ID, an affine transformation of squared MMD. Then the first part of the claim follows by the argument in Theorem 1. To see this, first, define the function K,, Rd x Rd RR as
K where K is the kernel used as part of the sliced energy objective. It is easy to see that K (2, y) is kernel. Consider any dataset 5 { then the matrix M defined as Mij K (1,, xr)) will be semi-definite because the corresponding matrix M' defined as M/J K(w' v, 22)) is also positive definite, because the kernel matrix for the set S = (w T x. ) Hence, by Mercer's theorem K., is kernel.
K(w xx w.T.T. y),
Next, define the function K Rdd x Rdd + R as
K (5,, v) Ewnp(()) K Kw
17

---

This is also kernel, because it is sum of kernels. Next, note that
CR = E ww-p(w))
2 BB (Y
Y. Yu'r - EpK( To Y, Y, wyy
2
R RrpR(YY
which an instance of the kernelized energy objective that uses modified kernel. Note that this is both proper score and rescaled version of the squared MMD with modified kernel. The two claims of this theorem follow directly from Theorem
C EXPANDED BACKGROUND ON PROPER SCORING RULES
C.I PREDICTIVE UNCERTAINTY IN MACHINE LEARNING
Probabilistic machine learning models predict probability distribution over the target variable- c.g. class membership probabilities or the parameters of an exponential family distribution. We seek to produce models with accurate probabilistic outputs that useful for generation.
Notation. Supervised models predict a target E y from an input E X. where are given model + [0. 1] within the set Ay of
realizations of random variables X, ~ and
data distribution.
H X Ay, which outputs probability distribution F(y) distributions over V: the probability density function of is f. We are also given a training set D { yi) X x V) consisting of i.i.d. realizations of random variables X, Y ~ P. where the data distribution.
C.2 PROPER SCORING RULES
Comparing point estimates from supervised learning models is straightforward: metrics such as accuracy or mean squared error. Probabilities, on the other hand, are more complex and require specialized metrics.
rely
In statistics, the standard tool for evaluating the quality of predictive forecasts is a proper scoring rule (Gneiting al., 2007). This paper advocates for evaluating the quality of uncertainties using proper scoring rules (Gneiting and Raftery, 2007b).
Formally, let L. Ay y IR. denote loss between a probabilistic forecast F Ay and realized outcome E y. Given a distribution G' e Ay over we use L(F, G) to denote the expected loss L(F, G+) E
We say that L is proper loss if it is minimized by G when G is the true distribution for L(F, G) L(G, G) for all F. One example is the log- log-likelihood L(F,v) v) log f(y), where is the probability density or probability mass function of F. Another example is the check score for e [0, 1]:
(T (T)
ify
(F,
((1-7) 3)
otherwise.
See Table for additional examples.
What are the qualities of a good probabilistic prediction, as measured by proper scoring rule? It can be shown that every proper loss decomposes into of the following terms (Gneiting et al., 2007):
proper loss = calibration sharpnesss + irreducible term
refinement
Thus, there precisely two qualities that define an ideal forecast: calibration and sharpness.
18

---

Proper Loss
Loss L(F.G)
Calibration L((F.Q)
Refinement (,
Logarithmic CRPS
BynG logf(y) BynG (F(y) - G(y)) 1 Byn Pr (F
S (F(y) KL(allf)
11(4)
dy
1 (1)((1-Qu)))))) B yr pr(Q. ((
So Jo (r) (Q(v) (Q(v) - T)dydr
((r)
Quantile
Table 9: Examples of three proper losses: the log-loss, the continuous ranked probability score (CRPS), and the quantile loss. A proper loss L(F, G) between distributions F, G assumed here to be cunulative distribution functions (CDFs) decomposs into a calibration loss term (P, Q) (also known reliability) plus refinement term L, (which itself decomposes into sharpness and an uncertainty term). Here, Qyy) denotes the CDF of IP(Y = y Fx F), and q(v), [y) are the probability density functions of Q and F.
D DETAILS ON ADDITIONAL TWO-SAMPLE TRAINING OBJECTIVES
D.1 GAUSSIAN TWO-SAMPLE BASELINE OBJECTIVE OVER LIGH-DIMENSIONAL VECTORS
We use the following classical objectives as baselines for our work and illustrate examples of alternative methods that can be derived from our two-sample-bee basee approach; both tests make Gaussian modeling assumption.
Hotelling's 's Two-Sample Test Being closely related to Student's ' t-test, it uses the following statistic:
H2 DG) (mæ mc) 5 (mp mg),
(8)
where mp = 1 E ED y(,, mc = 1 Ey Ly y are the sample means, and the matrices SF m Ey ED (y(9) ( mm" SG m-1 Eywep (y()) my) m)7
are sample covariances, while 5 (Sp + Sc) is their average. This objective encourages the two samples to have similar means.
Fréchet Distance
This is another Gaussian based distance that we use as an objective:
R(Dr, Dc) ||m mo 1/2 + tr(Sp + Sa 2(Sr))))1²)
(9)
where we using the same notation as above. This objective is encouraging the model to produce data with similar means and variances. It is derived from the Fréchet distance between two Gaussians.
D.2 SLICED TWO-SAMPLE BASELINES
Slicing also allows us to use univariate two-sample tests as objectives. We give examples below.
Kolmogoroy-Smirnov One of the most popular ways of comparing the similarity between two distributions is via the quantity
KS(F. G) = sup FF G(v)),
(10)
the maximum distance between two CDFs F and G. Empirical CDFs can be used for samples. While this test corresponds to an IPM, it does not have a widely accepted extension to higher dimensions.
Hotelling's Univariate Objective The sliced version of Hotelling's objective corresponds using Hotelling's 's 12 univariate test (which is just the squared version of Student's t-test) as an objective.
H.. Da) (mp mmm
(11)
s2
where mp, mD, and 5² are respectively the sample mean Dr, the sample mean of De, and the combined sample variance, defined as in the multivariate version. Note that this formula is much less computationally expensive than the inversion (in worst-case ((d)) time), while the sliced version takes only O(d) time.
one, which requires performing a matrix
19

---

Fréchet Univariate Objective Similarly, the sliced version of the Fréchet objective is written as:
(D Dc) = (m mG)² + (5 =
(12)
which encourages the sample means m., Bg and the sample variances s,, so to be the same. Again, this ((d) formula is less computationally expensive than in higher-dimension where it requires performing multiple matrix multiplications and matrix square root (O(d))).
E FEED-FORWARD ARCHITECTURES FOR ENERGY FLOWS.
Since our objective does not require computing Jacobians, are able to use flexible classes of invertible models that are difficult to train using log-likelihood. Previous work on invertible mapping leveraged integration- based transformers (Wehenkel and Louppe, 2021), spline approximations (Müller et al., 2019; Durkan et al., 2019; Dolatabadi et al., 2020), piece-wise separable models, and others. Our main requirement on the model architecture efficient sampling. We describe several feed-forward flow architectures that are compatible with our objective below.
Dense Invertible Layers The simplest architecture we consider consists of a sequence of small Fd - Fdd dense layers with invertible non- oonearirssss (such as tanh or leaky ReLUs). We enforce the invertibility of the dense layers by adding a scaled identity component øla for small o > 0; other options for inducing invertibility include positivity constraints on the weights (Huang et al., 2018). Although the tanh non-linearities are invertible, numerical values close to 1) tend to introduce numerical instability during inversion; we address this issue by via activity regularization (Chollet et al., 2015). With these two architectural choices, we were able to compute both + and + mappings analytically and in a numerically stable way for modestly sized d's; we provide experiments below.
Invertible Residual Networks Recently, residual networks with spectral normalization have been proposed as a flexible invertible architecture (Behrmann et al., 2019). Although one of the two directions of the flow is not computable analytically, it may be approximated using fixed-point iteration algorithm. Invertible residual networks are typically trained using maximum likelihood; computing the determinant of the Jacobian of each layer requires sophisticated approximation based on Taylor series expansion. Interestingly, when training these flows using maximum likelihood, the "fast" direction needs be y - in order enable fast training, but generation becomes non-analytic; when training using an energy objective, the = + direction fast both for training and generation.
Rectangular Flow Architectures Recently, several authors explored rectangular flows, in which the dimensionality of and z is not equal (Nielsen et al., 2020; Cunningham and Fiterau, 2021; Caterini et al., 202 Training with maximum likelihood involves sophisticated extensions to the change of variables formula; however, these models can be trained without modification using an energy loss long one sample from them efficiently. At the same time, may retain their pseudo-invertibility to perform posterior inference.
F PSEUDOCODE FOR SEMI-AUTORFGIESSIVE FLOWS
Training:
Sampling:
G MOTIVATION FOR ENERGY FLOWS
Flows have a somewhat broader set of benefits and motivations:
1. Exact training using maximum likelihodoo (as opposed e.g., ELBO).
Accurate (exact) latent inference, i.c. obtaining "good" z from x. Note: This is a major challenge
in generative models: it motivates complex inference frameworks in GANs (e.g., BiGAN, ALI
20

---

Algorithm 1 Semi-Autoregreesiive Energy Flow Training
1: for iteration = 1, 2,... do
2: Generate z of random normals
3: for b = ......., Blocks do
4:
hs = cb (x<b)
5:
Yı (Z6; ho)
6: y's = Tz'6; ha)
7: Compute Energy Loss(yb: y' X6)
8: end for
9:
Backpropogate sum of energy loss 7
10:
end for
Algorithm 2 Semi- Semo--uttregrresivve Energy Flow Training
for 1, 2, ...,.. Blocks do
2:
h2 = (y<b)
y. TZz;
4: end for
(Dumoulin et al., 2016; Donahue et al., 2016)), and there is also line of research advanced variational posteriors ((<]zz) in VABs (e.g., AAEs, AVB, etc. (Makhzani et al., 2015; Mescheder et al., 2017)).
Exact likelihood evaluation, which supports using the likelihood as an evaluation metric (separately from the training objective, which could be different) for principled comparisons between models.
The key point is that switching to energy objectives loses point 1, but preserves points and 3, and also yields the following additional benefits:
The ability to use highly expressive invertible parameterizations beyond the ones that are currently compatible with maximum likelihood training. This can help better fit the data distribution and enables fast sampling.
5. Fast training via an objective that does not require computing determinants of Jacobians.
As result, we think that energy flows are conceptually interesting as a potential improvement over likelihood- based flows. Furthermore, view flows simply as bijective mappings between simple and a complex distribution and likelihood is a tool for training flows, not an intrinsic goal or defining feature. By questioning the of likelihood flows: expressivity, fast training and inference, exactly inferable latents.
objective, we improve truly important features of
Flows are defined to be bijective, but this often requires compromises on expressivity. Our work shows ways to avoid these compromises while retaining most benefits of flows.
From experiments in Table both feed-forward and autoregressive flows trained with the energy loss feature poor log- biglliillihoods. However, their FID and CRPS metrics are excellent: these models still generate very good images. We note that previous work has already shown that models trained with non-likelihood objectives tend to have very weak log-likeihhood;; for example in existing work on training normalizing flows with adversarial losses (Grover al., 2018). Our work adds additional results to this line of work.
may appear that energy flows do not retain key benefit normalizing dows-being able to compare models via their log- legliikel However, we argue that there are additional nuances. First, even though model comparison using log-likelihood is challenging, energy flows can still be used to study the successes and limitations of log-likelihood exactly log-likelihook fails for model evaluation- for example, recent diffusion models attain high log-liielsooods despite being trained in practice with objective that is not principled lower bound on the likelihood. Thus, there may be other architectures, modified objectives, and use cases for flows where log- ligeeiioldood caleulations will be useful.
metric. Additionally, it is unclear when
Finally, energy flows still retain combination of features that is rare among generative models: (1) exact posterior inference; (2) fast feed-forward generation; (3) stable and effective training
21

---

objective; (4) implicitly defined distributions (i.e. p(xiz) is not assumed to be of any parametric form); (5) equal support for continuous and discrete outputs: (6) equal support for continuous and discrete latents. Closest in terms of features to energy flows are Generative Moment Matching Networks (GMMNets (Dziugaite et al., 2015; Li et al., - work can be seen as adding principled way of doing posterior inference in GMMNets (they normally don't support it). Also related are VAEs; however they do not possess properties (4) and (6), and across our experiments generate more blurry samples (we can just swap out the objective and keep the same generator), as evidenced by FID and qualitative inspection. Other families like GANs and autoregressive models have even greater differences in terms of training stability and generation speed respectively.
22


Reviews:
[{"review_id": "_Nm5EA6WC5j", "reviewer": "Reviewer_Ljxv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author's establish a novel method to train normalizing flows without the need to compute the likelihood. Thereby, the computation of the Jacobian determinant of the flow is avoided, which can be expensive or requires a restricted architecture to be cheap to compute. They propose objectives inspired by two-sample tests, such as the energy loss, and show how they can be estimated and used for training. This objective can be used to train a variety of flows, and the authors focus on applying it to flows having Jacobian determinants that are expensive to compute. Moreover, they introduce a novel flow architecture, called semi-autoregressive flow, that can only be trained with their energy loss.\nIn their experiments, the authors demonstrate that their approach is competitive with other training procedures on UCI and MNIST datasets, while being often faster to train and evaluate.", "summary_of_the_review": "Due to the importance and the novelty of this method, I tend towards accepting this paper. However, I ask the author to address the concerns I voiced regarding the experimental evaluation\n\nEdit after rebuttal:\n\nI thank the reviewers for their response and additional experiments. I am still inclined to accept the article.", "strength_and_weaknesses": "### Strengths\n\nComputing the Jacobian determinant of normalizing flows is a challenge, but is required for most existing training procedures. Often, the architecture is designed such that the computation becomes simple or approximations are used, but this can limit expressivity and accuracy. The authors address this challenge in a creative way by proposing a novel training procedure based on two-sample test objectives.\nFurthermore, they demonstrate that such a training procedure unlocks new opportunities in terms of flow design by introducing a novel flow model that can only be trained with their method.\nThey apply their method to flows of various architectures and, thereby, demonstrate its effectiveness. Important is the runtime comparison, since the superior runtime is one of the main arguments for using their method.\n\n### Weaknesses\n\nMy main criticism concerns the experimental evaluation. The authors apply their method to various normalizing flows for which alternative training procedures exist, but the baseline models trained via maximum likelihood have a different architecture. There should also be experiments with the same flow architecture but different training methods being used. Moreover, the use of the CRPS and the NLL as evaluation objectives make it difficult to compare the work with other articles in the field, where the likelihood (UCI) or bits per dim (MNIST) are the main evaluation metrics being used. Computing them might be expensive for some of the models, but this is important information to evaluate the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YaOP9e3v3SR", "reviewer": "Reviewer_gK13", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel strategy to train invertible neural networks that model a distribution, a.k.a. normalizing flows. In particular, the authors propose to use proper metrics as the objective function to train flows. This approach relaxes the requirement of computing the determinant of the Jacobian of the flow and unlocks architecture for which computing this determinant would be too computationally demanding. \n\nAuthors strongly motivate their approach with theoretical arguments for the soundness of proper metrics. They also validate their approach experimentally, showing improvements in the sample quality compared to likelihood-based training. ", "summary_of_the_review": "Although the idea and experiments presented in this paper are interesting. I find the presentation too poor to argue for acceptance. Hence I argue for a weak reject.", "strength_and_weaknesses": "# Strengths\n- The idea presented in this paper is exciting and somehow novel. It efficiently trains invertible architectures while keeping the potential advantage of their invertibility at test time. The generalisation of the ideas from Si et al. (2022) to non-autoregressive architectures is well-exposed and has a practical interest.\n- The empirical results demonstrate that using the energy metric to train the flow is a good objective function if one is interested in good sample qualities (although it is not always clear how we define good samples) or by the marginal densities (as well represented by the CRPS score).\n\n# Weaknesses\nAlthough I think the ideas presented in the paper make sense and could have a real interest for some practical use cases, I find the paper's organisation and way of presenting ideas confusing. For instance, the tables' order does not follow the structure of the paper and the captions are uninformative, which makes it hard for the reader to follow the results at first glance. In addition, the discussions in section 3 go in many different directions and are missing a link between them. It is only at a second read that we can more or less grasp what each paragraph brings to the paper.\n\nI am also a bit confused by the argument from the authors that autoregressive architectures lead to \"good sample quality\". They can be good indeed. Still, I do not see why they are better than other generative models in terms of sample quality. \n\nThe authors say that MAF leads to gaussian distribution, which is only valid for a 1-step flow but is immediately relaxed if there are dependencies between variables and one uses more than one step of the flow. Indeed, MAF allows non-linear interaction between the variables. Thus the corresponding joint and marginal distribution can be non-Gaussian.\n\nI also think the author should mention that CRPS only reflects the quality of the marginal distribution and cannot say whether the model captures dependencies well.\n\nAt some point, the authors say that using the sliced objective (which I find a neat idea) reduces the computational complexity. Of course, this is true. However, it is unclear to which extent this is necessary, I would have liked to see numbers for that.\n\nI recommend that the authors create a better structure and story for this paper. I do think the ideas are excellent and (probably) well-motivated theoretically. However, the article in its current form is not pleasant to read, which I find sad for the quality of the ideas presented.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9XLQ8-UIY8", "reviewer": "Reviewer_B3hX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes training normalizing flows using sample-based objectives. ", "summary_of_the_review": "Since the paper focuses on sample quality and sample-based metrics, I feel the experiments on only tabular and MNIST are not sufficient. Furthermore, I'm not convinced this is better than the existing variety of GAN objectives such as MMD-GAN. The main difference seems to be the lack of a neural network for defining a kernel function; however, human-designed metrics tend to scale poorly to more complex data sets.", "strength_and_weaknesses": "\nWeakness:\n  - The use of IPM are standard in the GAN literarture. From what I understand, the main thing separating this work from those training invertible generative models with a discriminator (e.g. [1]), is that there are no parameters in this objective. In fact, the CPRS objective (Eq 3) has the same gradient as MMD, so the only reason this isn't adversarial is because unlike the MMD-GAN objective, this work does not make use of a neural network-defined kernel function.\n  - Related to the above, the only experiments are in tabular and MNIST, where I imagine simple kernels and distance metrics suffice. But for training on more complex distributions like colored images, a neural network discriminator is crucial. \n  - Furthermore, this work does not convincingly show that training without a discriminator is sufficient, since Flow-GAN [1] is not compared against.\n  - In terms of evaluation metrics, it would be better to use an evaluation metric that is different from the training objective for SAEFs, which heavily biases towards SAEF. Have the authors considered metrics such as MMD, NLL, or a kernel-based density estimator [2]?\n\n[1] \"Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models.\" Grover et al. (2018)\n\n[2] \"On the Quantitative Analysis of Decoder-Based Generative Models.\" Wu et al. (2016)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7RkuzKaRP9", "reviewer": "Reviewer_WiBH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses the energy loss for training normalizing flows. This produces a consistent estimator that doesn't require computing determinants of the Jacobian of the transformation, making it more flexible in terms of the choice of architecture. The paper also introduces a few tricks to further improve scaling and/or performance: slicing the data using 1D projections, and a semi-autoregressive flow model that interpolates between a full autoregressive model and a completely parallel model (i.e., a typical normalizing flow). Experiments on UCI and handwritten digit datasets show favorable results compared to baseline flows, VAEs, and autoregressive models, while allowing for fast sampling and exact posterior inference.", "summary_of_the_review": "The use of the energy loss is compelling, and the background behind it is nicely written. This approach frees normalizing flows from their previous architectural constraints, and this paper explores this idea through the introduction of a semi-autoregressive flow model that interpolates between faster sampling and better sample quality.\n\nI think that the ideas certainly have merit, however it feels like there is a bit of a gap between the promise of the paper and the actual demonstration of the approach. Specifically, the experiments are very small-scale for 2022, only operating on UCI datasets and MNIST. I'm surprised that CIFAR-10 wasn't included, as it is commonly used as a baseline for generative models. For example, one method referenced and compared with (Papamakarios et al., 2017) uses CIFAR-10, and it is now 5 years old. Normalizing flows (e.g., GLOW),  PixelCNN, and VAEs have all been scaled to more challenging datasets many years ago.\n\nI certainly appreciate a careful and detailed study of the estimator and its effects on small-scale data, however the conspicuous lack of larger-scale experiments tell me that either the authors have not yet run these experiments, or there is a severe limitation that is keeping the model from working on those datasets.\n\nI personally appreciate new and elegant ideas, and so all I'm asking for is an attempt on something a more challenging baseline. If the results are negative, then a discussion about reasons for this, and avenues for improvement would be necessary. The intention is to ensure that this paper is carried forward and does not languish because other researchers find it cannot scale well.\n\nIf there is a strong reason why the existing experiments are sufficient, then I would be happy to discuss this with the authors.\n\nMinor comments:\nI think you should add additional references for normalizing flows.\nLaurent Dinh, David Krueger, Yoshua Bengio, \"NICE: Non-linear Independent Components Estimation\", ICLR 2015\nLaurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, \"Density estimation using Real NVP\", ICLR 2017\n\n", "strength_and_weaknesses": "Strengths\n- The energy loss is straightforward, faster to compute, and produces good samples.\n- Slicing provides an effective way of scaling the training.\n- Semi-autoregressive flows elegantly trade-off computation and expressivity.\n- Exact posterior calculation is possible, allowing for things like interpolation that is not easy to do in autoregressive models.\n\nWeaknesses\n- The experiments are only done on UCI and handwritten digit datasets. At the very least, CIFAR-10 would have been good.\n- Comparisons to larger-scale flow models like GLOW would be more convincing.\n- There are key differences between the behavior of the energy loss and NLL (they are not strictly correlated). More exploration of where they differ in terms of allocation of probability mass would be nice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject