Title: Training Energy-Based Normalizing Flow with Score-Matching Objectives
Venue: NeurIPS 2023
Accepted: yes
Keywords: flow-based models

score-matching methods
Abstract: In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy-based normalizing flow (EBFlow). We demonstrate that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be entirely bypassed. This feature enables the use of arbitrary linear layers in the construction of flow-based models without increasing the computational time complexity of each training iteration from $\mathcal{O}(D^2L)$ to $\mathcal{O}(D^3L)$ for an $L$-layered model that accepts $D$-dimensional inputs. This makes the training of EBFlow more efficient than the commonly-adopted maximum likelihood training method. In addition to the reduction in runtime, we enhance the training stability and empirical performance of EBFlow through a number of techniques developed based on our analysis of the score-matching methods. The experimental results demonstrate that our approach achieves a significant speedup compared to maximum likelihood estimation while outperforming prior methods with a noticeable margin in terms of negative log-likelihood (NLL).

Training Energy-Based Normalizing Flow with Score-Matching Objectives

Abstract

In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy-based normalizing flow (EBFlow). We demonstrate that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be entirely bypassed. This feature enables the use of arbitrary linear layers in the construction of flow-based models without increasing the computational time complexity of each training iteration from O ( D 2 L ) to O ( D 3 L ) for an L -layered model that accepts D -dimensional inputs. This makes the training of EBFlow more efficient than the commonly-adopted maximum likelihood training method. In addition to the reduction in runtime, we enhance the training stability and empirical performance of EBFlow through a number of techniques developed based on our analysis of the score-matching methods. The experimental results demonstrate that our approach achieves a significant speedup compared to maximum likelihood estimation while outperforming prior methods with a noticeable margin in terms of negative log-likelihood (NLL).

1 Introduction

Parameter estimation for probability density functions (pdf) has been a major interest in the research fields of machine learning and statistics. Given a D -dimensional random data vector x ∈ R D , the goal of such a task is to estimate the true pdf p x ( · ) of x with a function p ( · ; θ ) parameterized by θ . In the studies of unsupervised learning, flow-based modeling methods (e.g., [1-4]) are commonly-adopted for estimating p x due to their expressiveness and broad applicability in generative tasks.

Flow-based models represent p ( · ; θ ) using a sequence of invertible transformations based on the change of variable theorem, through which the intermediate unnormalized densities are re-normalized by multiplying the Jacobian determinant associated with each transformation. In maximum likelihood estimation, however, the explicit computation of the normalizing term may pose computational challenges for model architectures that use linear transformations, such as convolutions [4, 5] and fully-connected layers [6, 7]. To address this issue, several methods have been proposed in the recent literature, which includes constructing linear transformations with special structures [8-12] and exploiting special optimization processes [7]. Despite their success in reducing the training complexity, these methods either require additional constraints on the linear transformations or biased estimation on the gradients of the objective.

Motivated by the limitations of the previous studies, this paper introduces an approach that reinterprets flow-based models as energy-based models [13], and leverages score-matching methods [14-17] to

optimize p ( · ; θ ) according to the Fisher divergence [14, 18] between p x ( · ) and p ( · ; θ ) . The proposed method avoids the computation of the Jacobian determinants of linear layers during training, and reduces the asymptotic computational complexity of each training iteration from O ( D 3 L ) to O ( D 2 L ) for an L -layered model. Our experimental results demonstrate that this approach significantly improves the training efficiency as compared to maximum likelihood training. In addition, we investigate a theoretical property of Fisher divergence with respect to latent variables, and propose a Match-after-Preprocessing (MaP) technique to enhance the training stability of score-matching methods. Finally, our comparison on the MNIST dataset [19] reveals that the proposed method exhibit significant improvements in comparison to our baseline methods presented in [17] and [7] in terms of negative log likelihood (NLL).

2 Background

In this section, we discuss the parameterization of probability density functions in flow-based and energy-based modeling methods, and offer a number of commonly-used training methods for them.

2.1 Flow-based Models

Flow-based models describe p x ( · ) using a prior distribution p u ( · ) of a latent variable u ∈ R D and an invertible function g = g L ◦ · · · ◦ g 1 , where g i ( · ; θ ) : R D → R D , ∀ i ∈ { 1 , · · · , L } and is usually modeled as a neural network with L layers. Based on the change of variable theorem and the distributive property of the determinant operation det ( · ) , p ( · ; θ ) can be described as follows:

where x i = g i ◦ · · · ◦ g 1 ( x ; θ ) , x 0 = x , J g ( x ; θ ) = ∂ ∂ x g ( x ; θ ) represents the Jacobian of g with respect to x , and J g i ( x i -1 ; θ ) = ∂ ∂ x i -1 g i ( x i -1 ; θ ) represents the Jacobian of the i -th layer of g with respect to x i -1 . This work concentrates on model architectures employing linear flows [20] to design the function g . These model architectures primarily utilize linear transformations to extract crucial feature representations, while also accommodating non-linear transformations that enable efficient Jacobian determinant computation. Specifically, let S l be the set of linear transformations in g , and S n = { g i | i ∈ { 1 , · · · , L }} \ S l be the set of non-linear transformations. The general assumption of these model architectures is that ∏ L i =1 | det ( J g i ) | in Eq. (1) can be decomposed as ∏ g i ∈S n | det ( J g i ) | ∏ g i ∈S l | det ( J g i ) | , where ∏ g i ∈S n | det ( J g i ) | and ∏ g i ∈S l | det ( J g i ) | can be calculated within the complexity of O ( D 2 L ) and O ( D 3 L ) , respectively. Previous implementations of such model architectures include Generative Flows (Glow) [4], Neural Spline Flows (NSF) [5], and the independent component analysis (ICA) models presented in [6, 7].

Given the parameterization of p ( · ; θ ) , a commonly used approach for optimizing θ is maximum likelihood (ML) estimation, which involves minimizing the Kullback-Leibler (KL) divergence D KL [ p x ( x ) ∥ p ( x ; θ )] = E p x ( x ) [ log p x ( x ) p ( x ; θ ) ] between the true density p x ( x ) and the parameterized density p ( x ; θ ) . The ML objective L ML ( θ ) is derived by removing the constant term E p x ( x ) [log p x ( x )] with respect to θ from D KL [ p x ( x ) ∥ p ( x ; θ )] , and can be expressed as follows:

The ML objective explicitly evaluates p ( x ; θ ) , which involves the calculation of the Jacobian determinant of the layers in S l . This indicates that certain model architectures containing convolutional [4, 5] or fully-connected layers [6, 7] may encounter training inefficiency due to the O ( D 3 L ) cost of evaluating ∏ g i ∈S l | det ( J g i ) | . Although a number of alternative methods discussed in Section 3 can be adopted to reduce their computational cost, they either require additional constraints on the linear transformation or biased estimation on the gradients of the ML objective.

2.2 Energy-based Models

Energy-based models are formulated based on a Boltzmann distribution, which is expressed as the ratio of an unnormalized density function to an input-independent normalizing constant. Specif-

ically, given a scalar-valued energy function E ( · ; θ ) : R D → R , the unnormalized density function is defined as exp( -E ( x ; θ )) , and the normalizing constant Z ( θ ) is defined as the integration ∫ x ∈ R D exp( -E ( x ; θ )) d x . The parameterization of p ( · ; θ ) is presented in the following equation:

Optimizing p ( · ; θ ) in Eq. (3) through directly evaluating L ML in Eq. (2) is computationally infeasible, since the computation requires explicitly calculating the intractable normalizing constant Z ( θ ) . To address this issue, a widely-used technique [13] is to reformulate ∂ ∂θ L ML ( θ ) as its sampling-based variant ∂ ∂θ L SML ( θ ) , which is expressed as follows:

where sg( · ) indicates the stop-gradient operator. Despite the fact that Eq. (4) prevents the calculation of Z ( θ ) , sampling from p ( · ; θ ) typically requires running a Markov Chain Monte Carlo (MCMC) process (e.g., [21, 22]) until convergence, which can still be computationally expensive as it involves evaluating the gradients of the energy function numerous times. Although several approaches [23, 24] were proposed to mitigate the high computational costs involved in performing an MCMC process, these approaches make use of approximations, which often cause training instabilities in highdimensional contexts [25].

Another line of researches proposed to optimize p ( · ; θ ) through minimizing the Fisher divergence D F [ p x ( x ) ∥ p ( x ; θ )] = E p x ( x ) [ 1 2 ∥ ∂ ∂ x log ( p x ( x ) p ( x ; θ ) ) ∥ 2 ] between p x ( x ) and p ( x ; θ ) using the scorematching (SM) objective L SM ( θ ) = E p x ( x ) [ 1 2 ∥ ∂ ∂ x E ( x ; θ ) ∥ 2 -Tr( ∂ 2 ∂ x 2 E ( x ; θ )) ] [14] to avoid the explicit calculation of Z ( θ ) as well as the sampling process required in Eq. (4). Several computationally efficient variants of L SM , including sliced score matching (SSM) [16], finite difference sliced score matching (FDSSM) [17], and denoising score matching (DSM) [15], have been proposed.

SSM is derived directly based on L SM with an unbiased Hutchinson's trace estimator [26]. Given a random projection vector v ∈ R D drawn from p v and satisfying E p v ( v ) [ v T v ] = I , the objective function denoted as L SSM , is defined as follows:

FDSSMisaparallelizable variant of L SSM that adopts the finite difference method [27] to approximate the gradient operations in the objective. Given a uniformly distributed random vector ε , it accelerates the calculation by simultaneously forward passing E ( x ; θ ) , E ( x + ε ; θ ) , and E ( x -ε ; θ ) as follows:

where p ξ ( ε ) = U ( ε ∈ R D | ∥ ε ∥ = ξ ), and ξ is a hyper-parameter that usually assumes a small value. DSM approximates the true pdf through a surrogate that is constructed using the Parzen density estimator p σ (˜ x ) [28]. The approximated target p σ (˜ x ) = ∫ x ∈ R D p σ (˜ x | x ) p x ( x ) d x is defined based on an isotropic Gaussian kernel p σ (˜ x | x ) = N (˜ x | x , σ 2 I ) with a variance σ 2 . The objective L DSM , which excludes the Hessian term in L SSM , is written as follows:

To conclude, L SSM is an unbiased objective that satisfies ∂ ∂θ L SSM ( θ ) = ∂ ∂θ D F [ p x ( x ) ∥ p ( x ; θ )] [16], while L FDSSM and L DSM require careful selection of hyper-parameters ξ and σ , since ∂ ∂θ L FDSSM ( θ ) = ∂ ∂θ ( D F [ p x ( x ) ∥ p ( x ; θ )] + o ( ξ )) [17] contains an approximation error o ( ξ ) , and p σ in ∂ ∂θ L DSM ( θ ) = ∂ ∂θ D F [ p σ (˜ x ) ∥ p (˜ x ; θ )] may bear resemblance to p x only for small σ [15].

3 Related Works

3.1 Accelerating Maximum Likelihood Training of Flow-based Models

A key focus in the field of flow-based modeling is to reduce the computational expense associated with evaluating the ML objective [7-12, 29]. These acceleration methods can be classified into two categories based on their underlying mechanisms.

Specially Designed Linear Transformations. A majority of the existing works [8-12, 29] have attempted to accelerate the computation of Jacobian determinants in the ML objective by exploiting linear transformations with special structures. For example, the authors in [8] proposed to constrain the weights in linear layers as lower triangular matrices to speed up training. The authors in [9, 10] proposed to adopt convolutional layers with masked kernels to accelerate the computation of Jacobian determinants. The authors in [29] leveraged orthogonal transformations to bypass the direct computation of Jacobian determinants. More recently, the authors in [12] proposed to utilize linear operations with special butterfly structures [30] to reduce the cost of calculating the determinants. Although these techniques avoid the O ( D 3 L ) computation, they impose restrictions on the learnable transformations, which potentially limits their capacity to capture complex feature representations, as discussed in [7, 31, 32]. Our experimental findings presented in Appendix A.5 support this concept, demonstrating that flow-based models with unconstrained linear layers outperform those with linear layers restricted by lower / upper triangular weight matrices [8] or those using lower-upper (LU) decomposition [4].

Specially Designed Optimization Process. To address the aforementioned restrictions, a recent study [7] proposed the relative gradient method for optimizing flow-based models with arbitrary linear transformations. In this method, the gradients of the ML objective are converted into their relative gradients by multiplying themselves with W T W , where W ∈ R D × D represents the weight matrix in a linear transformation. Since ∂ ∂ W log | det ( W ) | W T W = W , evaluating relative gradients is more computationally efficient than calculating the standard gradients according to ∂ ∂ W log | det ( W ) | = ( W T ) -1 . While this method reduces the training time complexity from O ( D 3 L ) to O ( D 2 L ) , a significant downside to this approach is that it introduces approximation errors with a magnitude of o ( W ) , which can escalate relative to the weight matrix values.

3.2 Training Flow-based Models with Score-Matching Objectives

The pioneering study [14] is the earliest attempt to train flow-based models by minimizing the SM objective. Their results demonstrate that models trained using the SM loss are able to achieve comparable or even better performance to those trained with the ML objective in a low-dimensional experimental setup. More recently, the authors in [16] and [17] proposed two efficient variants of the SM loss, i.e., the SSM and FDSSM objectives, respectively. They demonstrated that these loss functions can be used to train a non-linear independent component estimation (NICE) [1] model on high-dimensional tasks. While the training approaches of these works bear resemblance to ours, our proposed method places greater emphasis on training efficiency. Specifically, they directly implemented the energy function E ( x ; θ ) in the score-matching objectives as -log p ( x ; θ ) , resulting in a significantly higher computational cost compared to our method introduced in Section 4. In Section 5, we further demonstrate that the models trained with the methods in [16, 17] yield less satisfactory results in comparison to our approach.

4 Methodology

In this section, we introduce a new framework for reducing the training cost of flow-based models with linear transformations, and discuss a number of training techniques for enhancing its performance.

4.1 Energy-Based Normalizing Flow

Instead of applying architectural constraints to reduce computational time complexity, we achieve the same goal through adopting the training objectives of energy-based models. We name this approach as Energy-Based Normalizing Flow (EBFlow). A key observation is that the parametric density function of a flow-based model can be reinterpreted as that of an energy-based model through identifying

the input-independent multipliers in p ( · ; θ ) . Specifically, p ( · ; θ ) can be explicitly factorized into an unnormalized density and a corresponding normalizing term as follows:

(8)

where the energy function E ( · ; θ ) and the normalizing constant Z -1 ( θ ) are selected as follows:

The detailed derivations of Eqs. (8) and (9) are elaborated in Lemma A.11 of Section A.1.2. By isolating the computationally expensive term in p ( · ; θ ) as the normalizing constant Z ( θ ) , the parametric pdf defined in Eqs. (8) and (9) becomes suitable for the training methods of energy-based models. In the subsequent paragraphs, we discuss the training, inference, and convergence property of EBFlow.

Training Cost. Based on the definition in Eqs. (8) and (9), the score-matching objectives specified in Eqs. (5)-(7) can be adopted to prevent the Jacobian determinant calculation for the elements in S l . As a result, the training complexity can be significantly reduced to O ( D 2 L ) , as the O ( D 3 L ) calculation of Z ( θ ) is completely avoided. Such a design allows the use of arbitrary linear transformations in the construction of a flow-based model without posing computational challenge during the training process. This feature is crucial to the architectural flexibility of a flow-based model. For example, fully-connected layers and convolutional layers with arbitrary padding and striding strategies can be employed in EBFlow without increasing the training complexity. EBFlow thus exhibits an enhanced flexibility in comparison to the related works that exploit specially designed linear transformations.

Inference Cost. Although the computational cost of evaluating the exact Jacobian determinants of the elements in S l still requires O ( D 3 L ) time, these operations can be computed only once after training and reused for subsequent inferences, since Z ( θ ) is a constant as long as θ is fixed. In cases where D is extremely large and Z ( θ ) cannot be explicitly calculated, stochastic estimators such as the importance sampling techniques (e.g., [33, 34]) can be used as an alternative to approximate Z ( θ ) . We provide a brief discussion of such a scenario in Appendix A.3.

Asymptotic Convergence Property. Similar to maximum likelihood training, score-matching methods that minimize Fisher divergence have theoretical guarantees on their consistency [14, 16]. This property is essential in ensuring the convergence accuracy of the parameters. Let N be the number of independent and identically distributed (i.i.d.) samples drawn from p x to approximate the expectation in the SM objective. In addition, assume that there exists a set of optimal parameters θ ∗ such that p ( x ; θ ∗ ) = p x ( x ) . Under the regularity conditions (i.e., Assumptions A.1-A.7 shown in Appendix A.1.1), consistency guarantees that the parameters θ N minimizing the SM loss converges (in probability) to its optimal value θ ∗ when N →∞ , i.e., θ N p - → θ ∗ as N →∞ . In Appendix A.1.1, we provide a formal description of this property based on [16] and derive the sufficient condition for g and p u to satisfy the regularity conditions (i.e., Proposition A.10).

4.2 Techniques for Enhancing the Training of EBFlow

As revealed in the recent studies [16, 17], training flow-based models with score-matching objectives is challenging as the training process is numerically unstable and usually exhibits significant variances. To address these issues, we propose to adopt two techniques: match after preprocessing (MaP) and exponential moving average (EMA), which are particularly effective in dealing with the above issues according to our ablation analysis in Section 5.3.

MaP. Score-matching methods rely on the score function -∂ ∂ x E ( x ; θ ) to match ∂ ∂ x log p x ( x ) , which requires backward propagation through each layer in g . This indicates that the training process could be numerically sensitive to the derivatives of g . For instance, logit pre-processing layers commonly used in flow-based models (e.g., [1, 4, 5, 7, 8, 35]) exhibit extremely large derivatives near 0 and 1, which might exacerbate the above issue. To address this problem, we propose to

exclude the numerically sensitive layer(s) from the model and match the pdf of the pre-processed variable during training. Specifically, let x k ≜ g k ◦· · · ◦ g 1 ( x ) be the pre-processed variable, where k represents the index of the numerically sensitive layer. This method aims to optimize a parameterized pdf p k ( · ; θ ) ≜ p u ( g L ◦ · · · ◦ g k +1 ( · ; θ )) ∏ L i = k +1 | det ( J g i ) | that excludes ( g k , · · · , g 1 ) through minimizing the Fisher divergence between the pdf p x k ( · ) of x k and p k ( · ; θ ) by considering the (local) behavior of D F , as presented in Proposition 4.1.

Proposition 4.1. Let p x j be the pdf of the latent variable of x j ≜ g j ◦ · · · ◦ g 1 ( x ) indexed by j . In addition, let p j ( · ) be a pdf modeled as p u ( g L ◦ · · · ◦ g j +1 ( · )) ∏ L i = j +1 | det ( J g i ) | , where j ∈ { 0 , · · · , L -1 } . It follows that:

The derivation is presented in Appendix A.1.3. In Section 5.3, we validate the effectiveness of the MaP technique on the score-matching methods formulated in Eqs. (5)-(7) through an ablation analysis. Please note that MaP does not affect maximum likelihood training, since it always satisfies D KL [ p x j ∥ p j ] = D KL [ p x ∥ p 0 ] , ∀ j ∈ { 1 , · · · , L -1 } as revealed in Lemma A.12.

EMA. In addition to the MaP technique, we have also found that the exponential moving average (EMA) technique introduced in [36] is effective in improving the training stability. EMA enhances the stability through smoothly updating the parameters based on ˜ θ ← m ˜ θ +(1 -m ) θ i at each training iteration, where ˜ θ is a set of shadow parameters [36], θ i is the model's parameters at iteration i , and m is the momentum parameter. In our experiments presented in Section 5, we adopt m = 0 . 999 for both EBFlow and the baselines.

5 Experiments

In the following experiments, we first compare the training efficiency of the baselines trained with L ML and EBFlow trained with L SML , L SSM , L FDSSM , and L DSM to validate the effectiveness of the proposed method in Sections 5.1 and 5.2. Then, in Section 5.3, we provide an ablation analysis of the techniques introduced in Section 4.2, and a performance comparison between EBFlow and a number of related studies [7, 16, 17]. Finally, in Section 5.4, we discuss how EBFlow can be applied to generation tasks. Please note that the performance comparison with [8-12, 29] is omitted, since their methods only support specialized linear layers and are not applicable to the employed model architecture [7] that involves fully-connected layers. The differences between EBFlow, the baseline, and the related studies are summarized in Table A4 in the appendix. The sampling process involved in the calculation of L SML is implemented by g -1 ( u ; θ ) , where u ∼ p u . The transformation g ( · ; θ ) for each task is designed such that S l = ϕ and S n = ϕ . For more details about the experimental setups, please refer to Appendix A.2.

̸

̸

5.1 Density Estimation on Two-Dimensional Synthetic Examples

In this experiment, we examine the performance of EBFlow and its baseline on three two-dimensional synthetic datasets. These data distributions are formed using Gaussian smoothing kernels to ensure p x ( x ) is continuous and the true score function ∂ ∂ x log p x ( x ) is well defined. The model g ( · ; θ ) is constructed using the Glow model architecture [4], which consists of actnorm layers, affine coupling layers, and fully-connected layers. The performance are evaluated in terms of the KL divergence and the Fisher divergence between p x ( x ) and p ( x ; θ ) using independent and identically distributed (i.i.d.) testing sample points.

True

EBFlow

Baseline

(ML)

(SML)

EBFlow

(SSM)

EBFlow

(DSM)

EBFlow

(FDSSM)

>

Table 1 and Fig. 1 demonstrate the results of the above setting. The results show that the performance of EBFlow trained with L SSM , L FDSSM , and L DSM in terms of KL divergence is on par with those trained using L SML as well as the baselines trained using L ML . These results validate the efficacy of training EBFlow with score matching.

5.2 Efficiency Evaluation on the MNIST and CIFAR-10 Datasets

In this section, we inspect the influence of data dimension D on the training efficiency of flow-based models. To provide a thorough comparison, we employ two types of model architectures and train them on two datasets with different data dimensions: the MNIST [19] ( D = 1 × 28 × 28 ) and CIFAR-10 [37] ( D = 3 × 32 × 32 ) datasets.

The first model architecture is exactly the same as that adopted by [7]. It is an architecture consisting of two fully-connected layers and a smoothed leaky ReLU non-linear layer in between. The second model is a parametrically efficient variant of the first model. It replaces the fully-connected layers with convolutional layers and increases the depth of the model to six convolutional blocks. Between every two convolutional blocks, a squeeze operation [2] is inserted to enlarge the receptive field. In the following paragraphs, we refer to these models as 'FC-based' and 'CNN-based' models, respectively.

The performance of the FC-based and CNN-based models is measured using the negative log likelihood (NLL) metric (i.e., E p x ( x ) [ -log p ( x ; θ )] ), which differs from the intractable KL divergence by a constant. In addition, its normalized variant, the Bits/Dim metric [38], is also measured and reported. The algorithms are implemented using PyTorch [39] with automatic differentiation [40], and the runtime is measured on NVIDIA Tesla V100 GPUs. In the subsequent paragraphs, we assess the models through scalability analysis, performance evaluation, and training efficiency examination.

Scalability. To demonstrate the scalability of KL-divergence-based (i.e., L ML and L SML ) and Fisherdivergence-based (i.e., L SSM , L DSM , and L FDSSM ) objectives used in EBFlow and the baseline method, we first present a runtime comparison for different choices of the input data size D . The results presented in Fig. 2 (a) reveal that Fisher-divergence-based objectives can be computed more efficiently than KL-divergence-based objectives. Moreover, the sampling-based objective L SML used in EBFlow, which excludes the calculation of Z ( θ ) in the computational graph, can be computed slightly faster than L ML adopted by the baseline.

Runtime

(Sec./Batch)

Dimension (3, n, n)

EBFlow (SSM)

Baseline (ML)

EBFlow (SML)

EBFlow (FDSSM)

EBFlow (DSM)

CNN-based

FC-based

Dimension (3, n, n)

(a)

CNN-based

FC-based

MNIST

CIFAR-10

Wall Time (hours)

Wall Time (hours)

Bits / Dim

Bits / Dim

(b)

Runtime

(Sec./Batch)

Dimension (1, n, n)

Dimension (1, n, n)

Wall Time (hours)

Wall Time (hours)

W/ MaP Technique

W/o MaP Technique

Training Steps

Gradient Norm

Performance. Table 2 demonstrates the performance of the FC-based and CNN-based models in terms of NLL on the MNIST and CIFAR-10 datasets. The results show that the models trained with Fisher-divergence-based objectives are able to achieve similar performance as those trained with KL-divergence-based objectives. Among the Fisher-divergence-based objectives, the models trained using L SSM and L DSM are able to achieve better performance in comparison to those trained using L FDSSM . The runtime and performance comparisons above suggest that L SSM and L DSM can deliver better training efficiency than L ML and L SML , since the objectives can be calculated faster while maintaining the models' performance on the NLL metric.

Training Efficiency. Fig. 2 (b) presents the trends of NLL versus training wall time when L ML , L SML , L SSM , L DSM , and L FDSSM are adopted as the objectives. It is observed that EBFlow trained with SSM and DSM consistently attain better NLL in the early stages of the training. The improvement is especially notable when both D and L are large, as revealed for the scenario of training CNN-based models on the CIFAR-10 dataset. These experimental results provide evidence to support the use of score-matching methods for optimizing EBFlow.

5.3 Analyses and Comparisons

Ablation Study. Table 3 presents the ablation results that demonstrate the effectiveness of the EMA and MaP techniques. It is observed that EMA is effective in reducing the variances. In addition, MaP significantly improves the overall performance. To further illustrate the influence of the proposed MaP technique on the score-matching methods, we compare the optimization pro-

(a)

(b)

Mask: KMNIST

Mask: MNIST

7416

cesses with ∂ ∂θ D F [ p x k ∥ p k ] and ∂ ∂θ D F [ p x ∥ p 0 ] = ∂ ∂θ E p x k ( x k ) [ 1 2 ∥ ( ∂ ∂ x k log( p x k ( x k ) p k ( x k ) )) ∏ k i =1 J g i ∥ 2 ] (i.e., Lemma A.13) by depicting the norm of their unbiased estimators ∂ ∂θ L SSM ( θ ) calculated with and without applying the MaP technique in Fig. 3. It is observed that the magnitude of ∥ ∥ ∂ ∂θ L SSM ( θ ) ∥ ∥ significantly decreases when MaP is incorporated into the training process. This could be attributed to the fact that the calculation of ∂ ∂θ D F [ p x k ∥ p k ] excludes the calculation of ∏ k i =1 J g i in ∂ ∂θ D F [ p x ∥ p 0 ] , which involves computing the derivatives of the numerically sensitive logit pre-processing layer.

Comparison with Related Works. Table 4 compares the performance of our method with a number of related works on the MNIST dataset. Our models trained with score-matching objectives using the same model architecture exhibit improved performance in comparison to the relative gradient method [7]. In addition, when compared to the results in [16] and [17], our models deliver significantly improved performance over them. Please note that the results of [7, 16, 17] presented in Table 4 are obtained from their original papers.

5.4 Application to Generation Tasks

The sampling process of EBFlow can be accomplished through the inverse function or an MCMC process. The former is a typical generation method adopted by flow-based models, while the latter is a more flexible sampling process that allows conditional generation without re-training the model. In the following paragraphs, we provide detailed explanations and visualized results of these tasks.

Inverse Generation. One benefit of flow-based models is that g -1 can be directly adopted as a generator. While inverting the weight matrices in linear transformations typically demands time complexity of O ( D 3 L ) , these inverse matrices are only required to be computed once θ has converged, and can then be reused for subsequent inferences In this experiment, we adopt the Glow [4] model architecture and train it using our method with L SSM on the MNIST dataset. We compare our visualized results with the current best flow-based model trained using the score matching objective [17]. The results of [17] are generated using their officially released code with their best setup (i.e., FDSSM). As presented in Fig. 4, the results generated using our model demonstrate significantly better visual quality than those of [17].

MCMCGeneration. In comparison to the inverse generation method, the MCMC sampling process is more suitable for conditional generation tasks such as data imputation due to its flexibility [41]. For the imputation task, a data vector x is separated as an observable part x O and a masked part x M . The goal of imputation is to generate the masked part x M based on the observable part x O . To achieve this goal, one can perform a Langevin MCMC process to update x M according to the gradient of the energy function ∂ ∂ x E ( x ; θ ) . Given a noise vector z sampled from N ( 0 , I ) and a

small step size α , the process iteratively updates x M based on the following equation:

where x ( t ) M represents x M at iteration t ∈ { 1 , · · · , T } , and T is the total number of iterations. MCMC generation requires an overall cost of O ( TD 2 L ) , potentially more economical than the O ( D 3 L ) computation of the inverse generation method. Fig. 5 depicts the imputation results of the FC-based model trained using L DSM on the CelebA [42] dataset ( D = 3 × 64 × 64 ). In this example, we implement the masking part x M using the data from the KMNIST [43] and MNIST [19] datasets.

6 Conclusion

In this paper, we presented EBFlow, a new flow-based modeling approach that associates the parameterization of flow-based and energy-based models. We showed that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be bypassed, resulting in an improved training time complexity. In addition, we demonstrated that the training stability and performance can be effectively enhanced through the MaP and EMA techniques. Based on the improvements in both theoretical time complexity and empirical performance, our method exhibits superior training efficiency compared to maximum likelihood training.

References

L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear Independent Components Estimation, 2015.

L. Dinh, J. N. Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. In Proc. Int. Conf. on Learning Representations (ICLR) , 2016.

G. Papamakarios, I. Murray, and T. Pavlakou. Masked Autoregressive Flow for Density Estimation. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2017.

D. P. Kingma and P. Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2018.

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2019.

A. Hyvärinen and E. Oja. Independent Component Analysis: Algorithms and Applications. Neural Networks: the Official Journal of the International Neural Network Society , 13 4-5:41130, 2000.

L. Gresele, G. Fissore, A. Javaloy, B. Schölkopf, and A. Hyvärinen. Relative Gradient Optimization of the Jacobian Term in Unsupervised Deep Learning. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2020.

Y. Song, C. Meng, and S. Ermon. MintNet: Building Invertible Neural Networks with Masked Convolutions. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2019.

E. Hoogeboom, R. v. d. Berg, and M. Welling. Emerging Convolutions for Generative Normalizing Flows. In Proc. Int. Conf. on Machine Learning (ICML) , 2019.

X. Ma and E. H. Hovy. MaCow: Masked Convolutional Generative Flow. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2019.

Y. Lu and B. Huang. Woodbury Transformations for Deep Generative Flows. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2020.

C. Meng, L. Zhou, K. Choi, T. Dao, and S. Ermon. ButterflyFlow: Building Invertible Layers with Butterfly Matrices. In Proc. Int. Conf. on Machine Learning (ICML) , 2022.

Y. LeCun, S. Chopra, R. Hadsell, A. Ranzato, and F. J. Huang. A Tutorial on Energy-Based Learning. 2006.

A. Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research (JMLR) , 6(24):695-709, 2005.

P. Vincent. A Connection between Score Matching and Denoising Autoencoders. Neural computation , 23(7):1661-1674, 2011.

Y. Song, S. Garg, J. Shi, and S. Ermon. Sliced Score Matching: A Scalable Approach to Density and Score Estimation. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI) , 2019.

T. Pang, K. Xu, C. Li, Y. Song, S. Ermon, and J. Zhu. Efficient Learning of Generative Models via Finite-Difference Score Matching. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2020.

S. Lyu. Interpretation and Generalization of Score Matching. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI) , 2009.

L. Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

G. Papamakarios, E. T. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research (JMLR) , 22:57:1-57:64, 2019.

G. O Roberts and R. L. Tweedie. Exponential Convergence of Langevin Distributions and Their Discrete Approximations. Bernoulli , 2(4):341 - 363, 1996.

G. O Roberts and J. S Rosenthal. Optimal Scaling of Discrete Approximations to Langevin Diffusions. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 60(1):255268, 1998.

G. E. Hinton. Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation , 14:1771-1800, 2002.

T. Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proc. Int. Conf. on Machine Learning (ICML) , 2008.

Y. Du, S. Li, B. J. Tenenbaum, and I. Mordatch. Improved Contrastive Divergence Training of Energy Based Models. In Proc. Int. Conf. on Machine Learning (ICML) , 2021.

M. F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics-Simulation and Computation , 18(3):10591076, 1989.

A. Neumaier. Introduction to Numerical Analysis . Cambridge University Press, 2001.

E. Parzen. On Estimation of a Probability Density Function and Mode. Annals of Mathematical Statistics , 33:1065-1076, 1962.

J. M. Tomczak and M. Welling. Improving Variational Auto-Encoders using Householder Flow. ArXiv , abs/1611.09630, 2016.

T. Dao, A. Gu, M. Eichhorn, A. Rudra, and C. Ré. Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations. In Proc. Int. Conf. on Machine Learning (ICML) , 2019.

J. Behrmann, D. K. Duvenaud, and J.-H. Jacobsen. Invertible Residual Networks. In Proc. Int. Conf. on Machine Learning (ICML) , 2018.

R. T. Q. Chen, J. Behrmann, D. K. Duvenaud, and J.-H. Jacobsen. Residual Flows for Invertible Generative Modeling. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2019.

R. M. Neal. Annealed Importance Sampling. Statistics and Computing , 11:125-139, 1998.

Y. Burda, R. B. Grosse, and R. Salakhutdinov. Accurate and Conservative Estimates of MRF Log-Likelihood using Reverse Annealing. volume abs/1412.8566, 2015.

W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. K. Duvenaud. FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. In Int. Conf. on Learning Representations (ICLR) , 2018.

Y. Song and S. Ermon. Improved Techniques for Training Score-Based Generative Models. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2020.

A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.

A. V. D. Oord, N. Kalchbrenner, L. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves. Conditional Image Generation with PixelCNN Decoders. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2016.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2019.

A. Griewank and A. Walther. Evaluating Derivatives - Principles and Techniques of Algorithmic Differentiation, Second Edition. In Frontiers in applied mathematics , 2000.

A. M Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR) , 2016.

Z. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild. In Proc. Int. Conf. on Computer Vision (ICCV) , December 2015.

T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep Learning for Classical Japanese Literature. In Proc. Conf. on Neural Information Processing Systems (NeurIPS) , 2018.

T. Anderson Keller, Jorn W. T. Peters, Priyank Jaini, Emiel Hoogeboom, Patrick Forr'e, and Max Welling. Self Normalizing Flows. In Proc. Int. Conf. on Machine Learning (ICML) , 2020.

C.-H. Chao, W.-F. Sun, B.-W. Cheng, Y.-C. Lo, C.-C. Chang, Y.-L. Liu, Y.-L. Chang, C.-P. Chen, and C.-Y. Lee. Denoising Likelihood Score Matching for Conditional Score-based Data Generation. In Proc. Int. Conf. on Learning Representations (ICLR) , 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. CoRR , abs/1412.6980, 2014.

I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In Proc. Int. Conf. on Learning Representations (ICLR) , 2017.

M. Ning, E. Sangineto, A. Porrello, S. Calderara, and R. Cucchiara. Input Perturbation Reduces Exposure Bias in Diffusion Models. In Proc. Int. Conf. on Machine Learning (ICML) , 2023.

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software available from tensorflow.org.

J. Sohl-Dickstein. Two Equalities Expressing the Determinant of a Matrix in terms of Expectations over Matrix-Vector Products, 2020.

K. P. Murphy. Probabilistic Machine Learning: Advanced Topics , page 811. MIT Press, 2023.

M. Zhang, O. Key, P. Hayes, D. Barber, B. Paige, and F.-X. Briol. Towards Healing the Blindness of Score Matching. Workshop on Score-Based Methods at Conf. on Neural Information Processing Systems (NeurIPS) , 2022.

A Appendix

A.1 Derivations

In the following subsections, we provide theoretical derivations. In Section A.1.1, we discuss the asymptotic convergence properties as well as the assumptions of score-matching methods. In Section A.1.2, we elaborate on the formulation of EBFlow (i.e., Eqs. (8) and (9)), and provide a explanation of their interpretation. Finally, in Section A.1.3, we present a theoretical analysis of KL divergence and Fisher divergence, and discuss the underlying mechanism behind the proposed MaP technique.

A.1.1 Asymptotic Convergence Property of Score Matching

In this subsection, we provide a formal description of the consistency property of score matching. The description follows [16] and the notations are replaced with those used in this paper. The regularity conditions for p ( · ; θ ) are defined in Assumptions A.1 ∼ A.7. In the following paragraph, the parameter space is defined as Θ . In addition, s ( x ; θ ) ≜ ∂ ∂ x log p ( x ; θ ) = -∂ ∂ x E ( x ; θ ) represents the score function. ˆ L SM ( θ ) ≜ 1 N ∑ N k =1 f ( x k ; θ ) denotes an unbiased estimator of L SM ( θ ) , where f ( x ; θ ) ≜ 1 2 ∥ ∥ ∂ ∂ x E ( x ; θ ) ∥ ∥ 2 -Tr ( ∂ 2 ∂ x 2 E ( x ; θ ) ) = 1 2 ∥ s ( x ; θ ) ∥ 2 +Tr ( ∂ ∂ x s ( x ; θ ) ) and { x 1 , · · · , x N } represents a collection of i.i.d. samples drawn from p x . For notational simplicity, we denote ∂h ( x ; θ ) ≜ ∂ ∂ x h ( x ; θ ) and ∂ i h j ( x ; θ ) ≜ ∂ ∂ x i h j ( x ; θ ) , where h j ( x ; θ ) denotes the j -th element of h .

Assumption A.1. (Positiveness) p ( x ; θ ) > 0 and p x ( x ) > 0 , ∀ θ ∈ Θ , ∀ x ∈ R D .

Assumption A.2. (Regularity of the score functions) The parameterized score function s ( x ; θ ) and the true score function ∂ ∂ x log p x ( x ) are both continuous and differentiable. In addition, their expectations E p x ( x ) [ s ( x ; θ )] and E p x ( x ) [ ∂ ∂ x log p x ( x ) ] are finite. (i.e., E p x ( x ) [ s ( x ; θ )] < ∞ and E p x ( x ) [ ∂ ∂ x log p x ( x ) ] < ∞ )

Assumption A.3. (Boundary condition) lim ∥ x ∥→∞ p x ( x ) s ( x ; θ ) = 0 , ∀ θ ∈ Θ .

Assumption A.4. (Compactness) The parameter space Θ is compact.

Assumption A.5. (Identifiability) There exists a set of parameters θ ∗ such that p x ( x ) = p ( x ; θ ∗ ) , where θ ∗ ∈ Θ , ∀ x ∈ R D .

̸

Assumption A.6. (Uniqueness) θ = θ ∗ ⇔ p ( x ; θ ) = p ( x ; θ ∗ ) , where θ, θ ∗ ∈ Θ , x ∈ R D .

̸

Assumption A.7. (Lipschitzness of f ) The function f is Lipschitz continuous w.r.t. θ , i.e., | f ( x ; θ 1 ) -f ( x ; θ 2 ) | ≤ L ( x ) ∥ θ 1 -θ 2 ∥ 2 , ∀ θ 1 , θ 2 ∈ Θ , where L ( x ) represents a Lipschitz constant satisfying E p x ( x ) [ L ( x )] < ∞ .

Theorem A.8. (Consistency of a score-matching estimator [16]) The score-matching estimator θ N ≜ argmin θ ∈ Θ ˆ L SM is consistent, i.e.,

Assumptions A.1 ∼ A.3 are the conditions that ensure ∂ ∂θ D F [ p x ( x ) ∥ p ( x ; θ )] = ∂ ∂θ L SM ( θ ) . Assumptions A.4 ∼ A.7 lead to the uniform convergence property [16] of a score-matching estimator, which gives rise to the consistency property. The detailed derivation can be found in Corollary 1 in [16]. In the following Lemma A.9 and Proposition A.10, we examine the sufficient condition for g and p u to satisfy Assumption A.7.

Lemma A.9. (Sufficient condition for the Lipschitzness of f ) The function f ( x ; θ ) = 1 2 ∥ s ( x ; θ ) ∥ 2 + Tr ( ∂ ∂ x s ( x ; θ ) ) is Lipschitz continuous if the score function s ( x ; θ ) satisfies the following conditions: ∀ θ, θ 1 , θ 2 ∈ Θ , ∀ i ∈ { 1 , · · · , D } ,

where L 1 , L 2 , and L 3 are Lipschitz constants satisfying E p x ( x ) [ L 1 ( x )] < ∞ , E p x ( x ) [ L 2 ( x )] < ∞ , and E p x ( x ) [ L 3 ( x )] < ∞ .

Proof. The Lipschitzness of f can be guaranteed by ensuring the Lipschitzness of ∥ s ( x ; θ ) ∥ 2 2 and Tr( ∂s ( x ; θ )) .

Step 1. (Lipschitzness of ∥ s ( x ; θ ) ∥ 2 )

where ( i ) is based on triangle inequality, ( ii ) is due to Cauchy-Schwarz inequality, and ( iii ) follows from the listed assumptions.

Step 2. (Lipschitzness of Tr( ∂s ( x ; θ )) )

where ( i ) holds by Von Neumann's trace inequality. ( ii ) is due to the property ∥ A ∥ 2 ≤ √ ∑ i ∥ a i ∥ 2 2 , where a i is the column vector of A . ( iii ) holds by the listed assumptions.

Based on Steps 1 and 2, the Lipschitzness of f is guaranteed, since

Proposition A.10. (Sufficient condition for the Lipschitzness of f ) The function f is Lipschitz continuous if g ( x ; θ ) has bounded first, second, and third-order derivatives, i.e., ∀ i, j ∈ { 1 , · · · , D } , ∀ θ ∈ Θ .

and smooth enough on Θ , i.e., θ 1 , θ 2 ∈ Θ :

In addition, it satisfies the following conditions:

where J -1 g represents the inverse matrix of J g . Furthermore, the prior distribution p u satisfies:

where s u ( u ) ≜ ∂ ∂ u log p u ( u ) is the score function of p u . The Lipschitz constants listed above (i.e., l 1 ∼ l 3 , r 0 ∼ r 3 , l ′ 1 ∼ l ′ 2 , and r ′ 1 ∼ r ′ 2 ) have finite expectations.

Proof. We show that the sufficient conditions stated in Lemma A.9 can be satisfied using the conditions listed above.

Step 1. (Sufficient condition of ∥ s ( x ; θ ) ∥ 2 ≤ L 1 ( x ) )

Since ∥ s ( x ; θ ) ∥ 2 = ∥ ∥ ∂ ∂ x log p u ( g ( x ; θ )) + ∂ ∂ x log | det J g ( x ; θ ) | ∥ ∥ 2 ≤ ∥ ∥ ∂ ∂ x log p u ( g ( x ; θ )) ∥ ∥ 2 + ∥ ∥ ∂ ∂ x log | det J g ( x ; θ ) | ∥ ∥ 2 , we first demonstrate that ∥ ∥ ∂ ∂ x log p u ( g ( x ; θ )) ∥ ∥ 2 and ∥ ∥ ∂ ∂ x log | det J g ( x ; θ ) | ∥ ∥ 2 are both bounded.

(1.1) ∥ ∥ ∂ ∂ x log p u ( g ( x ; θ )) ∥ ∥ 2 is bounded:

(1.2) ∥ ∥ ∂ log | det J g ( x ; θ ) | ∥ ∥

where ( i ) is derived using Jacobi's formula, and v i ( x ; θ ) = Tr ( J -1 g ( x ; θ ) ∂ i J g ( x ; θ ) ) .

where ( i ) holds by Von Neumann's trace inequality, ( ii ) is due to the property of matrix norm, and ( iii ) is follows from the listed assumptions.

Step 2. (Sufficient condition of the Lipschitzness of s ( x ; θ ) )

Since s ( x ; θ ) = ∂ ∂ x log p u ( g ( x ; θ )) + ∂ ∂ x log | det J g ( x ; θ ) | , we demonstrate that ∂ ∂ x log p u ( g ( x ; θ )) and ∂ ∂ x log | det J g ( x ; θ ) | are both Lipschitz continuous on Θ .

(2.1) Lipschitzness of ∂ ∂ x log p u ( g ( x ; θ )) :

where ( i ) is obtained using a similar derivation to Step 1 in Lemma A.9, while ( ii ) follows from the listed assumptions.

(2.2) Lipschitzness of ∂ ∂ x log | det J g ( x ; θ ) | :

Let M ( i, x ; θ ) ≜ J -1 g ( x ; θ 1 ) ∂ i J g ( x ; θ ) . We first demonstrate that M is Lipschitz continuous:

where ( i ) is obtained by an analogous derivation of the step 1 in Lemma A.9, and ( ii ) holds by the listed assumption.

The Lipschitzness of M leads to the Lipschitzness of ∂ ∂ x log | det J g ( x ; θ ) | , since:

where ( i ) holds by Von Neumann's trace inequality, ( ii ) is due to the Lipschitzness of M

Step 3. (Sufficient condition of the Lipschitzness of ∂ i s ( x ; θ )

. )

∂ i s ( x ; θ ) can be decomposed as ( ∂ i s u ( g ( x ; θ ))) T J g ( x ; θ ) , ( s u ( g ( x ; θ ))) T ∂ i J g ( x ; θ ) , and ∂ i [ v ( x ; θ )] as follows:

(3.1) The Lipschitzness of ( ∂ i s u ( g ( x ; θ ))) T J g ( x ; θ ) and ( s u ( g ( x ; θ ))) T ∂ i J g ( x ; θ ) can be derived using proofs similar to that in Step 2.1:

(3.2) Lipschitzness of ∂ i [ v ( x ; θ )] :

Let ∂ i [ v j ( x ; θ )] ≜ ∂ i Tr( M ( j, x ; θ )) = Tr ( ∂ i M ( j, x ; θ )) . We first show that ∂ i M ( j, x ; θ ) can be decomposed as:

The Lipschitz constant of ∂ i M equals to ( l ′ 2 ( x ) r 2 ( x ) + l 2 ( x ) r ′ 2 ( x ) ) + ( l ′ 1 ( x ) r 3 ( x ) + l 3 ( x ) r ′ 1 ( x ) ) based on a similar derivation as in Step 3.1. The Lipschitzness of ∂ i M ( j, x ; θ ) leads to the Lipschitzness of ∂ i [ v ( x ; θ )] :

where ( i ) holds by Von Neumann's trace inequality, ( ii ) is due to the Lipschitzness of ∂ i M .

A.1.2 Derivation of Eqs. (8) and (9)

Energy-based models are formulated based on the observation that any continuous pdf p ( x ; θ ) can be expressed as a Boltzmann distribution exp( -E ( x ; θ )) Z -1 ( θ ) [13], where the energy function E ( · ; θ ) can be modeled as any scalar-valued continuous function. In EBFlow, the energy function E ( x ; θ ) is selected as -log( p u ( g ( x ; θ )) ∏ g i ∈S n | det( J g i ( x i -1 ; θ )) | ) according to Eq. (9). This suggests that the normalizing constant Z ( θ ) = ∫ exp( -E ( x ; θ )) d x is equal to ( ∏ g i ∈S l | det( J g i ( θ )) | ) -1 according to Lemma A.11.

Lemma A.11.

Proof.

By multiplying ( ∏ g i ∈S l | det( J g i ( θ )) | ) -1 to both sides of the equation, we arrive at the conclusion:

pdf (model)

point-wise

density

evaluation

variable

pdf (data)

point-wise

density

evaluation

Fisher divergence

KL divergence

(in general)

(in general)

A.1.3 Theoretical Analyses of KL Divergence and Fisher Divergence

In this section, we provide formal derivations for Proposition 4.1, Lemma A.12, and Lemma A.13. To ensure a clear presentation, we provide a visualization of the relationship between the variables used in the subsequent derivations in Fig. A1.

LemmaA.12. Let p x j be the pdf of the latent variable of x j ≜ g j ◦· · ·◦ g 1 ( x ) indexed by j . In addition, let p j ( · ) be a pdf modeled as p u ( g L ◦ · · · ◦ g j +1 ( · )) ∏ L i = j +1 | det ( J g i ) | , where j ∈ { 0 , · · · , L -1 } . It follows that:

Proof. The equivalence D KL [ p x ∥ p 0 ] = D KL [ p x j ∥ p j ] holds for any j ∈ { 1 , · · · , L -1 } since:

where ( i ) is due to the property that E p x ( x ) [ f ◦ g j ◦ · · · ◦ g 1 ( x )] = E p x j ( x j ) [ f ( x j )] for a given function f . Therefore, D KL [ p x j ∥ p j ] = D KL [ p x ∥ p 0 ] , ∀ j ∈ { 1 , · · · , L -1 } .

LemmaA.13. Let p x j be the pdf of the latent variable of x j ≜ g j ◦· · ·◦ g 1 ( x ) indexed by j . In addition, let p j ( · ) be a pdf modeled as p u ( g L ◦ · · · ◦ g j +1 ( · )) ∏ L i = j +1 | det ( J g i ) | , where j ∈ { 0 , · · · , L -1 } . It follows that:

Proof. Based on the definition, the Fisher divergence between p x and p 0 is written as: D F [ p x ∥ p 0 ]

where j for a given function f .

̸

Remark A.14 . Lemma A.13 implies that D F [ p x j ∥ p j ] = D F [ p x ∥ p 0 ] in general, as the latter contains an additional multiplier ∏ j i =1 J g i as shown below:

Proposition 4.1. Let p x j be the pdf of the latent variable of x j ≜ g j ◦ · · · ◦ g 1 ( x ) indexed by j . In addition, let p j ( · ) be a pdf modeled as p u ( g L ◦ · · · ◦ g j +1 ( · )) ∏ L i = j +1 | det ( J g i ) | , where j ∈ { 0 , · · · , L -1 } . It follows that:

Proof. Based on Remark A.14, the following holds:

where ( i ) and ( ii ) both result from the positiveness condition presented in Assumption A.1. Specifically, for ( i ) , p x j ( x j ) = p x ( g -1 1 ◦ · · · ◦ g -1 j ( x j )) ∏ j i =1 ∣ ∣ ∣ det ( J g -1 i )∣ ∣ ∣ > 0 , since p x > 0 and ∏ j i =1 ∣ ∣ ∣ det ( J g -1 i )∣ ∣ ∣ = ∏ j i =1 | det ( J g i ) | -1 > 0 . Meanwhile ( ii ) holds since ∏ j i =1 | det ( J g i ) | > 0 and thus all of the singular values of ∏ j i =1 J g i are non-zero.

A.2 Experimental Setups

In this section, we elaborate on the experimental setups and provide the detailed configurations for the experiments presented in Section 5 of the main manuscript. The code implementation for the experiments is provided in the following repository: https://github.com/chen-hao-chao/ ebflow . Our code implementation is developed based on [7, 17, 44].

A.2.1 Experimental Setups for the Two-Dimensional Synthetic Datasets

Datasets. In Section 5.1, we present the experimental results on three two-dimensional synthetic datasets: Sine, Swirl, and Checkerboard. The Sine dataset is generated by sampling data points from the set { (4 w -2 , sin(12 w -6)) | w ∈ [0 , 1] } . The Swirl dataset is generated by sampling data points from the set { ( -π √ w cos( π √ w ) , π √ w sin( π √ w )) | w ∈ [0 , 1] } . The Checkerboard dataset is generated by sampling data points from the set { (4 w -2 , t -2 s + ⌊ 4 w -2 ⌋ mod2) | w ∈ [0 , 1] , t ∈ [0 , 1] , s ∈ { 0 , 1 }} , where ⌊·⌋ is a floor function, and mod represents the modulo operation.

To establish p x for all three datasets, we smooth a Dirac function using a Gaussian kernel. Specifically, we define the Dirac function as ˆ p (ˆ x ) ≜ 1 M ∑ M i =1 δ ( ∥ ∥ ˆ x -ˆ x ( i ) ∥ ∥ ) , where { ˆ x ( i ) } i M =1 are M uniformlysampled data points. The data distribution is defined as p x ( x ) ≜ ∫ ˆ p (ˆ x ) N ( x | ˆ x , ˆ σ 2 I ) d ˆ x = 1 M ∑ M i =1 N ( x | ˆ x ( i ) , ˆ σ 2 I ) . The closed-form expressions for p x ( x ) and ∂ ∂ x log p x ( x ) can be obtained using the derivation in [45]. In the experiments, M is set as 50 , 000 , and ˆ σ is fixed at 0 . 375 for all three datasets.

Implementation Details. The model architecture of g ( · ; θ ) consists of ten Glow blocks [4]. Each block comprises an actnorm [4] layer, a fully-connected layer, and an affine coupling layer. Table A2 provides the formal definitions of these operations. p u ( · ) is implemented as an isotropic Gaussian with zero mean and unit variance. To determine the best hyperparameters, we perform a grid search over the following optimizers, learning rates, and gradient clipping values based on the evaluation results in terms of the KL divergence. The optimizers include Adam [46], AdamW [47], and RMSProp. The learning rate and gradient clipping values are selected from (5e-3, 1e-3, 5e-4, 1e-4) and (None, 2.5, 10.0), respectively. Table A1 summarizes the selected hyperparameters. The optimization processes of Sine and Swirl datasets require 50,000 training iterations for convergence, while that of the Checkerboard dataset requires 100,000 iterations. The batch size is fixed at 5,000 for all setups.

A.2.2 Experimental Setups for the Real-world Datasets

Datasets. The experiments presented in Section 5.2 are performed on the MNIST [19] and CIFAR10 [37] datasets. The training and test sets of MNIST and CIFAR-10 contain 50,000 and 10,000 images, respectively. The data are smoothed using the uniform dequantization method presented in [1]. The observable parts (i.e., x O ) of the images in Fig. 5 are produced using the pre-trained model in [48].

Implementation Details. In Sections 5.2 and 5.4, we adopt three types of model architectures: FC-based [7], CNN-based, and Glow [4] models. The FC-based model contains two fully-connected layers and a smoothed leaky ReLU non-linearity [7] in between, which is identical to [7]. The CNNbased model consists of three convolutional blocks and two squeezing operations [2] between every convolutional block. Each convolutional block contains two convolutional layers and a smoothed leaky ReLU in between. The Glow model adopted in Section 5.4 is composed of 16 Glow blocks. Each of the Glow block consists of an actnorm [4] layer, a convolutional layer, and an affine coupling layer. The squeezing operation is inserted between every eight blocks. The operations used in these models are summarized in Table A2. The smoothness factor α of Smooth Leaky ReLU is set to 0.3 and 0.6 for models trained on MNIST and CIFAR-10, respectively. The scaling and transition functions s ( · ; θ ) and t ( · ; θ ) of the affine coupling layers are convolutional blocks with ReLU activation functions. The prior distribution p u ( · ) is implemented as an isotropic Gaussian with zero mean and unit variance. The FC-based and CNN-based models are trained with RMSProp using a learning rate initialized at 1e-4 and a batch size of 100. The Glow model is trained with an Adam optimizer using a learning rate initialized at 1e-4 and a batch size of 100. The gradient clipping value is set to 500 during the training for the Glow model. The learning rate scheduler MultiStepLR in PyTorch is used for gradually decreasing the learning rates. The hyper-parameters { σ, ξ } used in DSM and FDSSM are selected based on a grid search over { 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } . The selected { σ, ξ } are { 1 . 0 , 1 . 0 } and { 0 . 1 , 0 . 1 } for the MNIST and CIFAR-10 datasets, respectively. The parameter m in EMA is set to 0.999. The algorithms are implemented using PyTorch [39]. The gradients w.r.t. x and θ are both calculated using automatic differential tools [40] provided by PyTorch [39]. The runtime is evaluated on Tesla V100 NVIDIA GPUs. In the experiments performed on CIFAR-10 and CelebA using score-matching methods, the energy function (i.e., E p x ( x ) [ E ( x ; θ )] ) is added as a regularization loss with a balancing factor fixed at 0 . 001 during the optimization processes. The results in Fig. 2 (b) are smoothed with the exponential moving average function used in Tensorboard [49], i.e., w × d i -1 +(1 -w ) × d i , where w is set to 0.45 and d i represents the evaluation result at the i -th iteration.

Results of the Related Works. The results of the relative gradient [7], SSM [16], and FDSSM [17] methods are directly obtained from their original paper. On the other hand, the results of the DSM method is obtained from [17]. Please note that the reported results of [16] and [17] differ from each other given that they both adopt the NICE [1] model. Specifically, the SSM method achieves NLL = 3 , 355 and NLL = 6 , 234 in [16] and [17], respectively. Moreover, the DSM method achieves NLL = 4 , 363 and NLL = 3 , 398 in [16] and [17], respectively. In Table 4, we report the results with lower NLL.

A.3 Estimating the Jacobian Determinants using Importance Sampling

Importance sampling is a technique used to estimate integrals, which can be employed to approximate the normalizing constant Z ( θ ) in an energy-based model. In this method, a pdf q with a simple closed form that can be easily sampled from is selected. The normalizing constant can then be expressed as the following formula:

where { ˆ x ( j ) } j M =1 represents M i.i.d. samples drawn from q . According to Lemma A.11, the Jacobian determinants of the layers in S l can be approximated using Eq. (A5) as follows:

To validate this idea, we provide a simple simulation with p u = N ( 0 , I ) , q = N ( 0 , I ) , g ( x ; W ) = Wx , M = { 50 , 100 , 200 } , and D = { 50 , 100 , 200 } in Table A3. The results show that larger values of M lead to more accurate estimation of the Jacobian determinants. Typically, the choice of q is crucial to the accuracy of importance sampling. To obtain an accurate approximation, one can adopt the technique of annealed importance sampling (AIS) [33] or Reverse AIS Estimator (RAISE) [34], which are commonly-adopted algorithms for effectively estimating Z ( θ ) .

Eq. (A6) can be interpreted as a generalization of the stochastic estimator presented in [50], where the distributions p u and q are modeled as isotropic Gaussian distributions, and g is restricted as a linear transformation. For the further analysis of this concept, particularly in the context of determinant estimation for matrices, we refer readers to Section I of [50], where a more sophisticated approximation approach and the corresponding experimental findings are provided.

A.4 A Comparison among the Methods Discussed in this Paper

In Sections 2, 3, and 4, we discuss various methods for efficiently training flow-based models. To provide a comprehensive comparison of these methods, we summarize their complexity and characteristics in Table A4.

A.5 The Impacts of the Constraint of Linear Transformations on the Performance of a Flow-based Model

In this section, we examine the impact of the constraints of linear transformations on the performance of a flow-based model. A key distinction between constrained and unconstrained linear layers lies

Full Matrix (F)

Lower Triangular

Matrix (L)

Upper Triangular

Matrix (U)

Lower & Upper Triangular

Matrices (LU)

Learnable Weight

Masked Weight (Not Learnable)

-4

-2

0

2

4

-8

-6

-4

-2

0

8

6

4

2

-15

-10

0

-5

15

10

5

-30

-20

0

-10

30

20

10

in how they model the correlation between each element in a data vector. Constrained linear transformations, such as those used in the previous works [8-12, 29], impose predetermined correlations that are not learnable during the optimization process. For instance, masked linear layers [8-10] are constructed by masking either the upper or lower triangular weight matrix in a linear layer. In contrast, unconstrained linear layers have weight matrices that are fully learnable, making them more flexible than their constrained counterparts.

To demonstrate the influences of the constraint on the expressiveness of a model, we provide a performance comparison between flow-based models constructed using different types of linear layers. Specifically, we compare the performance of the models constructed using linear layers with full matrices, lower triangular matrices, upper triangular matrices, and matrices that are the multiplication of both lower and upper triangular matrices. These four types of linear layers are hereafter denoted as F, L, U, and LU, respectively, and the differences between them are depicted in Fig. A2. Furthermore, to highlight the performance discrepancy between these models, we construct the target distribution p x based on an autoregressive relationship of data vector x . Let x [ i ] denote the i -th element of x , and p x [ i ] represent its associated pdf. x [ i ] is constructed based on the following equation:

where u is sampled from an isotropic Gaussian, and s and d are coefficients controlling the shape and distance between each mode, respectively. In Eq. (A7), the function tanh( · ) can be intuitively viewed as a smoothed variant of the function 2 H ( · ) -1 , where H ( · ) represents the Heaviside step function. In this context, the values of ( x [ i -1] + d × 2 i ) are multiplied by a value close to either -1 or 1 , effectively transforming a positive number to a negative one. Fig. A3 depicts a number of examples of p x [ i ] constructed using this method. By employing this approach to design p x , where capturing p x [ i ] is presumed to be more challenging than modeling p x [ j ] for any j < i , we can inspect how the applied constraints impact performance. Inappropriately masking the linear layers, like the U-type layer, is anticipated to result in degraded performance, similar to the anti-casual effect explained in [51].

In this experiment, we constructed flow-based models using the smoothed leakyReLU activation and different types of linear layers (i.e., F, L, U, and LU) with a dimensionality of D = 10 . The models are optimized according to Eq. (2). The performance of these models is evaluated in terms of NLL, and its trends are depicted in Fig. A4. It is observed that the flow-based model built with the F-type layers achieved the lowest NLL, indicating the advantage of using unconstrained weight matrices in linear layers. In addition, there is a noticeable performance discrepancy between models with the L-type and U-type layers, indicating that imposing inappropriate constraints on linear layers may negatively affect the modeling abilities of flow-based mod-

Step

NLL

0

2,000

4,000

6,000

10,000

8,000

45

50

55

60

65

70

F

L

LU

U

els. Furthermore, even when both L-type and U-type layers were adopted, as shown in the red curve in Fig. A4, the performance remains inferior to those using the F-type layers. This experimental evidence suggests that linear layers constructed based on matrix decomposition (e.g., [4, 9]) may not possess the same expressiveness as unconstrained linear layers.

A.6 Limitations and Discussions

We noticed that score-matching methods sometimes exhibit difficulty in differentiating the weights between individual modes within a multi-modal distribution. This deficiency is illustrated in Fig. A5 (a), where EBFlow fails to accurately capture the density of the Checkerboard dataset. This phenomenon bears resemblance to the blindness problem discussed in [52]. While the solution proposed in [52] has the potential to address this issue, their approach is not directly applicable to the flow-based architectures employed in this paper.

FDSSM

SSM

DSM

(a)

(b)

epoch: 40

epoch: 80

(c)

$

In addition, we observed that the sampling quality of EBFlow occasionally experiences a significant reduction during the training iterations. This phenomenon is illustrated in Fig. A5 (b) and (c), where the Glow model trained using our approach demonstrates a decline in performance with extended training periods. The underlying cause of this phenomenon remains unclear, and we consider it a potential avenue for future investigation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider normalizing flows with linear transformations. Computing the determinant of their Jacobian is expensive, limiting their applicability when training via maximum likelihood. Therefore, the authors introduced an energy-based method to train normalizing flows with linear transformation, which circumvents the need of computing the determinant of the Jacobian. It only has to be computed once for inference, which leads to a significant speed-up in training.\n\nCompared to the maximum likelihood-based training, the authors achieve similar results while getting a significant speed-up.\n\nSTRENGTHS:\nThe idea of combining energy-based training with normalizing flows by phrasing the flow as an energy-based model is novel and interesting. It circumvents the need of having to compute the determinant of the Jacobian of some layers, which can save a lot of compute. It still needs to be computed for inference; however, since it is constant in the cases the authors considered, it has to be computed only once and can be stored for further use.\n\nWEAKNESSES:\nMy biggest concern is the relevance of this work. Normalizing flows typically use transformations that are designed in such a way that the Jacobian determinant is easy to compute. For that reason, linear transformations are not very popular, also because they are not very expressive. The claim of the authors, that they are used in Glow and subsequent flow models for images, is true, but the linear transformation was only applied across channels, and since the number of channels is much smaller than the total number of input dimensions, i.e. the number of all pixels across all channels, they are not very costly in terms of evaluating their Jacobian's determinant.\n\nThe authors apply their method to MNIST and CIFAR10, roughly matching the performance of their baseline with their method. However, both perform significantly worse than Glow and related methods. Hence, it is not clear whether their method gives them any advantage over these methods, either in terms of performance or speed.\n\nThere are also procedures to estimate the Jacobian's determinant to cut down cost, such as the Skilling-Hutchinson estimator used by residual flows. The authors should have compared their method with using such an estimator. Unfortunately, their method cannot be used for residual flows, as the determinant of the Jacobian cannot depend on the input to be treated as a normalization constant in an energy-based method.\n\nQUESTIONS:\n* How does your method compare against Glow in terms of runtime?\n* How does using an estimator for the Jacobian's determinant compare against your method?\n\nLIMITATIONS:\nLimitations were mentioned in the weaknesses section of the review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNormalizing flows are generative models trained by maximizing the log-likelihood of the dataset given the model, which is expressed as the sum of the base probability of a transformed variable plus a series of log-Jacobian terms that keep track of the volume distortion factors. Evaluating the log-Jacobian is usually the main bottleneck of normalizing flow architectures, so much so that the most popular architectures are designed so as to keep the log-Jacobian computations tractable. \n\nThis paper introduces the use of techniques from the energy-based models literature in order to more efficiently train normalizing flow architectures. In a normalizing flow architecture, it is common to have both elementwise non-linear layers, which give data dependent but tractable log-Jacobian terms, and linear layers whose log-Jacobian is data-independent but expensive to compute. The main idea introduced in this paper is to split the log-likelihood of a normalizing flow into the product of two terms: 1) an \"energy function\" that contains all the data-dependent terms and 2) a normalization constant that collects all the data-independent log-Jacobians of the linear layers.\n\nThis mirrors the split in energy-based models, where the energy function is tractable but the normalization function is intractable and it is not directly evaluated. Consequently, the authors apply a series of training objectives developed in order to train energy-based models to this new setting, obtaining an efficient way of training normalizing flows with expensive linear layers.\n\nSTRENGTHS:\nThe main idea is clever and original. It follows from the realization that the intractable terms of the log-Jacobian are independent from the data. This offers an interesting and well-motivated reason for borrowing techniques from the energy-based literature.\n\nThe experiment section is adequate, with experiments both on toy data and on real high-dimensional datasets. While the results are not spectacular, they certainly support the claim that the energy-based training can be used to train normalizing flows with a substantial speed-up and without a major loss of performance.\n\nThe paper is well-written and the relevant literature is properly referenced.\n\nWEAKNESSES:\nMajor points:\nIn general, I am not convinced that this approach solves a major problem in the current literature. In fact, most successful flow architectures are designed to have fast log-Jacobian evaluations. Moreover, both continuous flows and the more recent flow-matching models can be used to train architectures with intractable Jacobian terms. \n\nThe experiments only compare with the normalizing flow baseline and therefore they do not provide evidence that this approach has a competitive advantage over continuous flows and diffusion-like models such as flow-matching models. \n\nMinor points:\nI find the presentation somehow difficult to read, with several important equations and concepts scattered all over the paper. In particular, I think it would be useful to the reader to explicitly discuss the score-matching loss associated with Eq.9, for example, by writing the concrete form of Eq.5 as applied to Eq.9 in the main text.\n\nQUESTIONS:\n- Could you clarify in which situations this model should be preferred over continuous flows and score-based diffusion models?\n\nLIMITATIONS:\nThe limitations of the approach have not be extensively discussed. I do not see any direct negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The authors use a connection between flows and EBMs to devise a new training method for flows via score-matching.
Their training procedure improves efficiency via avoiding computation of the Jacobian determinants that are independant to values of the samples (i.e. only contribute to the normalizing constant of the PDF).
Additionally they identify two methods for improving training of the flows via score-matching, namely (1) match-after-preprossing whereby the score-matching occurs to the pre-processed variables avoiding numerical instability from the pre-processing layers, and (2) using an exponential moving average for the weight updates.
In their experiments, they show their approach significantly improves training efficiency.

STRENGTHS:
- The paper identifies that computing the unnormalized PDF of a flow is significantly cheaper than the normalized PDF for linear flow layers, O(D^2) instead of O(D^3), noting that computing the Jacobian determinant is not necessary for the unnormalized PDF. 
This contribution is novel and valuable. 
- The experiments of the paper are strong, and demonstrate the utility of their approach.

WEAKNESSES:
For sample generation by inverting the flow, the inverse matrix for the linear flow layers will cost $D^3$ - this is not mentioned in the paper. 
This inversion may be performed once, and then re-used for sample generation but for large $D$ this could be prohibitive.

QUESTIONS:
This is the first time I've seen imputation done with flows - is this a novel contribution from the paper?
How would sampling work for large $D$?

LIMITATIONS:
There is no discussion around inverting the flow when $D$ is large (see Weaknesses/Questions).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
This paper proposes a new flow based approach to generative modeling called EBFlow. EBFlow uses training objectives from EBMs to reduce training cost for flow based methods unlike previous approaches that mostly achieved this via different architectures or biased estimation methods. The main insight for EBFlow is that the change of variables formula for a series of transformations for a NF is broken in to two parts: one containing only linear flows term and the other containing terms from non-linear transformations. The linear flows term is then interpreted as the normalizing constant and this interpretation aids in faster training strategies for flow based methods.

STRENGTHS:
The motivation for reducing training costs for NFs is discussed well and is a very relevant and difficult problem. Several studies have been conducted to address this and the authors here propose a nice and novel methodology to address this.

The exploration and discussion of the method as well as the motivation is thorough. 

The empirical analysis considered is detailed.

WEAKNESSES:
I am not sure how to interpret the empirical results. It is obvious from the plots in Fig 2 that the proposed method is much faster than ML based objective. However, the performance in all the tasks for the ML based objective is usually better or very close to EBFlow. I was curious why this is? 

Also, for the results reported in Tables 1 and 2, were the methods trained for the same wall clock time?

QUESTIONS:
See above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has proposed an energy-based normalizing flow model, where the computaiton of Jacobin determinants for linear transformations can be skiped with score-matching objectives. This could enable deeper layers of linear transformation and make the flow model more efficient.\n\nSTRENGTHS:\nThis paper has proposed an interesting idea about the relationship between flow model and energy-based model. The flow model could be interpreted as an energy-based model with a tractable normalizaiton term and the proposed score-matching objective could significantly improve the efficiency.\n\nWEAKNESSES:\n1. The experiments lack comparison with other energy-based flow model like [1][2][3] and recent flow models like [4][5][6].\n2. Is there any theoretic analysis for the complexity between different objective function?\n3. The author misses citations for the very first deep-learning-based energy-based models like [7][8].\n\nWriting weakness:\n1. The background in section 2.2 could break up into two sections, i.e., energy-based model and score-matching model.\n2. It is hard to see the difference between results in the first two rows in Figure 1.\n3. In line 51, even the tranformation function in flow model is invertible, the author should specify that $g()$ is the inverse transformation function.\n4. In line 51, using $g_i(\\cdot;\\theta)$ is misleading. Does different transformation functions share the same parameters?\n5. I think the author intends to model $g(\\theta)$. Therefore, it is better to avoid using $E(x;\\theta)$ to avoid confusion as it is usually used for representing a parametric energy model.\n6. For Table 2, the author could change the unit to batch/second for easier comparison.\n\nReference:\n[1] Xie, Jianwen, et al. \"A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference.\" arXiv preprint arXiv:2301.09300 (2023).\n\n[2] Xie, Jianwen, et al. \"A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model.\" arXiv preprint arXiv:2205.06924 (2022).\n\n[3] Nijkamp, Erik, et al. \"Learning energy-based model with flow-based backbone by neural transport mcmc.\" arXiv preprint arXiv:2006.06897 2 (2020).\n\n[4] Chen, Ricky TQ, et al. \"Residual flows for invertible generative modeling.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[5] Maaløe, Lars, et al. \"Biva: A very deep hierarchy of latent variables for generative modeling.\" Advances in neural information processing systems 32 (2019).\n\n[6] Vahdat, Arash, Karsten Kreis, and Jan Kautz. \"Score-based generative modeling in latent space.\" Advances in Neural Information Processing Systems 34 (2021): 11287-11302.\n\n[7] Xie, Jianwen, et al. \"A theory of generative convnet.\" International Conference on Machine Learning. PMLR, 2016.\n\n[8] Xie, Jianwen, et al. \"Cooperative learning of energy-based model and latent variable model via mcmc teaching.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\nQUESTIONS:\n1. Could the author explain in detail why the complexity of calcuating Jacobin determinants is $O(D^3 \\cdot L)$ for non-linear transformation? Is it general in all kinds of non-linear functions?\n2. What does the sentence in line 62-63 mean?\n3. What is the FID score in generation tasks?\n4. What is the distribution of $p_{u}$?\n\nLIMITATIONS:\nThis paper has propose an interesting view of taking flow model as an eneryg-based model with score matching objectives. However, the expriments lack comparison of many baseline models as listed in 'weakness' part and other eval metrics in generation. Additional, the author should rephrase some sentences and formulars to avoid misleading.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}