SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Asymptotic Convergence Property. Similar to maximum likelihood training, score-matching methods that minimize Fisher divergence have theoretical guarantees on their consistency [14, 16]. This property is essential in ensuring the convergence accuracy of the parameters. Let  $N$  be the number of independent and identically distributed (i.i.d.) samples drawn from  $p_{\mathbf{x}}$  to approximate the expectation in the SM objective. In addition, assume that there exists a set of optimal parameters  $\theta^{*}$  such that  $p(\boldsymbol {x};\theta^{*}) = p_{\mathbf{x}}(\boldsymbol {x})$ . Under the regularity conditions (i.e., Assumptions A.1-A.7 shown in Appendix A.1.1), consistency guarantees that the parameters  $\theta_N$  minimizing the SM loss converges (in probability) to its optimal value  $\theta^{*}$  when  $N\to \infty$ , i.e.,  $\theta_N\xrightarrow{p}\theta^*$  as  $N\to \infty$ . In Appendix A.1.1, we provide a formal description of this property based on [16] and derive the sufficient condition for  $g$  and  $p_{\mathbf{u}}$  to satisfy the regularity conditions (i.e., Proposition A.10).
4.2 Techniques for Enhancing the Training of EBFlow
As revealed in the recent studies [16, 17], training flow-based models with score-matching objectives is challenging as the training process is numerically unstable and usually exhibits significant variances. To address these issues, we propose to adopt two techniques: match after preprocessing (MaP) and exponential moving average (EMA), which are particularly effective in dealing with the above issues according to our ablation analysis in Section 5.3.
MaP. Score-matching methods rely on the score function  $-\frac{\partial}{\partial x} E(\pmb{x};\theta)$  to match  $\frac{\partial}{\partial x}\log p_{\mathbf{x}}(\pmb{x})$ , which requires backward propagation through each layer in  $g$ . This indicates that the training process could be numerically sensitive to the derivatives of  $g$ . For instance, logit pre-processing layers commonly used in flow-based models (e.g., [1, 4, 5, 7, 8, 35]) exhibit extremely large derivatives near 0 and 1, which might exacerbate the above issue. To address this problem, we propose to
exclude the numerically sensitive layer(s) from the model and match the pdf of the pre-processed variable during training. Specifically, let  $\mathbf{x}_k \triangleq g_k \circ \dots \circ g_1(\mathbf{x})$  be the pre-processed variable, where  $k$  represents the index of the numerically sensitive layer. This method aims to optimize a parameterized pdf  $p_k(\cdot; \theta) \triangleq p_{\mathbf{u}}(g_L \circ \dots \circ g_{k+1}(\cdot; \theta)) \prod_{i=k+1}^{L} |\det(\mathbf{J}_{g_i})|$  that excludes  $(g_k, \dots, g_1)$  through minimizing the Fisher divergence between the pdf  $p_{\mathbf{x}_k}(\cdot)$  of  $\mathbf{x}_k$  and  $p_k(\cdot; \theta)$  by considering the (local) behavior of  $\mathbb{D}_{\mathrm{F}}$ , as presented in Proposition 4.1.
Proposition 4.1. Let  $p_{\mathbf{x}_j}$  be the pdf of the latent variable of  $\mathbf{x}_j \triangleq g_j \circ \dots \circ g_1(\mathbf{x})$  indexed by  $j$ . In addition, let  $p_j(\cdot)$  be a pdf modeled as  $p_{\mathbf{u}}(g_L \circ \dots \circ g_{j+1}(\cdot)) \prod_{i=j+1}^{L} |\operatorname{det}(\mathbf{J}_{g_i})|$ , where  $j \in \{0, \dots, L-1\}$ . It follows that:
$$
\mathbb {D} _ {\mathrm {F}} \left[ p _ {\mathbf {x} _ {j}} \| p _ {j} \right] = 0 \Leftrightarrow \mathbb {D} _ {\mathrm {F}} \left[ p _ {\mathbf {x}} \| p _ {0} \right] = 0, \forall j \in \{1, \dots , L - 1 \}. \tag {10}
$$
The derivation is presented in Appendix A.1.3. In Section 5.3, we validate the effectiveness of the MaP technique on the score-matching methods formulated in Eqs. (5)-(7) through an ablation analysis. Please note that MaP does not affect maximum likelihood training, since it always satisfies  $\mathbb{D}_{\mathrm{KL}}[p_{\mathbf{x}_j}||p_j] = \mathbb{D}_{\mathrm{KL}}[p_{\mathbf{x}}||p_0], \forall j \in \{1,\dots ,L - 1\}$  as revealed in Lemma A.12.
EMA. In addition to the MaP technique, we have also found that the exponential moving average (EMA) technique introduced in [36] is effective in improving the training stability. EMA enhances the stability through smoothly updating the parameters based on  $\tilde{\theta} \gets m\tilde{\theta} + (1 - m)\theta_{i}$  at each training iteration, where  $\tilde{\theta}$  is a set of shadow parameters [36],  $\theta_{i}$  is the model's parameters at iteration  $i$ , and  $m$  is the momentum parameter. In our experiments presented in Section 5, we adopt  $m = 0.999$  for both EBFlow and the baselines.
In the following experiments, we first compare the training efficiency of the baselines trained with  $\mathcal{L}_{\mathrm{ML}}$  and EBFlow trained with  $\mathcal{L}_{\mathrm{SML}}$ ,  $\mathcal{L}_{\mathrm{SSM}}$ ,  $\mathcal{L}_{\mathrm{FDSSM}}$ , and  $\mathcal{L}_{\mathrm{DSM}}$  to validate the effectiveness of the proposed method in Sections 5.1 and 5.2. Then, in Section 5.3, we provide an ablation analysis of the techniques introduced in Section 4.2, and a performance comparison between EBFlow and a number of related studies [7, 16, 17]. Finally, in Section 5.4, we discuss how EBFlow can be applied to generation tasks. Please note that the performance comparison with [8-12, 29] is omitted, since their methods only support specialized linear layers and are not applicable to the employed model architecture [7] that involves fully-connected layers. The differences between EBFlow, the baseline, and the related studies are summarized in Table A4 in the appendix. The sampling process involved in the calculation of  $\mathcal{L}_{\mathrm{SML}}$  is implemented by  $g^{-1}(\mathbf{u};\theta)$ , where  $\mathbf{u}\sim p_{\mathbf{u}}$ . The transformation  $g(\cdot ;\theta)$  for each task is designed such that  $S_{l}\neq \phi$  and  $S_{n}\neq \phi$ . For more details about the experimental setups, please refer to Appendix A.2.
5.1 Density Estimation on Two-Dimensional Synthetic Examples
In this experiment, we examine the performance of EBFlow and its baseline on three two-dimensional synthetic datasets. These data distributions are formed using Gaussian smoothing kernels to ensure  $p_{\mathbf{x}}(\pmb{x})$  is continuous and the true score function  $\frac{\partial}{\partial x} \log p_{\mathbf{x}}(\pmb{x})$  is well defined. The model  $g(\cdot ;\theta)$  is constructed using the Glow model architecture [4], which consists of actnorm layers, affine coupling layers, and fully-connected layers. The performance are evaluated in terms of the KL divergence and the Fisher divergence between  $p_{\mathbf{x}}(\pmb{x})$  and  $p(\pmb{x};\theta)$  using independent and identically distributed (i.i.d.) testing sample points.
Baseline (ML)
EBFlow (SSM)
Figure 1: The visualized density functions on the Sine, Swirl, and Checkerboard datasets. The column 'True' illustrates the visualization of the true density functions.
EBFlow (DSM)
EBFlow (FDSSM)
Table 1 and Fig. 1 demonstrate the results of the above setting. The results show that the performance of EBFlow trained with  $\mathcal{L}_{\mathrm{SSM}}$ ,  $\mathcal{L}_{\mathrm{FDSSM}}$ , and  $\mathcal{L}_{\mathrm{DSM}}$  in terms of KL divergence is on par with those trained using  $\mathcal{L}_{\mathrm{SML}}$  as well as the baselines trained using  $\mathcal{L}_{\mathrm{ML}}$ . These results validate the efficacy of training EBFlow with score matching.
Table 1: The evaluation results in terms of KL-divergence and Fisher-divergence of the flow-based models trained with  $\mathcal{L}_{\mathrm{ML}}$ ,  $\mathcal{L}_{\mathrm{SML}}$ ,  $\mathcal{L}_{\mathrm{SSM}}$ ,  $\mathcal{L}_{\mathrm{DSM}}$ , and  $\mathcal{L}_{\mathrm{FDSSM}}$  on the Sine, Swirl, and Checkerboard datasets. The results are reported as the mean and  $95\%$  confidence interval of three independent runs.
Dataset	Metric	Baseline (ML)	EBFlow (SML)	EBFlow (SSM)	EBFlow (DSM)	EBFlow (FDSSM)
Sine	Fisher Divergence (↓)	6.86 ± 0.73 e-1	6.65 ± 1.05 e-1	6.25 ± 0.84 e-1	6.66 ± 0.44 e-1	6.66 ± 1.33 e-1
KL Divergence (↓)	4.56 ± 0.00 e+0	4.56 ± 0.00 e+0	4.56 ± 0.01 e+0	4.57 ± 0.02 e+0	4.57 ± 0.01 e+0
Swirl	Fisher Divergence (↓)	1.42 ± 0.48 e+0	1.42 ± 0.53 e+0	1.35 ± 0.10 e+0	1.34 ± 0.06 e+0	1.37 ± 0.07 e+0
KL Divergence (↓)	4.21 ± 0.00 e+0	4.21 ± 0.01 e+0	4.25 ± 0.04 e+0	4.22 ± 0.02 e+0	4.25 ± 0.08 e+0
Checkerboard	Fisher Divergence (↓)	7.24 ± 11.50 e+1	1.23 ± 0.75 e+0	7.07 ± 1.93 e-1	7.03 ± 1.99 e-1	7.08 ± 1.62 e-1
KL Divergence (↓)	4.80 ± 0.02 e+0	4.81 ± 0.02 e+0	4.85 ± 0.05 e+0	4.82 ± 0.05 e+0	4.83 ± 0.03 e+0
Table 2: The evaluation results in terms of the performance (i.e., NLL and Bits/Dim) and the throughput (i.e., Batch/Sec.) of the FC-based and CNN-based models trained with the baseline and the proposed method on MNIST and CIFAR-10. Each result is reported in terms of the mean and  $95\%$  confidence interval of three independent runs after  $\theta$  is converged. The throughput is measured on NVIDIA Tesla V100 GPUs.
MNIST (D=784)
Model	FC-based	CNN-based
Num. Param.	1.230 M	0.027 M
Method	Baseline (ML)	EBFlow (SML)	EBFlow (SSM)	EBFlow (DSM)	EBFlow (FDSSM)	Baseline (ML)	EBFlow (SML)	EBFlow (SSM)	EBFlow (DSM)
NLL (↓)	1092.4 ± 0.1	1092.3 ± 0.6	1092.8 ± 0.3	1099.2 ± 0.2	1104.1 ± 0.5	1101.3 ± 1.3	1098.3 ± 6.6	1107.5 ± 1.4	1109.5 ± 2.4
Bits/Dim (↓)	2.01 ± 0.00	2.01 ± 0.00	2.01 ± 0.00	2.02 ± 0.00	2.03 ± 0.00	2.03 ± 0.00	2.02 ± 0.01	2.03 ± 0.00	2.04 ± 0.00
Batch/Sec. (↑)	8.00	12.27	33.11	66.67	130.21	0.21	0.29	7.09	18.32
CIFAR-10 (D=3,072)
Model	FC-based	CNN-based
Num. Param.	18.881 M	0.241 M
Method	Baseline (ML)	EBFlow (SML)	EBFlow (SSM)	EBFlow (DSM)	EBFlow (FDSSM)	Baseline (ML)	EBFlow (SML)	EBFlow (SSM)	EBFlow (DSM)
NLL (↓)	11912.9 ± 10.5	11915.6 ± 5.6	11917.7 ± 15.5	11940.0 ± 6.6	12347.8 ± 6.8	11408.7 ± 26.7	11553.6 ± 151.7	11435.5 ± 12.0	11462.3 ± 7.9
Bits/Dim (↓)	5.59 ± 0.00	5.60 ± 0.00	5.60 ± 0.01	5.61 ± 0.00	5.80 ± 0.00	5.36 ± 0.01	5.41 ± 0.07	5.37 ± 0.00	5.38 ± 0.00
Batch/Sec. (↑)	5.05	7.35	29.85	57.14	62.50	0.02	0.03	7.35	18.41
5.2 Efficiency Evaluation on the MNIST and CIFAR-10 Datasets
In this section, we inspect the influence of data dimension  $D$  on the training efficiency of flow-based models. To provide a thorough comparison, we employ two types of model architectures and train them on two datasets with different data dimensions: the MNIST [19] ( $D = 1 \times 28 \times 28$ ) and CIFAR-10 [37] ( $D = 3 \times 32 \times 32$ ) datasets.
The first model architecture is exactly the same as that adopted by [7]. It is an architecture consisting of two fully-connected layers and a smoothed leaky ReLU non-linear layer in between. The second model is a parametrically efficient variant of the first model. It replaces the fully-connected layers with convolutional layers and increases the depth of the model to six convolutional blocks. Between every two convolutional blocks, a squeeze operation [2] is inserted to enlarge the receptive field. In the following paragraphs, we refer to these models as 'FC-based' and 'CNN-based' models, respectively.
The performance of the FC-based and CNN-based models is measured using the negative log likelihood (NLL) metric (i.e.,  $\mathbb{E}_{p_{\mathbf{x}}(\boldsymbol{x})}[-\log p(\boldsymbol{x};\theta)]$ ), which differs from the intractable KL divergence by a constant. In addition, its normalized variant, the Bits/Dim metric [38], is also measured and reported. The algorithms are implemented using PyTorch [39] with automatic differentiation [40], and the runtime is measured on NVIDIA Tesla V100 GPUs. In the subsequent paragraphs, we assess the models through scalability analysis, performance evaluation, and training efficiency examination.
Scalability. To demonstrate the scalability of KL-divergence-based (i.e.,  $\mathcal{L}_{\mathrm{ML}}$  and  $\mathcal{L}_{\mathrm{SML}}$ ) and Fisher-divergence-based (i.e.,  $\mathcal{L}_{\mathrm{SSM}}$ ,  $\mathcal{L}_{\mathrm{DSM}}$ , and  $\mathcal{L}_{\mathrm{FDSSM}}$ ) objectives used in EBFlow and the baseline method, we first present a runtime comparison for different choices of the input data size  $D$ . The results presented in Fig. 2 (a) reveal that Fisher-divergence-based objectives can be computed more efficiently than KL-divergence-based objectives. Moreover, the sampling-based objective  $\mathcal{L}_{\mathrm{SML}}$  used in EBFlow, which excludes the calculation of  $Z(\theta)$  in the computational graph, can be computed slightly faster than  $\mathcal{L}_{\mathrm{ML}}$  adopted by the baseline.
Figure 2: (a) A runtime comparison of calculating the gradients of different objectives for different input sizes  $(D)$ . The input sizes are  $(1, n, n)$  and  $(3, n, n)$ , with the x-axis in the figures representing  $n$ . In the format  $(c, h, w)$ , the first value indicates the number of channels, while the remaining values correspond to the height and width of the input data. The curves depict the evaluation results in terms of the mean of three independent runs. (b) A comparison of the training efficiency of the FC-based and CNN-based models evaluated on the validation set of MNIST and CIFAR-10. Each curve and the corresponding shaded area depict the mean and confidence interval of three independent runs.
Figure 3: The norm of  $\frac{\partial}{\partial\theta}\mathcal{L}_{\mathrm{SSM}}(\theta)$  of an FC-based model trained on the MNIST dataset. The curves and shaded area depict the mean and  $95\%$  confidence interval of three independent runs.
Table 3: The results in terms of NLL of the FC-based and CNN-based models trained using SSM, DSM, and FDSSM losses on MNIST. The performance is reported in terms of the means and  $95\%$  confidence intervals of three independent runs.
FC-based
EMA	MaP	EBFlow(SSM)	EBFlow(DSM)	EBFlow(FDSSM)
1757.5 ± 28.0	4660.3 ± 19.8	3267.0 ± 99.2
✓	1720.5 ± 0.8	4455.0 ± 1.6	3166.3 ± 17.3
✓	✓	1092.8 ± 0.3	1099.2 ± 0.2	1104.1 ± 0.5
CNN-based
EMA	MaP	EBFlow(SSM)	EBFlow(DSM)	EBFlow(FDSSM)
3518.0 ± 33.9	3170.0 ± 7.2	3593.3 ± 12.5
✓	3504.5 ± 2.4	3180.0 ± 2.9	3560.3 ± 1.7
✓	✓	1107.5 ± 1.4	1109.5 ± 2.6	1122.1 ± 3.1
Performance. Table 2 demonstrates the performance of the FC-based and CNN-based models in terms of NLL on the MNIST and CIFAR-10 datasets. The results show that the models trained with Fisher-divergence-based objectives are able to achieve similar performance as those trained with KL-divergence-based objectives. Among the Fisher-divergence-based objectives, the models trained using  $\mathcal{L}_{\mathrm{SSM}}$  and  $\mathcal{L}_{\mathrm{DSM}}$  are able to achieve better performance in comparison to those trained using  $\mathcal{L}_{\mathrm{FDSSM}}$ . The runtime and performance comparisons above suggest that  $\mathcal{L}_{\mathrm{SSM}}$  and  $\mathcal{L}_{\mathrm{DSM}}$  can deliver better training efficiency than  $\mathcal{L}_{\mathrm{ML}}$  and  $\mathcal{L}_{\mathrm{SML}}$ , since the objectives can be calculated faster while maintaining the models' performance on the NLL metric.
Training Efficiency. Fig. 2 (b) presents the trends of NLL versus training wall time when  $\mathcal{L}_{\mathrm{ML}}$ ,  $\mathcal{L}_{\mathrm{SML}}$ ,  $\mathcal{L}_{\mathrm{SSM}}$ ,  $\mathcal{L}_{\mathrm{DSM}}$ , and  $\mathcal{L}_{\mathrm{FDSSM}}$  are adopted as the objectives. It is observed that EBFlow trained with SSM and DSM consistently attain better NLL in the early stages of the training. The improvement is especially notable when both  $D$  and  $L$  are large, as revealed for the scenario of training CNN-based models on the CIFAR-10 dataset. These experimental results provide evidence to support the use of score-matching methods for optimizing EBFlow.
5.3 Analyses and Comparisons
Ablation Study. Table 3 presents the ablation results that demonstrate the effectiveness of the EMA and MaP techniques. It is observed that EMA is effective in reducing the variances. In addition, MaP significantly improves the overall performance. To further illustrate the influence of the proposed MaP technique on the score-matching methods, we compare the optimization pro
Figure 4: A qualitative comparison between (a) our model  $(\mathrm{NLL} = 728)$  and (b) the model in [17]  $(\mathrm{NLL} = 1,637)$  on the inverse generation task.
Mask: MNIST
Figure 5: A qualitative demonstration of the FC-based model trained using  $\mathcal{L}_{\mathrm{DSM}}$  on the data imputation task.
Mask: KMNIST
cesses with  $\frac{\partial}{\partial\theta}\mathbb{D}_{\mathrm{F}}[p_{\mathbf{x}_k}\| p_k]$  and  $\frac{\partial}{\partial\theta}\mathbb{D}_{\mathrm{F}}[p_{\mathbf{x}}\| p_0] = \frac{\partial}{\partial\theta}\mathbb{E}_{p_{\mathbf{x}_k}(\mathbf{x}_k)}[\frac{1}{2} ](\frac{\partial}{\partial x_k}\log (\frac{p_{\mathbf{x}_k}(\mathbf{x}_k)}{p_k(\mathbf{x}_k)}))\prod_{i = 1}^k\mathbf{J}_{g_i}\| ^2 ]$  (i.e., Lemma A.13) by depicting the norm of their unbiased estimators  $\frac{\partial}{\partial\theta}\mathcal{L}_{\mathrm{SSM}}(\theta)$  calculated with and without applying the MaP technique in Fig. 3. It is observed that the magnitude of  $\left\| \frac{\partial}{\partial\theta}\mathcal{L}_{\mathrm{SSM}}(\theta)\right\|$  significantly decreases when MaP is incorporated into the training process. This could be attributed to the fact that the calculation of  $\frac{\partial}{\partial\theta}\mathbb{D}_{\mathrm{F}}[p_{\mathbf{x}_k}\| p_k]$  excludes the calculation of  $\prod_{i = 1}^{k}\mathbf{J}_{g_i}$  in  $\frac{\partial}{\partial\theta}\mathbb{D}_{\mathrm{F}}[p_{\mathbf{x}}\| p_0]$ , which involves computing the derivatives of the numerically sensitive logit pre-processing layer.
Comparison with Related Works. Table 4 compares the performance of our method with a number of related works on the MNIST dataset. Our models trained with score-matching objectives using the same model architecture exhibit improved performance in comparison to the relative gradient method [7]. In addition, when compared to the results in [16] and [17], our models deliver significantly improved performance over them. Please note that the results of [7, 16, 17] presented in Table 4 are obtained from their original papers.
Table 4: A comparison of performance and training complexity between EBFlow and a number of related works [16, 7, 17] on the MNIST dataset.
Method	Complexity	NLL (↓)
DKL-Based	Baseline (ML)	O(D3L)	1092.4 ± 0.1
EBFlow (SML)	O(D3L)	1092.3 ± 0.6
Relative Grad. [7]	O(D2L)	1375.2 ± 1.4
DF-Based	EBFlow (SSM)	O(D2L)	1092.8 ± 0.3
EBFlow (DSM)	O(D2L)	1099.2 ± 0.2
EBFlow (FDSSM)	O(D2L)	1104.1 ± 0.5
SSM [16]	-	3355
DSM [17]	-	3398 ± 1343
FDSSM [17]	-	1647 ± 306
5.4 Application to Generation Tasks
The sampling process of EBFlow can be accomplished through the inverse function or an MCMC process. The former is a typical generation method adopted by flow-based models, while the latter is a more flexible sampling process that allows conditional generation without re-training the model. In the following paragraphs, we provide detailed explanations and visualized results of these tasks.
Inverse Generation. One benefit of flow-based models is that  $g^{-1}$  can be directly adopted as a generator. While inverting the weight matrices in linear transformations typically demands time complexity of  $\mathcal{O}(D^3 L)$ , these inverse matrices are only required to be computed once  $\theta$  has converged, and can then be reused for subsequent inferences. In this experiment, we adopt the Glow [4] model architecture and train it using our method with  $\mathcal{L}_{\mathrm{SSM}}$  on the MNIST dataset. We compare our visualized results with the current best flow-based model trained using the score matching objective [17]. The results of [17] are generated using their officially released code with their best setup (i.e., FDSSM). As presented in Fig. 4, the results generated using our model demonstrate significantly better visual quality than those of [17].
MCMC Generation. In comparison to the inverse generation method, the MCMC sampling process is more suitable for conditional generation tasks such as data imputation due to its flexibility [41]. For the imputation task, a data vector  $\pmb{x}$  is separated as an observable part  $\pmb{x}_O$  and a masked part  $\pmb{x}_M$ . The goal of imputation is to generate the masked part  $\pmb{x}_M$  based on the observable part  $\pmb{x}_O$ . To achieve this goal, one can perform a Langevin MCMC process to update  $\pmb{x}_M$  according to the gradient of the energy function  $\frac{\partial}{\partial x} E(x;\theta)$ . Given a noise vector  $\pmb{z}$  sampled from  $\mathcal{N}(\mathbf{0},\mathbf{I})$  and a
small step size  $\alpha$ , the process iteratively updates  $\mathbf{x}_M$  based on the following equation:
$$
\boldsymbol {x} _ {M} ^ {(t + 1)} = \boldsymbol {x} _ {M} ^ {(t)} - \alpha \frac {\partial}{\partial \boldsymbol {x} _ {M} ^ {(t)}} E \left(\boldsymbol {x} _ {O}, \boldsymbol {x} _ {M} ^ {(t)}; \theta\right) + \sqrt {2 \alpha} \boldsymbol {z}, \tag {11}
$$
where  $\boldsymbol{x}_M^{(t)}$  represents  $\boldsymbol{x}_M$  at iteration  $t \in \{1, \dots, T\}$ , and  $T$  is the total number of iterations. MCMC generation requires an overall cost of  $\mathcal{O}(TD^2 L)$ , potentially more economical than the  $\mathcal{O}(D^3 L)$  computation of the inverse generation method. Fig. 5 depicts the imputation results of the FC-based model trained using  $\mathcal{L}_{\mathrm{DSM}}$  on the CelebA [42] dataset  $(D = 3 \times 64 \times 64)$ . In this example, we implement the masking part  $\boldsymbol{x}_M$  using the data from the KMNIST [43] and MNIST [19] datasets.
In this paper, we presented EBFlow, a new flow-based modeling approach that associates the parameterization of flow-based and energy-based models. We showed that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be bypassed, resulting in an improved training time complexity. In addition, we demonstrated that the training stability and performance can be effectively enhanced through the MaP and EMA techniques. Based on the improvements in both theoretical time complexity and empirical performance, our method exhibits superior training efficiency compared to maximum likelihood training.
The authors gratefully acknowledge the support from the National Science and Technology Council (NSTC) in Taiwan under grant number MOST 111-2223-E-007-004-MY3, as well as the financial support from MediaTek Inc., Taiwan. The authors would also like to express their appreciation for the donation of the GPUs from NVIDIA Corporation and NVIDIA AI Technology Center (NVAITC) used in this work. Furthermore, the authors extend their gratitude to the National Center for High-Performance Computing (NCHC) for providing the necessary computational and storage resources.
[1] L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear Independent Components Estimation, 2015.
[2] L. Dinh, J. N. Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. In Proc. Int. Conf. on Learning Representations (ICLR), 2016.
[3] G. Papamakarios, I. Murray, and T. Pavlakou. Masked Autoregressive Flow for Density Estimation. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2017.
[4] D. P. Kingma and P. Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2018.
[5] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural Spline Flows. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2019.
[6] A. Hyvarinen and E. Oja. Independent Component Analysis: Algorithms and Applications. Neural Networks: the Official Journal of the International Neural Network Society, 13 4-5:411-30, 2000.
[7] L. Gresele, G. Fissore, A. Javaloy, B. Scholkopf, and A. Hyvarinen. Relative Gradient Optimization of the Jacobian Term in Unsupervised Deep Learning. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2020.
[8] Y. Song, C. Meng, and S. Ermon. MintNet: Building Invertible Neural Networks with Masked Convolutions. In Proc. Conf. on Neural Information Processing Systems (NeurIPS), 2019.
[9] E. Hoogeboom, R. v. d. Berg, and M. Welling. Emerging Convolutions for Generative Normalizing Flows. In Proc. Int. Conf. on Machine Learning (ICML), 2019.