SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation Details. In Sections 5.2 and 5.4, we adopt three types of model architectures: FC-based [7], CNN-based, and Glow [4] models. The FC-based model contains two fully-connected layers and a smoothed leaky ReLU non-linearity [7] in between, which is identical to [7]. The CNN-based model consists of three convolutional blocks and two squeezing operations [2] between every convolutional block. Each convolutional block contains two convolutional layers and a smoothed leaky ReLU in between. The Glow model adopted in Section 5.4 is composed of 16 Glow blocks. Each of the Glow block consists of an actnorm [4] layer, a convolutional layer, and an affine coupling layer. The squeezing operation is inserted between every eight blocks. The operations used in these models are summarized in Table A2. The smoothness factor  $\alpha$  of Smooth Leaky ReLU is set to 0.3 and 0.6 for models trained on MNIST and CIFAR-10, respectively. The scaling and transition functions  $s(\cdot ;\theta)$  and  $t(\cdot ;\theta)$  of the affine coupling layers are convolutional blocks with ReLU activation functions. The prior distribution  $p_{\mathbf{u}}(\cdot)$  is implemented as an isotropic Gaussian with zero mean and unit variance. The FC-based and CNN-based models are trained with RMSProp using a learning rate initialized at 1e-4 and a batch size of 100. The Glow model is trained with an Adam optimizer using a learning rate initialized at 1e-4 and a batch size of 100. The gradient clipping value is set to 500 during the training for the Glow model. The learning rate scheduler MultiStepLR in PyTorch is used for gradually decreasing the learning rates. The hyper-parameters  $\{\sigma ,\xi \}$  used in DSM and FDSSM are selected based on a grid search over  $\{0.05,0.1,0.5,1.0\}$ . The selected  $\{\sigma ,\xi \}$  are  $\{1.0,1.0\}$  and  $\{0.1,0.1\}$  for the MNIST and CIFAR-10 datasets, respectively. The parameter  $m$  in EMA is set to 0.999. The algorithms are implemented using PyTorch [39]. The gradients w.r.t.  $\pmb{x}$  and  $\theta$  are both calculated using automatic differential tools [40] provided by PyTorch [39]. The runtime is evaluated on Tesla V100 NVIDIA GPUs. In the experiments performed on CIFAR-10 and CelebA using score-matching methods, the energy function (i.e.,  $\mathbb{E}_{p_{\mathbf{x}}(\boldsymbol {x})}[E(\boldsymbol {x};\theta)])$  is added as a regularization loss with a balancing factor fixed at 0.001 during the optimization processes. The results in Fig. 2 (b) are smoothed with the exponential moving average function used in Tensorboard [49], i.e.,  $w\times d_{i - 1} + (1 - w)\times d_i$ , where  $w$  is set to 0.45 and  $d_{i}$  represents the evaluation result at the  $i$ -th iteration.
Table A1: The hyper-parameters used in the two-dimensional synthetic example in Section 5.1.
Dataset	ML	SML	SSM	DSM	FDSSM
Sine	Optimizer	Adam	AdamW	Adam	Adam	Adam
Learning Rate	5e-4	5e-4	1e-4	1e-4	1e-4
Gradient Clip	1.0	None	1.0	1.0	1.0
Swirl	Optimizer	Adam	Adam	Adam	Adam	Adam
Learning Rate	5e-3	1e-4	1e-4	1e-4	1e-4
Gradient Clip	None	10.0	10.0	10.0	2.5
Checkerboard	Optimizer	AdamW	AdamW	AdamW	AdamW	Adam
Learning Rate	1e-4	1e-4	1e-4	1e-4	1e-4
Gradient Clip	10.0	10.0	10.0	10.0	10.0
Table A2: The components of  $g(\cdot ;\theta)$  used in this paper. In this table,  $z$  and  $y$  are the output and the input of a layer, respectively.  $\beta$  and  $\gamma$  represent the mean and variance of an actnorm layer.  $w$  is a convolutional kernel, and  $w \star y \triangleq \hat{W} y$ , where  $\star$  is a convolutional operator, and  $\hat{W}$  is a  $D \times D$  matrix.  $W$  and  $b$  represent the weight and bias in a fully-connected layer.  $\alpha$  is a hyper-parameter for adjusting the smoothness of smooth leaky ReLU. In the affine coupling layer,  $z$  and  $y$  are split into two parts  $\{z_{a},z_{b}\}$  and  $\{y_{a},y_{b}\}$ , respectively.  $s(\cdot ;\theta)$  and  $t(\cdot ;\theta)$  are the scaling and transition networks parameterized with  $\theta$ .  $\operatorname{sig}(y) = 1 / (1 + \exp(-y))$  represents the sigmoid function.  $\dim(\cdot)$  represents the dimension of the input vector.  $y_{[i]}$  represents the  $i$ -th element of vector  $y$ .
Layer	Function	Log Jacobian Determinant	Set
actnorm [4]	z=(y-β)/γ	∑i=1D log |1/γ[i]|	Sl
convolutional	z=w* y+b	log |det(W)|	S1
fully-connected	z=Wy+b	log |det(W)|	S1
smooth leakyReLU [7]	z=αy+(1-α) log(1+exp(y))	∑i=1D log |α+(1-α)sig(y[i])|	Sn
affine coupling [4]	za=s(yb;θ)yta+t(yb;θ), zb=yb	∑i=1dim(yb) log |s(yb;θ)[i]|	Sn
Table A3: The simulation results of Eq. (A6). The error rate is measured by  $|d_{\mathrm{true}} - d_{\mathrm{est}}| / |d_{\mathrm{true}}|$ , where  $d_{\mathrm{true}}$  and  $d_{\mathrm{est}}$  represent the true and estimated Jacobian determinants, respectively.
D=50	D=100	D=200
Error Rate (M=50)	0.004211	0.099940	0.355314
Error Rate (M=100)	0.003503	0.034608	0.076239
Error Rate (M=200)	0.002332	0.015411	0.011175
Results of the Related Works. The results of the relative gradient [7], SSM [16], and FDSSM [17] methods are directly obtained from their original paper. On the other hand, the results of the DSM method is obtained from [17]. Please note that the reported results of [16] and [17] differ from each other given that they both adopt the NICE [1] model. Specifically, the SSM method achieves  $\mathrm{NLL} = 3,355$  and  $\mathrm{NLL} = 6,234$  in [16] and [17], respectively. Moreover, the DSM method achieves  $\mathrm{NLL} = 4,363$  and  $\mathrm{NLL} = 3,398$  in [16] and [17], respectively. In Table 4, we report the results with lower NLL.
A.3 Estimating the Jacobian Determinants using Importance Sampling
Importance sampling is a technique used to estimate integrals, which can be employed to approximate the normalizing constant  $Z(\theta)$  in an energy-based model. In this method, a pdf  $q$  with a simple closed form that can be easily sampled from is selected. The normalizing constant can then be expressed as the following formula:
$$
\begin{array}{l} Z (\theta) = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \exp (- E (\boldsymbol {x}; \theta)) d \boldsymbol {x} = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} q (\boldsymbol {x}) \frac {\exp (- E (\boldsymbol {x} ; \theta))}{q (\boldsymbol {x})} d \boldsymbol {x} \\ = \mathbb {E} _ {q (\boldsymbol {x})} \left[ \frac {\exp (- E (\boldsymbol {x} ; \theta))}{q (\boldsymbol {x})} \right] \approx \frac {1}{M} \sum_ {j = 1} ^ {M} \frac {\exp (- E \left(\hat {\boldsymbol {x}} ^ {(j)} ; \theta\right))}{q \left(\hat {\boldsymbol {x}} ^ {(j)}\right)}, \tag {A5} \\ \end{array}
$$
where  $\{\hat{\pmb{x}}^{(j)}\}_{j = 1}^{M}$  represents  $M$  i.i.d. samples drawn from  $q$ . According to Lemma A.11, the Jacobian determinants of the layers in  $S_{l}$  can be approximated using Eq. (A5) as follows:
$$
\left(\prod_ {g _ {i} \in \mathcal {S} _ {l}} | \det  \left(\mathbf {J} _ {g _ {i}} (\theta)\right) |\right) ^ {- 1} \approx \frac {1}{M} \sum_ {j = 1} ^ {M} \frac {p _ {\mathbf {u}} \left(g \left(\hat {\boldsymbol {x}} ^ {(j)} ; \theta\right)\right) \prod_ {g _ {i} \in \mathcal {S} _ {n}} \left| \det  \left(\mathbf {J} _ {g _ {i}} \left(\hat {\boldsymbol {x}} _ {i - 1} ^ {(j)} ; \theta\right)\right) \right|}{q \left(\hat {\boldsymbol {x}} ^ {(j)}\right)}. \tag {A6}
$$
Table A4: An overall comparison between EBFlow, the baseline method, the Relative Gradient method [7], and the methods that utilize specially designed linear layers [8-12, 29]. The notations  $\sqrt{} / X$  in row 'Unbiased' represent whether the models are optimized according to an unbiased target. On the other hand, the notations  $\sqrt{} / X$  in row 'Unconstrained' represent whether the models can be constructed with arbitrary linear transformations. (†) The approximation errors  $o(\xi)$  of FDSSM is controlled by its hyper-parameter  $\xi$ . (‡) The error  $o(W)$  of the Relative Gradient method is determined by the values of a model's weights.
KL-Divergence-Based	Fisher-Divergence-Based
Baseline (ML)	EBFlow (SML)	Relative Grad.	Special Linear	EBFlow (SSM)	EBFlow (DSM)	EBFlow (FDSSM)
Complexity	O(D3L)	O(D3L)	O(D2L)	O(D2L)	O(D2L)	O(D2L)	O(D2L)
Unbiased	✓	✓	x(†)	✓	✓	x	x(†)
Unconstrained	✓	✓	✓	x	✓	✓	✓
To validate this idea, we provide a simple simulation with  $p_{\mathbf{u}} = \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $q = \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $g(\boldsymbol{x};\boldsymbol{W}) = \boldsymbol{W}\boldsymbol{x}$ ,  $M = \{50,100,200\}$ , and  $D = \{50,100,200\}$  in Table A3. The results show that larger values of  $M$  lead to more accurate estimation of the Jacobian determinants. Typically, the choice of  $q$  is crucial to the accuracy of importance sampling. To obtain an accurate approximation, one can adopt the technique of annealed importance sampling (AIS) [33] or Reverse AIS Estimator (RAISE) [34], which are commonly-adopted algorithms for effectively estimating  $Z(\theta)$ .
Eq. (A6) can be interpreted as a generalization of the stochastic estimator presented in [50], where the distributions  $p_{\mathbf{u}}$  and  $q$  are modeled as isotropic Gaussian distributions, and  $g$  is restricted as a linear transformation. For the further analysis of this concept, particularly in the context of determinant estimation for matrices, we refer readers to Section I of [50], where a more sophisticated approximation approach and the corresponding experimental findings are provided.
A.4 A Comparison among the Methods Discussed in this Paper
In Sections 2, 3, and 4, we discuss various methods for efficiently training flow-based models. To provide a comprehensive comparison of these methods, we summarize their complexity and characteristics in Table A4.
A.5 The Impacts of the Constraint of Linear Transformations on the Performance of a Flow-based Model
In this section, we examine the impact of the constraints of linear transformations on the performance of a flow-based model. A key distinction between constrained and unconstrained linear layers lies
Figure A2: An illustration of the weight matrices in the F, L, U, and LU layers described in Section A.5.
Figure A3: Visualized marginal distributions of  $p_{\mathbf{x}[i]}$  for  $i = 1,2,3$ , and 4.
in how they model the correlation between each element in a data vector. Constrained linear transformations, such as those used in the previous works [8-12, 29], impose predetermined correlations that are not learnable during the optimization process. For instance, masked linear layers [8-10] are constructed by masking either the upper or lower triangular weight matrix in a linear layer. In contrast, unconstrained linear layers have weight matrices that are fully learnable, making them more flexible than their constrained counterparts.
To demonstrate the influences of the constraint on the expressiveness of a model, we provide a performance comparison between flow-based models constructed using different types of linear layers. Specifically, we compare the performance of the models constructed using linear layers with full matrices, lower triangular matrices, upper triangular matrices, and matrices that are the multiplication of both lower and upper triangular matrices. These four types of linear layers are hereafter denoted as F, L, U, and LU, respectively, and the differences between them are depicted in Fig. A2. Furthermore, to highlight the performance discrepancy between these models, we construct the target distribution  $p_{\mathbf{x}}$  based on an autoregressive relationship of data vector  $\mathbf{x}$ . Let  $\mathbf{x}_{[i]}$  denote the  $i$ -th element of  $\mathbf{x}$ , and  $p_{\mathbf{x}_{[i]}}$  represent its associated pdf.  $\mathbf{x}_{[i]}$  is constructed based on the following equation:
$$
\mathbf {x} _ {[ i ]} = \left\{ \begin{array}{l l} \mathbf {u} _ {[ 0 ]} & \text {i f} i = 1, \\ \tanh  \left(\mathbf {u} _ {[ i ]} \times s\right) \times \left(\mathbf {x} _ {[ i - 1 ]} + d \times 2 ^ {i}\right), & \text {i f} i \in \{2, \dots , D \}, \end{array} \right. \tag {A7}
$$
where  $\mathbf{u}$  is sampled from an isotropic Gaussian, and  $s$  and  $d$  are coefficients controlling the shape and distance between each mode, respectively. In Eq. (A7), the function  $\tanh (\cdot)$  can be intuitively viewed as a smoothed variant of the function  $2H(\cdot) - 1$ , where  $H(\cdot)$  represents the Heaviside step function. In this context, the values of  $(\mathbf{x}_{[i-1]} + d \times 2^i)$  are multiplied by a value close to either  $-1$  or  $1$ , effectively transforming a positive number to a negative one. Fig. A3 depicts a number of examples of  $p_{\mathbf{x}_{[i]}}$  constructed using this method. By employing this approach to design  $p_{\mathbf{x}}$ , where capturing  $p_{\mathbf{x}_{[i]}}$  is presumed to be more challenging than modeling  $p_{\mathbf{x}_{[j]}}$  for any  $j < i$ , we can inspect how the applied constraints impact performance. Inappropriately masking the linear layers, like the U-type layer, is anticipated to result in degraded performance, similar to the anti-casual effect explained in [51].
In this experiment, we constructed flow-based models using the smoothed leakyReLU activation and different types of linear layers (i.e., F, L, U, and LU) with a dimensionality of  $D = 10$ . The models are optimized according to Eq. (2). The performance of these models is evaluated in terms of NLL, and its trends are depicted in Fig. A4. It is observed that the flow-based model built with the F-type layers achieved the lowest NLL, indicating the advantage of using unconstrained weight matrices in linear layers. In addition, there is a noticeable performance discrepancy between models with the L-type and U-type layers, indicating that imposing inappropriate constraints on linear layers may negatively affect the modeling abilities of flow-based mod
Figure A4: The evaluation curves in terms of NLL of the flow-based models constructed with the F-type, L-type, U-type, and LU-type layers. The curves and shaded area depict the mean and  $95\%$  confidence interval of three independent runs.
els. Furthermore, even when both L-type and U-type layers were adopted, as shown in the red curve in Fig. A4, the performance remains inferior to those using the F-type layers. This experimental evidence suggests that linear layers constructed based on matrix decomposition (e.g., [4, 9]) may not possess the same expressiveness as unconstrained linear layers.
A.6 Limitations and Discussions
We noticed that score-matching methods sometimes exhibit difficulty in differentiating the weights between individual modes within a multi-modal distribution. This deficiency is illustrated in Fig. A5 (a), where EBFlow fails to accurately capture the density of the Checkerboard dataset. This phenomenon bears resemblance to the blindness problem discussed in [52]. While the solution proposed in [52] has the potential to address this issue, their approach is not directly applicable to the flow-based architectures employed in this paper.
(b)
Figure A5: (a) Visualized examples of EBFlow trained with SSM, DSM, and FDSSM on the Checkerboard dataset. (b) The samples generated by the Glow model at the 40-th training epoch. (c) The samples generated by the Glow model at the 80-th training epoch.
In addition, we observed that the sampling quality of EBFlow occasionally experiences a significant reduction during the training iterations. This phenomenon is illustrated in Fig. A5 (b) and (c), where the Glow model trained using our approach demonstrates a decline in performance with extended training periods. The underlying cause of this phenomenon remains unclear, and we consider it a potential avenue for future investigation.