SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {v a r} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ] = \mathbb {E} _ {x _ {0}} [ \psi^ {2} (\eta (1 - \alpha_ {t} x _ {0})) ] - (\mathbb {E} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ]) ^ {2} \\ \approx \max  \left(\frac {1}{1 0 0} \sum_ {i = 0} ^ {1 0 0} \frac {\psi^ {2} \left(\eta \left(1 - \alpha_ {t} \left(x _ {\operatorname* {m i n}} + \frac {i}{1 0 0} (x _ {\operatorname* {m a x}} - x _ {\operatorname* {m i n}})\right)\right)\right)}{2 \mathbf {1} (i = 0) + \mathbf {1} (i = 1 0 0)} - (\mathbb {E} _ {x _ {0}} [ \psi (\eta (1 - \alpha_ {t} x _ {0})) ]) ^ {2}, 0\right). \\ \end{array}
$$
Now we are ready to precondition  $\mathrm{logit}(z_t)$  as
$$
g (z _ {t}) = \frac {\operatorname {l o g i t} (z _ {t}) - \mathbb {E} [ \operatorname {l o g i t} (z _ {t}) ]}{\sqrt {\operatorname {v a r} [ \operatorname {l o g i t} (z _ {t}) ]}}.
$$
We use  $g(z_{t})$  as the input to the generator  $f_{\theta}$  and add a skip connection layer to add  $g(z_{t})$  into the output of the generator. Specifically, following the notation of EDM, we define the network as
$$
f _ {\theta} (z _ {t}, t) = c _ {\text {s k i p}} (t) g (z _ {t}) + c _ {\text {o u t}} (t) F _ {\theta} \left(c _ {\text {i n}} (t) g (z _ {t}), c _ {\text {n o i s e}} (t)\right),
$$
where for simplicity, we set  $c_{\mathrm{skip}}(t) = c_{\mathrm{out}}(t) = c_{\mathrm{in}}(t) = 1$  and  $c_{\mathrm{noise}}(t) = -\log \mathrm{i}(\alpha_t) / 8$ . A more sophisticated selection of these parameters has the potential to enhance the performance of beta diffusion. However, due to our current limitations in computing resources, we defer this investigation to future studies.
G Parameter Settings of Beta Diffusion on CIFAR10
We present the intuition on how we set the model parameters, including  $S_{\text{shift}}$ ,  $S_{\text{scale}}$ ,  $\eta$ ,  $\omega$ ,  $\pi$ ,  $c_0$ , and  $c_1$ . Given the limitation of our computation resources, we leave the careful tuning of these model parameters to future work.
We set  $\omega = 0.99$  and  $\pi = 0.95$  across all experiments conducted on CIFAR10 with beta diffusion.
We pre-process the range-bounded data by linearly scaling and shifting them to lie between  $[S_{shift}, S_{shift} + S_{\text{scale}}]$ , where  $0 < S_{shift} < S_{shift} + S_{\text{scale}} < 1$ . This linear transformation serves two purposes: firstly, it enables us to use beta diffusion to model any range-bounded data, and secondly, it helps avoid numerical challenges in computing the log-beta divergence when its arguments are small. When evaluating the performance, we linearly transform the generated data back to their original scale by reversing the scaling and shifting operations applied during the pre-processing step. For CIFAR10 images, as beta diffusion is diffusing pixels towards  $\mathrm{Beta}(0, \eta)$ , our intuition is to set both  $S_{\text{scale}}$  and  $S_{shift}$  large enough to differentiate the diffusion trajectories of different pixel values. However,  $S_{\text{scale}} + S_{shift}$  needs to be smaller than 1, motivating us to choose  $S_{\text{scale}} = 0.39$  and  $S_{shift} = 0.60$ .
For the diffusion concentration parameter  $\eta$ , our intuition is that a larger  $\eta$  provides a higher ability to differentiate different pixel values and allows a finer discretization of the reverse diffusion process, but leads to slower training and demands more discretized steps during sampling. We set  $\eta = 10000$  and the mini-batch size as  $B = 288$  by default, but also perform an ablation study on CIFAR10 with several different values, as shown in Table 3.
We use a sigmoid-based diffusion schedule as
$$
\alpha_ {t} = 1 / \left(1 + e ^ {- c _ {0} - \left(c _ {1} - c _ {0}\right) t}\right),
$$
where we set  $c_{0} = 10$  and  $c_{1} = -13$  by default. We note a similar schedule had been introduced in Kingma et al. [35] and Jabri et al. [29] for Gaussian diffusion.
For the CIFAR-10 dataset $^2$ , we utilize the parameterization of  $\mathrm{EDM}^3$  [34] as the code base. We replace the variance preserving (VP) loss and the corresponding network parameterization implemented in EDM, which is the weighted negative ELBO of Gaussian diffusion, with the KLUB loss of beta diffusion that is given by (24). We keep the generator network the same as that of VP-EDM, except that we set  $c_{noise} = -\mathrm{logit}(\alpha_t) / 8$  and simplify the other parameters as  $c_{\mathrm{skip}} = 1$ ,  $c_{\mathrm{out}} = 1$ , and  $c_{\mathrm{in}} = 1$ . The image pixel values between 0 and 255 are divided by 255 and then scaled by  $S_{\mathrm{scale}}$  and shifted by  $S_{\mathrm{shift}}$ , and hence we have  $x_0 \in [S_{\mathrm{shift}}, S_{\mathrm{shift}} + S_{\mathrm{scale}}]$ . The two inputs to the generator network  $f_\theta(\cdot, \cdot)$ , originally designed for VP-EDM and adopted directly for beta diffusion, are  $g(\mathrm{logit}(z_t))$  and  $\mathrm{logit}(\alpha_t)$ , where  $g(\cdot)$  is a preconditioning function described in detail in Appendix F. The output of the generator network is first transformed by a sigmoid function and then scaled by  $S_{\mathrm{scale}}$  and shifted by  $S_{\mathrm{shift}}$ , and hence  $\hat{x}_0 = f_\theta(z_t, t) \in [S_{\mathrm{shift}}, S_{\mathrm{shift}} + S_{\mathrm{scale}}]$ .
$^{2}$ https://www.cs.toronto.edu/~kriz/cifar.html
<sup>3</sup>https://github.com/NVlabs/edm