SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Diederik P. Kingma
Google DeepMind  
durk@google.com
Ruiqi Gao Google DeepMind ruiqig@gmail.com
To achieve the highest perceptual quality, state-of-the-art diffusion models are optimized with objectives that typically look very different from the maximum likelihood and the Evidence Lower Bound (ELBO) objectives. In this work, we reveal that diffusion model objectives are actually closely related to the ELBO.
Specifically, we show that all commonly used diffusion model objectives equate to a weighted integral of ELBOs over different noise levels, where the weighting depends on the specific objective used. Under the condition of monotonic weighting, the connection is even closer: the diffusion objective then equals the ELBO, combined with simple data augmentation, namely Gaussian noise perturbation. We show that this condition holds for a number of state-of-the-art diffusion models.
In experiments, we explore new monotonic weightings and demonstrate their effectiveness, achieving state-of-the-art FID scores on the high-resolution ImageNet benchmark.
Diffusion-based generative models, or diffusion models in short, were first introduced by Sohl-Dickstein et al. [2015]. After years of relative obscurity, this class of models suddenly rose to prominence with the work of Song and Ermon [2019] and Ho et al. [2020] who demonstrated that, with further refinements in model architectures and objective functions, diffusion models can perform state-of-the-art image generation.
(a)  $512\times 512$
(b)  $256\times 256$
(c)  $128 \times 128$
Figure 1: Samples generated from our  $VDM++$  diffusion models trained on the ImageNet dataset; see Section 5 for details and Appendix M for more samples.
The diffusion model framework has since been successfully applied to text-to-image generation [Rombach et al., 2022b, Nichol et al., 2021, Ramesh et al., 2022, Sahara et al., 2022b, Yu et al., 2022, Nichol and Dhariwal, 2021, Ho et al., 2022, Dhariwal and Nichol, 2022, Ding et al., 2021], image-to-image generation [Saharia et al., 2022c,a, Whang et al., 2022], 3D synthesis [Poole et al., 2022, Watson et al., 2022], text-to-speech [Chen et al., 2021a, Kong et al., 2021, Chen et al., 2021b], and density estimation [Kingma et al., 2021, Song et al., 2021a].
Diffusion models can be interpreted as a special case of deep variational autoencoders (VAEs) [Kingma and Welling, 2013, Rezende et al., 2014] with a particular choice of inference model and generative model. Just like VAEs, the original diffusion models [Sohl-Dickstein et al., 2015] were optimized by maximizing the variational lower bound of the log-likelihood of the data, also called the evidence lower bound, or ELBO for short. It was shown by Variational Diffusion Models (VDM) [Kingma et al., 2021] and [Song et al., 2021a] how to optimize continuous-time diffusion models with the ELBO objective, achieving state-of-the-art likelihoods on image density estimation benchmarks.
However, the best results in terms of sample quality metrics such as FID scores were achieved with other objectives, for example a denoising score matching objective [Song and Ermon, 2019] or a simple noise-prediction objective [Ho et al., 2020]. These now-popular objective functions look, on the face of it, very different from the traditionally popular maximum likelihood and ELBO objectives. Through the analysis in this paper, we reveal that all training objectives used in state-of-the-art diffusion models are actually closely related to the ELBO objective.
This paper is structured as follows:
- In Section 2 we introduce the broad diffusion model family under consideration.
- In Section 3, we show how the various diffusion model objectives in the literature can be understood as special cases of a weighted loss [Kingma et al., 2021, Song et al., 2021a], with different choices of weighting. The weighting function specifies the weight per noise level. In Section 3.2 we show that during training, the noise schedule acts as a importance sampling distribution for estimating the loss, and is thus important for efficient optimization. Based on this insight we propose a simple adaptive noise schedule.
- In Section 4, we present our main result: that if the weighting function is a monotonic function of time, then the weighted loss corresponds to maximizing the ELBO with data augmentation, namely Gaussian noise perturbation. This holds for, for example, the v-prediction loss of [Salimans and Ho, 2022] and flow matching with the optimal transport path [Lipman et al., 2022].
- In Section 5 we perform experiments with various new monotonic weights on the ImageNet dataset, and find that our proposed monotonic weighting produces models with sample quality that are competitive with the best published results, achieving state-of-art FID and IS scores on high resolution ImageNet generation.
The main sections reference much of the related work. Earlier work [Kingma et al., 2021, Song et al., 2021a, Huang et al., 2021, Vahdat et al., 2021], including Variational Diffusion Models [Kingma et al., 2021], showed how to optimize continuous-time diffusion models towards the ELBO objective. We generalize these earlier results by showing that any diffusion objective that corresponds with monotonic weighting corresponds to the ELBO, combined with a form of DistAug [Child et al., 2019]. DistAug is a method of training data distribution augmentation for generative models where the model is conditioned on the data augmentation parameter at training time, and conditioned on 'no augmentation' at inference time. The type of data augmentation under consideration in this paper, namely additive Gaussian noise, is also a form of data distribution smoothing, which has been shown to improve sample quality in autoregressive models by Meng et al. [2021].
Kingma et al. [2021] showed how the ELBO is invariant to the choice of noise schedule, except for the endpoints. We generalize this result by showing that the invariance holds for any weighting function.
Figure 2: Left: Noise schedules used in our experiments: cosine [Nichol and Dhariwal, 2021] and EDM [Karras et al., 2022] training and sampling schedules. Right: The same noise schedules, expressed as probability densities  $p(\lambda) = -dt / d\lambda$ . See Section 2.1 and Appendix E.3 for details.
Suppose we have a dataset of datapoints drawn from  $q(\mathbf{x})$ . We wish to learn a generative model  $p_{\theta}(\mathbf{x})$  that approximates  $q(\mathbf{x})$ . We'll use shorthand notation  $p \coloneqq p_{\theta}$ .
The observed variable  $\mathbf{x}$  might be the output of a pre-trained encoder, as in latent diffusion models [Vahdat et al., 2021, Rombach et al., 2022a], on which the popular Stable Diffusion model is based. Our theoretical analysis also applies to this type of model.
In addition to the observed variable  $\mathbf{x}$ , we have a series of latent variables  $\mathbf{z}_t$  for timesteps  $t \in [0,1]$ :  $\mathbf{z}_{0,\dots,1} := \mathbf{z}_0, \dots, \mathbf{z}_1$ . The model consists of two parts: a forward process forming a conditional joint distribution  $q(\mathbf{z}_{0,\dots,1}|\mathbf{x})$ , and a generative model forming a joint distribution  $p(\mathbf{z}_{0,\dots,1})$ .
2.1 Forward process and noise schedule
The forward process is a Gaussian diffusion process, giving rise to a conditional distribution  $q(\mathbf{z}_{0,\dots,1}|\mathbf{x})$ ; see Appendix E.1 for details. For every  $t \in [0,1]$ , the marginal distribution  $q(\mathbf{z}_t|\mathbf{x})$  is given by:
$$
\mathbf {z} _ {t} = \alpha_ {\lambda} \mathbf {x} + \sigma_ {\lambda} \epsilon \text {w h e r e} \epsilon \sim \mathcal {N} (0, \mathbf {I}). \tag {1}
$$
In case of the often-used variance preserving (VP) forward process,  $\alpha_{\lambda}^{2} = \mathrm{sigmoid}(\lambda_{t})$  and  $\sigma_{\lambda}^{2} = \mathrm{sigmoid}(-\lambda_{t})$ , but other choices are possible; our results are agnostic to this choice. The log signal-to-noise ratio (log-SNR) for timestep  $t$  is given by  $\lambda = \log (\alpha_{\lambda}^{2} / \sigma_{\lambda}^{2})$ .
The noise schedule is a strictly monotonically decreasing function  $f_{\lambda}$  that maps from the time variable  $t \in [0,1]$  to the corresponding log-SNR  $\lambda$ :  $\lambda = f_{\lambda}(t)$ . We sometimes denote the log-SNR as  $\lambda_t$  to emphasize that it is a function of  $t$ . The endpoints of the noise schedule are given by  $\lambda_{\max} \coloneqq f_{\lambda}(0)$  and  $\lambda_{\min} \coloneqq f_{\lambda}(1)$ . See Figure 2 for a visualization of commonly used noise schedules in the literature, and Appendix E.3 for more details.
Due to its monotonicity,  $f_{\lambda}$  is invertible:  $t = f_{\lambda}^{-1}(\lambda)$ . Given this bijection, we can do a change of variables: a function of the value  $t$  can be equivalently written as a function of the corresponding value  $\lambda$ , and vice versa, which we'll make use of in this work.
During model training, we sample time  $t$  uniformly:  $t \sim \mathcal{U}(0,1)$ , then compute  $\lambda = f_{\lambda}(t)$ . This results in a distribution over noise levels  $p(\lambda) = -dt / d\lambda = -1 / f_{\lambda}'(t)$  (see Section E.3), which we also plot in Figure 2.
Sometimes it is best to use a different noise schedule for sampling from the model than for training. During sampling, the density  $p(\lambda)$  gives the relative amount of time the sampler spends at different noise levels.
2.2 Generative model
The data  $\mathbf{x} \sim \mathcal{D}$ , with density  $q(\mathbf{x})$ , plus the forward model defines a joint distribution  $q(\mathbf{z}_0, \dots, \mathbf{z}_1) = \int q(\mathbf{z}_0, \dots, \mathbf{z}_1 | \mathbf{x}) q(\mathbf{x}) d\mathbf{x}$ , with marginals  $q_t(\mathbf{z}) := q(\mathbf{z}_t)$ . The generative model defines a corresponding joint distribution over latent variables:  $p(\mathbf{z}_0, \dots, \mathbf{z}_1)$ .
For large enough  $\lambda_{\mathrm{max}}$ ,  $\mathbf{z}_0$  is almost identical to  $\mathbf{x}$ , so learning a model  $p(\mathbf{z}_0)$  is practically equivalent to learning a model  $p(\mathbf{x})$ . For small enough  $\lambda_{\mathrm{min}}$ ,  $\mathbf{z}_1$  holds almost no information about  $\mathbf{x}$ , such that there exists a distribution  $p(\mathbf{z}_1)$  satisfying  $D_{KL}(q(\mathbf{z}_1|\mathbf{x})||p(\mathbf{z}_1)) \approx 0$ . Usually we can use  $p(\mathbf{z}_1) = \mathcal{N}(0,\mathbf{I})$ .
Let  $\mathbf{s}_{\theta}(\mathbf{z};\lambda)$  denote a score model, which is a neural network that we let approximate  $\nabla_{\mathbf{z}}\log q_t(\mathbf{z})$  through methods introduced in the next sections. If  $\mathbf{s}_{\theta}(\mathbf{z};\lambda) = \nabla_{\mathbf{z}}\log q_t(\mathbf{z})$ , then the forward process can be exactly reversed; see Appendix E.4.
If  $D_{KL}(q(\mathbf{z}_1)||p(\mathbf{z}_1))\approx 0$  and  $\mathbf{s}_{\theta}(\mathbf{z};\lambda)\approx \nabla_{\mathbf{z}}\log q_t(\mathbf{z})$ , then we have a good generative model in the sense that  $D_{KL}(q(\mathbf{z}_{0,\dots,1})||p(\mathbf{z}_{0,\dots,1}))\approx 0$ , which implies that  $D_{KL}(q(\mathbf{z}_0)||p(\mathbf{z}_0))\approx 0$  which achieves our goal. So, our generative modeling task is reduced to learning a score network  $\mathbf{s}_{\theta}(\mathbf{z};\lambda)$  that approximates  $\nabla_{\mathbf{z}}\log q_t(\mathbf{z})$ .
Sampling from the generative model can be performed by sampling  $\mathbf{z}_1\sim p(\mathbf{z}_1)$ , then (approximately) solving the reverse SDE using the estimated  $\mathbf{s}_{\theta}(\mathbf{z};\lambda)$ . Recent diffusion models have used increasingly sophisticated procedures for approximating the reverse SDE; see Appendix E.4. In experiments we use the DDPM sampler from Ho et al. [2020] and the stochastic sampler with Heun's second order method proposed by Karras et al. [2022].
Denoising score matching. Above, we saw that we need to learn a score network  $\mathbf{s}_{\theta}(\mathbf{z};\lambda_t)$  that approximates  $\nabla_{\mathbf{z}}\log q_t(\mathbf{z})$ , for all noise levels  $\lambda_t$ . It was shown by [Vincent, 2011, Song and Ermon, 2019] that this can be achieved by minimizing a denoising score matching objective over all noise scales and all datapoints  $\mathbf{x}\sim \mathcal{D}$ :
$$
\mathcal {L} _ {\mathrm {D S M}} (\mathbf {x}) = \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \boldsymbol {\epsilon} \sim \mathcal {N} (0, \mathbf {I})} \left[ \tilde {w} (t) \cdot | | \mathbf {s} _ {\boldsymbol {\theta}} (\mathbf {z} _ {t}; \lambda_ {t}) - \nabla_ {\mathbf {z} _ {t}} \log q (\mathbf {z} _ {t} | \mathbf {x}) | | _ {2} ^ {2} \right] \tag {2}
$$
where  $\mathbf{z}_t = \alpha_\lambda \mathbf{x} + \sigma_\lambda \boldsymbol{\epsilon}$
The  $\epsilon$ -prediction objective. Most contemporary diffusion models are optimized towards a noise-prediction loss introduced by [Ho et al., 2020]. In this case, the score network is typically parameterized through a noise-prediction ( $\epsilon$ -prediction) model:  $s_{\theta}(\mathbf{z}; \lambda) = -\hat{\epsilon}_{\theta}(\mathbf{z}; \lambda) / \sigma_{\lambda}$ . (Other options, such as  $\mathbf{x}$ -prediction,  $\mathbf{v}$ -prediction, and EDM parameterizations, are explained in Appendix E.2.) The noise-prediction loss is:
$$
\mathcal {L} _ {\boldsymbol {\epsilon}} (\mathbf {x}) = \frac {1}{2} \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \boldsymbol {\epsilon} \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \left| \hat {\epsilon} _ {\boldsymbol {\theta}} \left(\mathbf {z} _ {t}; \lambda_ {t}\right) - \boldsymbol {\epsilon} \right| \right| _ {2} ^ {2} \right] \tag {3}
$$
Since  $||\mathbf{s}_{\pmb{\theta}}(\mathbf{z}_t;\lambda_t) - \nabla_{\mathbf{x}_t}\log q(\mathbf{z}_t|\mathbf{x})||_2^2 = \sigma_\lambda^{-2}||\hat{\epsilon}_{\pmb{\theta}}(\mathbf{z}_t;\lambda_t) - \pmb {\epsilon}||_2^2$  , this is simply a version of the denoising score matching objective in Equation 2, but where  $\tilde{w} (t) = \sigma_t^2$  . Ho et al. [2020] showed that this noise-prediction objective can result in high-quality samples. Dhariwal and Nichol [2022] later improved upon these results by switching from a 'linear' to a cosine' noise schedule  $\lambda_{t}$  (see Figure 2). This noise-prediction loss with the cosine schedule is currently broadly used.
The ELBO objective. It was shown by [Kingma et al., 2021, Song et al., 2021a] that the evidence lower bound (ELBO) of continuous-time diffusion models simplifies to:
$$
- \operatorname {E L B O} (\mathbf {x}) = \frac {1}{2} \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \epsilon \sim \mathcal {N} (0, \mathbf {I})} \left[ - \frac {d \lambda}{d t} \cdot | | \hat {\epsilon} _ {\boldsymbol {\theta}} (\mathbf {z} _ {t}; \lambda_ {t}) - \epsilon | | _ {2} ^ {2} \right] + c \tag {4}
$$
where  $c$  is constant w.r.t. the score network parameters.
Loss function	Implied weighting w(λ)	Monotonic?
ELBO [Kingma et al., 2021, Song et al., 2021a]	1	✓
IDDPM (ε-prediction with 'cosine' schedule) [Nichol and Dhariwal, 2021]	sech(λ/2)
EDM [Karras et al., 2022] (Appendix D.1)	N(λ; 2.4, 2.42) · (e-λ + 0.52)
v-prediction with 'cosine' schedule [Salimans and Ho, 2022] (Appendix D.2)	e-λ/2	✓
Flow Matching with OT path (FM-OT) [Lipman et al., 2022] (Appendix D.3)	e-λ/2	✓
InDI [Delbracio and Milanfar, 2023] (Appendix D.4)	e-λsech2(λ/4)	✓
P2 weighting with 'cosine' schedule [Choi et al., 2022] (Appendix D.5)	sech(λ/2)/(1 + eλ)γ, γ = 0.5 or 1
Min-SNR-γ [Hang et al., 2023] (Appendix D.6)	sech(λ/2) · min(1, γe-λ)
Table 1: Diffusion model objectives in the literature are special cases of the weighted loss with a weighting function  $w(\lambda)$  given in this table. See Section 3.1 and Appendix D for more details and derivations. Most existing weighting functions are non-monotonic, except for the ELBO objective and the v-prediction objective with 'cosine' schedule.
Figure 3: Diffusion model objectives in the literature are special cases of the weighted loss with non-monotonic (left) or monotonic (right) weighting functions. Each weighting function is scaled such that the maximum is 1 over the plotted range. See Table 1 and Appendix D.
3.1 The weighted loss
The different objective functions used in practice, including the ones above, can be viewed as special cases of the weighted loss introduced by Kingma et al. [2021] with a different choices of weighting function  $w(\lambda_t)$ :
$$
\mathcal {L} _ {w} (\mathbf {x}) = \frac {1}{2} \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \boldsymbol {\epsilon} \sim \mathcal {N} (0, \mathbf {I})} \left[ w \left(\lambda_ {t}\right) \cdot - \frac {d \lambda}{d t} \cdot | | \hat {\boldsymbol {\epsilon}} _ {\boldsymbol {\theta}} \left(\mathbf {z} _ {t}; \lambda_ {t}\right) - \boldsymbol {\epsilon} | | _ {2} ^ {2} \right] \tag {5}
$$
See Appendix D for a derivation of the implied weighting functions for all popular diffusion losses. Results are compiled in Table 1, and visualized in Figure 3.
The ELBO objective (Equation 4) corresponds to uniform weighting, i.e.  $w(\lambda_t) = 1$ .
The popular noise-prediction objective (Equation 3) corresponds to  $w(\lambda_t) = -1 / (d\lambda /dt)$ . This is more compactly expressed as  $w(\lambda_{t}) = p(\lambda_{t})$ , i.e., the PDF of the implied distribution over noise levels  $\lambda$  during training. Typically, the noise-prediction objective is used with the cosine schedule  $\lambda_{t}$  which implies  $w(\lambda_t)\propto \mathrm{sech}(\lambda_t / 2)$ . See Section E.3 for the expression of  $p(\lambda_t)$  for various noise schedules.
3.2 Invariance of the weighted loss to the noise schedule  $\lambda_{t}$
In Kingma et al. [2021], it was shown that the ELBO objective (Equation 4) is invariant to the choice of noise schedule, except for its endpoints  $\lambda_{\mathrm{min}}$  and  $\lambda_{\mathrm{max}}$ . This result can be extended to the much
<sup>1</sup>More specifically, Kingma et al. [2021] expressed the weighted diffusion loss in terms of  $\mathbf{x}$ -prediction, which is equivalent to the expression above due to the relationship  $\int ||\pmb{\epsilon} - \hat{\pmb{\epsilon}}_{\pmb{\theta}}(\mathbf{z}_t; \lambda)||_2^2 d\lambda = \int ||\mathbf{x} - \hat{\mathbf{x}}_{\pmb{\theta}}(\mathbf{z}_t; \lambda)||_2^2 e^{\lambda} d\lambda = \int ||\mathbf{x} - \hat{\mathbf{x}}_{\pmb{\theta}}(\mathbf{z}_t; \lambda)||_2^2 de^{\lambda}$ , where  $e^{\lambda}$  equals the signal-to-noise ratio (SNR).
more general weighted diffusion loss of Equation 5, since with a change of variables from  $t$  to  $\lambda$ , it can be rewritten to:
$$
\mathcal {L} _ {w} (\mathbf {x}) = \frac {1}{2} \int_ {\lambda_ {\min }} ^ {\lambda_ {\max }} w (\lambda) \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \left| \hat {\epsilon} _ {\theta} (\mathbf {z} _ {\lambda}; \lambda) - \epsilon \right| \right| _ {2} ^ {2} \right] d \lambda \tag {6}
$$
Note that this integral does not depend on the noise schedule  $f_{\lambda}$  (the mapping between  $t$  and  $\lambda$ ), except for its endpoints  $\lambda_{\mathrm{max}}$  and  $\lambda_{\mathrm{min}}$ . The shape of the function  $f_{\lambda}$  between  $\lambda_{\mathrm{min}}$  and  $\lambda_{\mathrm{max}}$  does not affect the loss; only the weighting function  $w(\lambda)$  does. Given a chosen weighting function  $w(\lambda)$ , the loss is invariant to the noise schedule  $\lambda_t$  between  $t = 0$  and  $t = 1$ . This is important, since it means that the only real difference between diffusion objectives is their difference in weighting  $w(\lambda)$ .
This invariance does not hold for the Monte Carlo estimator of the loss that we use in training, based on random samples  $t \sim \mathcal{U}(0,1)$ ,  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ . The noise schedule still affects the variance of this Monte Carlo estimator and its gradients; therefore, the noise schedule affects the efficiency of optimization. More specifically, we'll show that the noise schedule acts as an importance sampling distribution for estimating the loss integral of Equation 6. Note that  $p(\lambda) = -1 / (d\lambda /dt)$ . We can therefore rewrite the weighted loss as the following, which clarifies the role of  $p(\lambda)$  as an importance sampling distribution:
$$
\mathcal {L} _ {w} (\mathbf {x}) = \frac {1}{2} \mathbb {E} _ {\boldsymbol {\epsilon} \sim \mathcal {N} (0, \mathbf {I}), \lambda \sim p (\lambda)} \left[ \frac {w (\lambda)}{p (\lambda)} | | \hat {\epsilon} _ {\boldsymbol {\theta}} (\mathbf {z} _ {\lambda}; \lambda) - \epsilon | | _ {2} ^ {2} \right] \tag {7}
$$
Using this insight, we implemented an adaptive noise schedule, detailed in Appendix F. We find that by lowering the variance of the loss estimator, this often significantly speeds up optimization.
4 The weighted loss as the ELBO with data augmentation
We prove the following main result:
Theorem 1. If the weighting  $w(\lambda_t)$  is monotonic, then the weighted diffusion objective of Equation 5 is equivalent to the ELBO with data augmentation (additive noise).
With monotonic  $w(\lambda_t)$  we mean that  $w$  is a monotonically increasing function of  $t$ , and therefore a monotonically decreasing function of  $\lambda$ .
We'll use shorthand notation  $\mathcal{L}(t;\mathbf{x})$  for the KL divergence between the joint distributions of the forward process  $q(\mathbf{z}_{t,\dots 1}|\mathbf{x})$  and the reverse model  $p(\mathbf{z}_{t,\dots 1})$ , for the subset of timesteps from  $t$  to 1:
$$
\mathcal {L} (t; \mathbf {x}) := D _ {K L} \left(q \left(\mathbf {z} _ {t, \dots , 1} | \mathbf {x}\right) \| p \left(\mathbf {z} _ {t, \dots , 1}\right)\right) \tag {8}
$$
In Appendix A.1, we prove that:
$$
\frac {d}{d t} \mathcal {L} (t; \mathbf {x}) = \frac {1}{2} \frac {d \lambda}{d t} \mathbb {E} _ {\boldsymbol {\epsilon} \sim \mathcal {N} (0, \mathbf {I})} \left[ \| \boldsymbol {\epsilon} - \hat {\boldsymbol {\epsilon}} _ {\boldsymbol {\theta}} (\mathbf {z} _ {\lambda}; \lambda) \| _ {2} ^ {2} \right] \tag {9}
$$
As shown in Appendix A.1, this allows us to rewrite the weighted loss of Equation 5 as simply:
$$
\mathcal {L} _ {w} (\mathbf {x}) = - \int_ {0} ^ {1} \frac {d}{d t} \mathcal {L} (t; \mathbf {x}) w \left(\lambda_ {t}\right) d t \tag {10}
$$
In Appendix A.2, we prove that using integration by parts, the weighted loss can then be rewritten as:
$$
\mathcal {L} _ {w} (\mathbf {x}) = \int_ {0} ^ {1} \frac {d}{d t} w \left(\lambda_ {t}\right) \mathcal {L} (t; \mathbf {x}) d t + w \left(\lambda_ {\max }\right) \mathcal {L} (0; \mathbf {x}) + \text {c o n s t a n t} \tag {11}
$$
${}^{2}$  Interestingly, this reveals a new relationship between the KL and Fisher divergences:  $\frac{d}{d\lambda }{D}_{KL}\left( {q\left( {{\mathbf{z}}_{t,\ldots ,1} \mid  \mathbf{x}}\right) }\right| p\left( {{\mathbf{z}}_{t,\ldots ,1}}\right) )$ $= \frac{1}{2}{\sigma }_{\lambda }^{2}{D}_{F}\left( {q\left( {{\mathbf{z}}_{t} \mid  \mathbf{x}}\right) }\right| p\left( {\mathbf{z}}_{t}\right) )$  . See Appendix G for a derivation,a discussion, and a comparison with a similar result by Lyu [2012].
Now, assume that  $w(\lambda_t)$  is a monotonically increasing function of  $t \in [0,1]$ . Also, without loss of generality, assume that  $w(\lambda_t)$  is normalized such that  $w(\lambda_1) = 1$ . We can then further simplify the weighted loss to an expected KL divergence:
$$
\mathcal {L} _ {w} (\mathbf {x}) = \mathbb {E} _ {p _ {w} (t)} [ \mathcal {L} (t; \mathbf {x}) ] + \text {c o n s t a n t} \tag {12}
$$
where  $p_w(t)$  is a probability distribution determined by the weighting function, namely  $p_w(t) \coloneqq (d / dt w(\lambda_t))$ , with support on  $t \in [0,1]$ . The probability distribution  $p_w(t)$  has Dirac delta peak of typically very small mass  $w(\lambda_{\max})$  at  $t = 0$ .
$$
\mathcal {L} (t; \mathbf {x}) = D _ {K L} \left(q \left(\mathbf {z} _ {t, \dots , 1} | \mathbf {x}\right) \mid \mid p \left(\mathbf {z} _ {t, \dots , 1}\right)\right) \tag {13}
$$
$$
\geq D _ {K L} \left(q \left(\mathbf {z} _ {t} \mid \mathbf {x}\right) \| p \left(\mathbf {z} _ {t}\right)\right) = - \mathbb {E} _ {q \left(\mathbf {z} _ {t} \mid \mathbf {x}\right)} \left[ \log p \left(\mathbf {z} _ {t}\right) \right] + \text {c o n s t a n t}. \tag {14}
$$
More specifically,  $\mathcal{L}(t;\mathbf{x})$  equals the expected negative ELBO of noise-perturbed data, plus a constant; see Section C for a detailed derivation.