SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \nabla_ {\theta} L _ {t, t ^ {\prime}, x} ^ {\mathrm {S M}} (\theta) = \frac {1}{2} \nabla_ {\theta} \left(\mathbb {E} _ {\theta} [ h _ {\theta} (x _ {t ^ {\prime}}, t ^ {\prime}) ] - h _ {\theta} (x, t)\right) ^ {2} \\ = \left(\mathbb {E} _ {\theta} \left[ h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) \right] - h _ {\theta} (x, t)\right) ^ {\top} \left(\nabla_ {\theta} \mathbb {E} \left[ h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) \right] - \nabla_ {\theta} h _ {\theta} (x, t)\right) \\ = \mathbb {E} _ {\theta} \left[ h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) - h _ {\theta} (x, t) \right] ^ {\top} \mathbb {E} _ {\theta} \left[ \nabla_ {\theta} h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) - \nabla_ {\theta} h _ {\theta} (x, t) \right] \\ \left. + \mathbb {E} _ {\theta} \left[ h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) - h _ {\theta} (x, t) \right] ^ {\top} \mathbb {E} _ {\theta} \left[ h _ {\theta} \left(x _ {t ^ {\prime}}, t ^ {\prime}\right) \nabla_ {\theta} \log \left(p _ {\theta} \left(x _ {[ t ^ {\prime}, t ]} \mid x _ {t} = x\right)\right) \right] \right. \\ \end{array}
$$
Let us compute the gradient of the log density. We use the discrete process, and let us assume that  $t = t_0 > t_1 > \dots > t_k = t'$  are the sampling times. Then,
$$
p _ {\theta} \left(x _ {[ t ^ {\prime}, t ]} \mid x _ {t} = x\right) = \prod_ {i = 1} ^ {k} p _ {\theta} \left(x _ {t _ {i}} \mid x _ {t _ {i - 1}}\right).
$$
We assume that
$$
p _ {\theta} \left(x _ {t _ {i}} \mid t _ {i - 1}\right) = \mathcal {N} \left(\mu_ {\theta , i}, g _ {i} I _ {d}\right).
$$
$$
p _ {\theta} (x _ {[ t ^ {\prime}, t ]} \mid x _ {t} = x) \propto \prod_ {i = 1} ^ {k} \exp \left(- \frac {\| \mu_ {\theta , i} - (x _ {t _ {i}} - x _ {t _ {i - 1}}) \| ^ {2}}{2 g _ {i} ^ {2}}\right)
$$
$$
\log p _ {\theta} (x _ {[ t ^ {\prime}, t ]} \mid x _ {t} = x) = C + \sum_ {i = 1} ^ {k} \frac {\| \mu_ {\theta , i} - (x _ {t _ {i}} - x _ {t _ {i - 1}}) \| ^ {2}}{2 g _ {i} ^ {2}}
$$
where  $C$  corresponds to the normalizing factor that is independent of  $\theta$ . Differentiating, we get that
$$
\nabla_ {\theta} \log p _ {\theta} (x _ {[ t ^ {\prime}, t ]} \mid x _ {t} = x) = \sum_ {i = 1} ^ {k} \frac {\left(\mu_ {\theta , i} - (x _ {t _ {i}} - x _ {t _ {i - 1}})\right) ^ {\top} \nabla_ {\theta} \mu_ {\theta , i}}{g _ {i} ^ {2}}
$$
B.2 Proof of Lemma 3.1
In what appears below, the expectation  $\mathbb{E}[\cdot \mid x_t = x]$  is taken with respect to the distribution obtained by Eq. (7), namely, the backward SDE that corresponds to the function  $s$ , with the initial condition  $x_{t} = x$ . Similarly,  $\mathbb{E}[\cdot \mid x_{t^{\prime}}]$  is taken with the initial condition at  $x_{t^{\prime}}$ . To prove the first direction in the equivalence, assume that Property 1 holds and our goal is to prove the two consequences as described in the lemma. To prove the first consequence, by the law of total expectation and by the fact that  $x_{t} - x_{t^{\prime}} - x_{0}$  is a Markov chain, namely,  $x_0$  and  $x_{t}$  are independent conditioned on  $x_{t^{\prime}}$ , we obtain that
$$
h (x, t) = \mathbb {E} [ x _ {0} \mid x _ {t} = x ] = \mathbb {E} [ \mathbb {E} [ x _ {0} \mid x _ {t ^ {\prime}} ] \mid x _ {t} = x ] = \mathbb {E} [ h (x _ {t ^ {\prime}}, t ^ {\prime}) \mid x _ {t} = x ].
$$
To prove the second consequence, by Property 1
$$
h (x, 0) = \mathbb {E} [ x _ {0} \mid x _ {0} = x ] = x _ {0}.
$$
This concludes the first direction in the equivalence.
To prove the second direction, assume that  $h(x,t) = \mathbb{E}[h(x_{t'},t')\mid x_t = x]$  and that  $h(x,0) = x$  and notice that by substituting  $t^\prime = 0$  we derive the following:
$$
h (x, t) = \mathbb {E} [ h (x _ {0}, 0) \mid x _ {t} = x ] = \mathbb {E} [ x _ {0} \mid x _ {t} = x ],
$$
C.1 Property Testing
Discussion. We add some details about the plots:
- We note that the plots are not monotonic. This might be attributed to the model being closer to the ideal denoiser at certain noise levels and perhaps for having the capacity to correct previous mistakes.
- For each plot, we sample 32 images, and for each  $t$  we get 32 stochastic samples conditioned on each one of the images and we measure the violation of MP. Standard deviation was small that it was not visible in the plot.
-  $\sigma_{t}$  denotes the standard deviation of the noise at time  $t$  and it is a monotonic function of  $t$ .
We underline that for the perfect denoiser, the error would have been 0 everywhere. However, even after the MP regularization, the learning is not perfect hence there are errors.
Consistency Property Testing (CIFAR10) Backward Sampler (300) steps
Figure 4: Consistency Property Testing on CIFAR-10. The plot illustrates how the Consistency Loss,  $L_{t,t',x_t}^{\mathrm{CP}}$ , behaves for  $t' = 0$ , as  $t$  changes.
Consistency Property Testing (CIFAR10) Backward Sampler (300) steps
Figure 5: Consistency Property Testing on CIFAR-10. The plot illustrates how the Consistency Loss,  $L_{t,t',x_t}^{\mathrm{CP}}$ , behaves for  $t = 0$ , as  $t'$  changes.
Consistency Property Testing (FFHQ) Backward Sampler (1000) steps
Figure 6: Consistency Property Testing on FFHQ. The plot illustrates how the Consistency Loss,  $L_{t,t',x_t}^{\mathrm{CP}}$ , behaves for  $t' = 0$ , as  $t$  changes.
Consistency Property Testing (FFHQ) Backward Sampler (1000) steps
Figure 7: Consistency Property Testing on FFHQ. The plot illustrates how the Consistency Loss,  $L_{t,t',x_t}^{\mathrm{CP}}$ , behaves for  $t = 0$ , as  $t'$  changes.
C.2 Uncurated Samples
Figure 8: Uncurated generated images by our fine-tuned model on FFHQ. FID: 2.61, NFEs: 79.
D Implementation details
Input: dataset  $\mathcal{D}$ , noise schedule  $\{\sigma_t\}_0^T$ , initial model parameters  $\theta$ , learning rate  $\eta$ , Consistency Property (CP) regularization strength  $\lambda$ , maximum distance  $\epsilon$  between  $t$  and  $t'$ , and step size  $\Delta t$  for discretizing the Reverse SDE.
Algorithm 1 Consistent Diffusion Models (CDM) Training
repeat Sample  $x_0\sim \mathcal{D}$  (Clean image) Sample  $\sigma_t$  uniformly from  $\{\sigma_t\}_{\epsilon}^T$  (Corruption level) Sample  $x_{t}\sim \mathcal{N}(x_{0},\sigma_{t}^{2}I_{d})$  (Corrupted image)  $L^{\mathrm{SM}}\gets ||h_{\theta}(x_{t},t) - x_{0}||^{2}$  (Compute Score Matching loss) Sample  $t^\prime \sim \mathcal{U}[t - \epsilon ,t]$  (Level with less corruption) Stopgrad Sample  $x_{t'}^{(1)},x_{t'}^{(2)}$  by running  $\lfloor \frac{\epsilon}{\Delta t}\rfloor$  steps of the Reverse SDE (Sample less corrupted images) Resumegrad  $L^{\mathrm{CP}}\leftarrow \left(h_{\theta}(x_{t'}^{(1)},t') - h_{\theta}(x_{t},t)\right)^{T}\left(h_{\theta}(x_{t'}^{(2)},t') - h_{\theta}(x_{t},t)\right)$  Enforce CP with stochastic gradient.  $\theta \leftarrow \theta -\eta \nabla_{\theta}\left(L^{\mathrm{SM}} + \lambda L^{\mathrm{CP}}\right)$  (Update parameters) until convergence
The capacity for generative models to exert consequential societal influence in myriad ways is undeniable, and it also brings along a multiplicity of inherent risks [37, 25, 26, 27]. These models, for instance, may be exploited to fabricate counterfeit images, and furthermore, they have the potential to intensify societal prejudices. This work does not seem to exert a direct influence on these particular biases. It is imperative to acknowledge that addressing such biases presents a substantial challenge.