SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {\mathbf {v}} _ {t} = \hat {\mathbf {v}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \mathbf {c}) \quad \triangleright \text {T e a c h e r U N e t f o r s t} \\ \Rightarrow \hat {\mathbf {x}} _ {t} = \alpha_ {t} \mathbf {z} _ {t} - \sigma_ {t} \hat {\mathbf {v}} _ {t}, \triangleright \text {T e a c h e r p r e d i c t e d c l e a n l a t e n t a t t i m e} t \\ \hat {\mathbf {e}} _ {t} = \sigma_ {t} \mathbf {z} _ {t} + \alpha_ {t} \hat {\mathbf {v}} _ {t}, \triangleright \text {T e a c h e r p r e d i c t e d n o i s e a t t i m e} t \tag {12} \\ \Rightarrow \mathbf {z} _ {t ^ {\prime}} = \alpha_ {t ^ {\prime}} \hat {\mathbf {x}} _ {t} + \sigma_ {t ^ {\prime}} \hat {\boldsymbol {\epsilon}} _ {t} \quad \triangleright \text {T e a c h e r p r e d i c t e d n o i s y l a t e n t a t t i m e} t ^ {\prime} \\ = \alpha_ {t ^ {\prime}} \left(\alpha_ {t} \mathbf {z} _ {t} - \sigma_ {t} \hat {\mathbf {v}} _ {t}\right) + \sigma_ {t ^ {\prime}} \left(\sigma_ {t} \mathbf {z} _ {t} + \alpha_ {t} \hat {\mathbf {v}} _ {t}\right). \\ \end{array}
$$
The process from  $t'$  to  $t''$  can be derived just like the above, by replacing  $t$  and  $t'$  with  $t'$  and  $t''$ , respectively:
$$
\begin{array}{l} \hat {\mathbf {v}} _ {t ^ {\prime}} = \hat {\mathbf {v}} _ {\boldsymbol {\theta}} \left(t ^ {\prime}, \mathbf {z} _ {t ^ {\prime}}, \mathbf {c}\right) \quad \triangleright \text {T e a c h e r U N e t s e c o n d f o r w a r d} \\ \Rightarrow \hat {\mathbf {x}} _ {t ^ {\prime}} = \alpha_ {t ^ {\prime}} \mathbf {z} _ {t ^ {\prime}} - \sigma_ {t ^ {\prime}} \hat {\mathbf {v}} _ {t ^ {\prime}}, \triangleright \text {T e a c h e r p r e d i c t e d c l e a n l a t e n t a t t i m e} t ^ {\prime} \\ \hat {\boldsymbol {\epsilon}} _ {t ^ {\prime}} = \sigma_ {t ^ {\prime}} \mathbf {z} _ {t ^ {\prime}} + \alpha_ {t ^ {\prime}} \hat {\mathbf {v}} _ {t ^ {\prime}}, \triangleright \text {T e a c h e r p r e d i c t e d n o i s e a t t i m e} t ^ {\prime} \tag {13} \\ \Rightarrow \mathbf {z} _ {t ^ {\prime \prime}} = \alpha_ {t ^ {\prime \prime}} \hat {\mathbf {x}} _ {t ^ {\prime}} + \sigma_ {t ^ {\prime \prime}} \hat {\mathbf {e}} _ {t ^ {\prime}} \quad \triangleright \text {T e a c h e r p r e d i c t e d n o i s y l a t e n t a t t i m e} t ^ {\prime \prime} \\ = \alpha_ {t ^ {\prime \prime}} \left(\alpha_ {t ^ {\prime}} \mathbf {z} _ {t ^ {\prime}} - \sigma_ {t ^ {\prime}} \hat {\mathbf {v}} _ {t ^ {\prime}}\right) + \sigma_ {t ^ {\prime \prime}} \left(\sigma_ {t} \mathbf {z} _ {t ^ {\prime}} + \alpha_ {t} \hat {\mathbf {v}} _ {t ^ {\prime}}\right). \\ \end{array}
$$
$$
\begin{array}{l} \hat {\mathbf {v}} _ {t} ^ {(s)} = \hat {\mathbf {v}} _ {\boldsymbol {\eta}} (t, \mathbf {z} _ {t}, \mathbf {c}) \quad \triangleright \text {S t u d e n t U N e t f o r w a r d} \\ \Rightarrow \hat {\mathbf {x}} _ {t} ^ {(s)} = \alpha_ {t} \mathbf {z} _ {t} - \sigma_ {t} \hat {\mathbf {v}} _ {t} ^ {(s)}, \quad \triangleright \text {S t u d e n t p r e d i c t e d c l e a n l a t e n t a t t i m e} t \\ \hat {\boldsymbol {\epsilon}} _ {t} ^ {(s)} = \left(\mathbf {z} _ {t} - \alpha_ {t} \hat {\mathbf {x}} _ {t} ^ {(s)}\right) / \sigma_ {t}, \quad \triangleright \text {S t u d e n t p r e d i c t e d n o i s e a t t i m e} t \\ \Rightarrow \mathbf {z} _ {t ^ {\prime \prime}} ^ {(s)} = \alpha_ {t ^ {\prime \prime}} \hat {\mathbf {x}} _ {t} ^ {(s)} + \sigma_ {t ^ {\prime \prime}} \hat {\boldsymbol {\epsilon}} _ {t} ^ {(s)} \quad \triangleright \text {S t u d e n t p r e d i c t e d n o i s y l a t e n t a t t i m e} t ^ {\prime \prime} \\ = \alpha_ {t ^ {\prime \prime}} \hat {\mathbf {x}} _ {t} ^ {(s)} + \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \left(\mathbf {z} _ {t} - \alpha_ {t} \hat {\mathbf {x}} _ {t} ^ {(s)}\right) \tag {14} \\ = (\alpha_ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \alpha_ {t}) \dot {\mathbf {x}} _ {t} ^ {(s)} + \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \mathbf {z} _ {t}, \\ \Rightarrow \hat {\mathbf {x}} _ {t} ^ {(s)} = \frac {\mathbf {z} _ {t ^ {\prime \prime}} ^ {(s)} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \mathbf {\bar {z}} _ {t}}{\alpha_ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \alpha_ {t}}, \\ \end{array}
$$
where the super-script  $(s)$  indicates these variables are for the student UNet. The student UNet is supposed to predict the noisy latent  $\mathbf{z}_{t^{\prime \prime}}$  from  $\mathbf{z}_t$  of the teacher with just one denoising step, namely,
$$
\mathbf {z} _ {t ^ {\prime \prime}} ^ {(s)} = \mathbf {z} _ {t ^ {\prime \prime}}. \tag {15}
$$
Replacing  $\mathbf{z}_{t^{\prime \prime}}^{(s)}$  with  $\mathbf{z}_{t^{\prime \prime}}$  in the final equation of Eq. (14), we arrive at the following loss objective,
$$
\mathcal {L} _ {\text {v a n i} \_ \text {d s t l}} = \varpi (\lambda_ {t}) | | \hat {\mathbf {x}} _ {t} ^ {(s)} - \frac {\mathbf {z} _ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \mathbf {z} _ {t}}{\alpha_ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \alpha_ {t}} | | _ {2} ^ {2}, \tag {16}
$$
where  $\varpi (\lambda_t) = \max (\frac{\alpha_t^2}{\sigma_t^2},1)$  is the truncated SNR weighting coefficients [33].
D Different Teacher Options for Step Distillation
It is non-trivial to decide the best teacher model to distill our final 8-step efficient UNet. In Fig. 4, we conduct several experiments to explore different teacher options. As straightforward choices, self-distillation from our 16-step efficient UNet or distillation from the 16-step SD-v1.5 baseline model can effectively boost the performance of our 8-step model. Additionally, we investigate whether stronger teachers can further boost performance by training a CFG-aware distilled 16-step SD-v1.5 model, as discussed in Sec. 4. We obtain significant improvements in CLIP scores, demonstrating the potential of employing better teacher models. We would like to mention that we also experiment with SD-v2 as the teacher model. Surprisingly, we observe much worse results. We attribute this to the different text embeddings used in SD-v1.5 and SD-v2 pipelines. Distillation between different infrastructures might be a possible future direction to explore.
We provide more generated images from our text-to-image diffusion model in Fig. 9. As an acceleration work for generic Stable Diffusion [4], our efficient model demonstrates a sufficient capability to synthesize various contents with high aesthetics, such as realistic objects (food, animals), scenery, and artistic and cartoon styles.
field flower, macro, bokeh, panoramic, and close to ground-at_2/1
dog illustrations
elephant illustration AI
large plate of delicious greasy fried chicken, with a side of dipping sauces, realistic and authentic. 4K resolution, spot lighting
A beautiful image of a cute animal surrounded by natural light, capturing their delicate beauty and charm
Grand Teton Landscape ar 16/9 -- hd
The eldritch landscape with mountains in the background, Sci Fantasy desktop wallpaper, painted, 4k, high detail, sharp
underground fantasy village dark 8k cinematic- ar_21/9-hd
Waiting in traffic, oil painting
batman in the rain, cinematic shot, beautiful night, photorealistic, concept art, high quality,  $8\mathrm{k}$
a beautiful hyperrealistic vase with a huge bouquet of celestial flowers, still life classical renaissance lighting at 3/4
2045 train station city landscape, conceptual, illustration, highly detailed, artwork, hyper realistic, in style of Ivan avizovsky
A photo of an astronaut riding a horse on mars
an aerial view of a forest in spiral with a giant tree in the center, referencing the movie Avatar, realistic render, detailed: ar 16/9
Figure 9: Example generated images by using our efficient text-to-image diffusion model.
concept art of a single bowl filled with a few moist freshly picked plums on a wooden table, digital painting, volumetric lighting, small scale, artistic, trending on artstation.
a beautiful picture of a dark distant thundercloud lit from below in a desert, intricate detail, impending doom, sunset, serene, volumetric lighting, and the clouds. 8 k's, hyperrealistic, digital art trending on artstation
an astronaut rest on mars in a beach chair, vibrant lighting, elegant, highly detailed, smooth, sharp, beautiful, geometric, trendy on artstation, full body, cinematic, artwork by borovikovsky
cream cheese smoked salmon bagel, splash art, movie still, cinematic lighting, dramaticOctopus shell shallow depth of field, beokh, anamorphics lens flare, 8 kJ, hyper detailed, 3.5 mm film grain