Photorealistic Text-to-Image Diffusion
Models with Deep Language
Understanding
Google Research, Brain Team
Xinjie Li
03/15/2023

Timeline

https://maciejdomagala.github.io/generative_models/2022/06/06/The-recent-rise-of-diffusion-based-models.html

Diffusion

Objective:
Generate Image X
from latent space Z
Z -> X

What are Diffusion Models? Lilian Weng. 2021

Diffusion

X_T == Z

Forward:Get the ground truth
Backward:
Z -> X (generation)Diffusion
Backward:
X_T -> X_0How?
We remove some noise in X_T to get X_T-1
Step by step, we remove all the noise and get X_0

So?
We have X_T now, all we need is the noise.
So?
U-Net: input-> X_T output->noise.

Diffusion
Forward:X_0 -> X_T

How?
Start distribution:
An image sample from:

Aim:
Define this process:😵we don’t like step by step: re-parametrizationDiffusion
Backward:
X_T -> X_0

How to train?
How?

Output:We remove some noise in X_T to get X_T-1
Step by step, we remove all the noise and get X_0

Ground Truth:So?
We have X_T now, all we need is the noise.

Loss:So?
U-Net: input-> X_T output->noise.

Diffusion
Training and inference

Denoising Diffusion Probabilistic Models. Jonathan Ho, Ajay Jain, Pieter Abbeel. 2020

GLIDEHow to add the additional information to Diffusion? : Guided Diffusion

Backward process of DDPM

Classifier-guided Diffusion
Classifier-free Diffusion
GLIDE(more data; larger model; more GPUs)
Diffusion models beat gans on image synthesis. Dhariwal, P. and Nichol, A. 2021.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. Alex Nichol et al. 2021

GLIDEcaption + mask area + super resolution
Diffusion models beat gans on image synthesis. Dhariwal, P. and Nichol, A. 2021.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. Alex Nichol et al. 2021

GLIDEDiffusion models beat gans on image synthesis. Dhariwal, P. and Nichol, A. 2021.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. Alex Nichol et al. 2021

DALL·E 2
CLIP + GLIDE
Training stage I: CLIP

Hierarchical Text-Conditional Image Generation with CLIP Latents. Aditya Ramesh et al. 2022

DALL·E 2
CLIP + GLIDE

Training stage II:Prior (latent space)

Hierarchical Text-Conditional Image Generation with CLIP Latents. Aditya Ramesh et al. 2022

DALL·E 2
CLIP + GLIDE

Training stage III:Decoder (GLIDE)

Hierarchical Text-Conditional Image Generation with CLIP Latents. Aditya Ramesh et al. 2022

DALL·E 2
CLIP + GLIDE
Inference

Hierarchical Text-Conditional Image Generation with CLIP Latents. Aditya Ramesh et al. 2022

DALL·E 2

Hierarchical Text-Conditional Image Generation with CLIP Latents. Aditya Ramesh et al. 2022

ImagenA pretrained, frozen T5-XXL model

GLIDE (dynamic)

Efficient U-net

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Chitwan Saharia et al. 2022

ImagenPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Chitwan Saharia et al. 2022

Imagen
DrawBench

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Chitwan Saharia et al. 2022

ImagenPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Chitwan Saharia et al. 2022

Future work
Latent Space: Stable Diffusion
Fine-grained Control: ControlNet, T2I-Adapter and Composer
Inversion: DreamBooth
Applications: Make-A-Video; Make-A-Story; Magic3D

Text-to-image Diffusion Model in Generative AI: A Survey
https://github.com/heejkoo/Awesome-Diffusion-Models