# Latent Diffusion Model

The Latent Diffusion Model (LDM) is a deep generative model architecture developed by CompVis (Computer Vision and Learning, LMU Munich). The Stable Diffusion 1.1 up to SD 2.1 were all based on the LDM architecture.

## Architecture

LDM uses a variational autoencoder (VAE), a U-Net, and optionally a Transformer, to generate images.

### Variational Autoencoder

To compress the image data, a variational autoencoder (VAE) is first trained on a dataset of images. The encoder part of the VAE takes an image as input and outputs a lower-dimensional latent representation of the image. This latent representation is then used as input to the U-Net. Once the model is trained, the encoder is used to encode images into latent representations, and the decoder is used to decode latent representations back into images.

### U-Net

The U-Net backbone takes the following kinds of inputs:
* A latent image array, produced by the VAE encoder. It has dimensions $({\text{channel}},{\text{width}},{\text{height}})$. For example, if it equals $(3,64,64)$, then it can be visualized as a 64-by-64 RGB image. However, the latent image is not intended to be visualized directly.
* A timestep-embedding vector, which indicates to the backbone about how much noise there is in the image. For example, an embedding of timestep $t=0$ would indicate that the input image is already noiseless, while $t=100$ would indicate a large amount of noise.
* A modality-embedding vector sequence, which indicates to the backbone about additional conditions for denoising. For example, in text-to-image generation, the text is divided into a sequence of tokens, then encoded by a text encoder, such as a CLIP encoder, before feeding into the backbone. As another example, an input image can be processed by a Vision Transformer into a sequence of vectors, which can then be used to condition the backbone for tasks such as generating an image in the same style.
Each run through the UNet backbone produces a predicted noise vector. This noise vector is scaled down and subtracted away from the latent image array, resulting in a slightly less noisy latent image. The denoising is repeated according to a denoising schedule ("noise schedule"), and the output of the last step is processed by the VAE decoder into a finished image.
Similar to the standard U-Net, the U-Net backbone used in the SD 1.5 is essentially composed of down-scaling layers followed by up-scaling layers. However, the UNet backbone has additional modules to allow for it to handle the embedding. As an illustration, we describe a single down-scaling layer in the backbone:
* The latent array and the time-embedding are processed by a ResBlock:
** The latent array is processed by a convolutional layer.
** The time-embedding vector is processed by a one-layered feedforward network, then added to the previous array (broadcast over all pixels).
** This is processed by another convolutional layer, then another time-embedding.
* The latent array and the embedding vector sequence are processed by a SpatialTransformer, which is essentially a standard pre-LN Transformer decoder without causal masking. In the cross-attentional blocks, the latent array itself serves as the query sequence, one query-vector per pixel. For example, if, at this layer in the UNet, the latent array has dimensions $(128,32,32)$, then the query sequence has $1024$ vectors, each of which has $128$ dimensions. The embedding vector sequence serves as both the key sequence and as the value sequence. In the self-attention blocks, the latent array serves as the query, key, and value.
In pseudocode,
The detailed architecture may be found in.
This architecture does not have self-attention mechanisms, although the cross-attention mechanisms can be modified into self-attention mechanisms.

### Training

The LDM is trained by using a Markov chain to gradually add noise to the training images. The model is then trained to reverse this process, starting with a noisy image and gradually removing the noise until it recovers the original image. More specifically, the training process can be described as follows:
* Forward diffusion process: Given a real image $x_{0}$, a sequence of latent variables $x_{1:T}$ are generated by gradually adding Gaussian noise to the image, according to a pre-determined "noise schedule".
* Reverse diffusion process: Starting from a Gaussian noise sample $x_{T}$, the model learns to predict the noise added at each step, in order to reverse the diffusion process and obtain a reconstruction of the original image $x_{0}$.
The model is trained to minimize the difference between the predicted noise and the actual noise added at each step. This is typically done using a mean squared error (MSE) loss function.

### Image Generation

Once the model is trained, it can be used to generate new images by simply running the reverse diffusion process starting from a random noise sample. The model gradually removes the noise from the sample, guided by the learned noise distribution, until it generates a final image.