Boosting Latent Diffusion with Perceptual Objectives
Authors: Tariq Berrada
Pietro Astolfi
Melissa Hall
Marton Havasi
Yohann Benchetrit
Adriana Romero-Soriano
Karteek Alahari
Michal Drozdzal
Jakob Verbeek
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion
flows
latent diffusion
LDM
latent generative models
T2I
image generation
generative models.
Primary area: generative models
TL;DR: Latent diffusion models do not take into account the structure of the latent space in which they operate, doing so boosts sample quality.

Abstract: Latent diffusion models (LDMs) power state-of-the-art high-resolution generative image models. LDMs learn the data distribution in the latent space of an autoencoder (AE) and produce images by mapping the generated latents into RGB image space using the AE decoder. While this approach allows for efficient model training and sampling, it induces a disconnect between the training of the diffusion model and the decoder, resulting in a loss of detail in the generated images. To remediate this disconnect, we propose to leverage the internal features of the decoder to define a latent perceptual loss (LPL). This loss encourages the models to create sharper and more realistic images. Our loss can be seamlessly integrated with common autoencoders used in latent diffusion models, and can be applied to different generative modeling paradigms such as DDPM with epsilon and velocity prediction, as well as flow matching. Extensive experiments with models trained on three datasets at 256 and 512 resolution show improved quantitative -- with boosts between 6% and 20% in  FID -- and qualitative results when using our perceptual loss.

Published as a conference paper at ICLR 2025
BOOSTING LATENT DIFFUSION WITH PERCEPTUAL OBJECTIVES
Tariq Berrada Ifriqi'? Pietro Astolfi', Melissa Hall', Marton Havasi', Yohann Benchetrit' Adriana Hamero-Soriano---e Karteek Alahari', Michal Drozdzal
Jakob Verbeek
FAIR at Meta, Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France McGill University, Mila, Quebee Al institute, Canada CIFAR Al chair
tarigberrada eeeerradddammaa..mmmmm
W/O LPL
with LPL
Aitten with pawe
stone wall leads to solinory nee Statue of boautffu molden Ruddon the pink, flovers.
Plach and cream brides bouquer for weddingg
Figure I: Samples from models trained with and without our latent perceptual loss on CC12M. Samples from our model with latent perceptual loss (bottom) have more detail and realistic textures.
ABSTRACT
Latent diffusion models (LDMs) power state-of-the-an high- resolution generative image models. LDMs learn the data distribution in the latent space of an autoencoder (AE) and produce images by mapping the generated latents into RGB image space using the AE decoder. While this approach allows for efficient model training and sampling, it induces disconnect between the training of the diffusion model and the decoder, resulting in loss of detail the generated images. To remediate this disconnect, we propose to leverage the internal features of the decoder to define a latent perceptual loss (LPL). This loss encourages the models create sharper and more realistic images. Our loss be seamlessly integrated with common autoencoders used in latent diffusion models, and can be applied to different generative modeling paradigms such DDPM with epsilon and velocity prediction, as well as flow matching. Extensive experiments with models trained on three datasets at 256 and 512 resolution show improved quantitative with boosts between 6% and 20% in FID and qualitative results when using our perceptual loss.
INTRODUCTION
Latent diffusion models (LDMs) (Rombach et al., 2022) have enabled considerable advances in image generation, and elevated the problem of generative image modeling level where it has

---

Published as a conference paper at ICLR 2025
become available as technology to the public. A critical part to this success is to define the gen- erative model in the latent space of an autoencoder (AE), which reduces the resolution of the repre- sentation over which the model is defined, thereby making it possible to scale diffusion methods to larger datasets, resolutions, and architectures than original pixel-hased diffusion models (Dhariwal al., 2015).
& Nichol, 2021; Sohl-
To train an LDM, all images are first projected into a latent space with the encoder of a pre-trained autoencoder, and then, the diffusion model is optimized directly in the latent space. Note that when learning the diffusion model the AE decoder is not used - the diffusion model does not receive any training feedback that would ensure that all latent values reachable by the diffusion process decode to high quality image. This training procedure leads to disconnect between the diffusion model and the AE decoder, prompting the LDM to produce low quality images that oftentimes lack high frequency image components. Moreover, we note that the latent spaces of pre-trained LDM's s autoencoders tend to be highly irregular, the sense that small changes in the latent space can lead to large changes in the generated images, further exacerbating the autoencoder-diffuuse disconnect problem.
In this work, we propose to alleviate this autoencoder-diffusioo disconnect and propose to include the AE decoder in the training objective of LDM. In particular, we introduce latent perceptual loss (LPL) that acts on the decoder's intermediate features to enrich the training signal of LDM. This is similar to the use of perceptual losses for image-to-image translation tasks (Johnson et al., 2016; Zhang et al., 2018), but we apply this idea in the context of generative modeling and use the fea- ture space of the pre-trained AE decoder rather than that of an external pre-trained discriminative network. Our latent perceptual loss results in sharper and more realistic images, and leads to better structural consistency than the baseline see Figure 1. We validate LPL on three datasets of dif- ferent sizes- the commonly used datasets ImageNet-Ik 1k (IM data points) and CC12M (12M data points), and additionally a private dataset S320M (320M data points) as well as three generative models formulation DDPM (Ho et al., 2020) with veloeity and epsilon prediction, and conditional flow matching model (Lipman et al., 2023). In our experiments, we report standard image genera- tive model metries such as FID (Heusel et al., 2017), CLIPScore (Hessel et al., 2021), as well as Precision and Recall (Sajjadi et al., 2018; Kynkäänniemi al., 2019). Our experiments show that the use of LPL leads to consistent performance boosts between 6% and 20% in terms of FID. Our qualitative analysis further highlights the benefits of LPL, showing images that are sharp and contain image details.
In summary, our contributions are:
We identify a slight disconnect between latent and pixel-space diffusion which can lead to subop- timal results when training latent diffusion and flow models.
We propose the latent perceptual loss (LPL), a perceptual loss variant leveraging the intermediate feature representation of the autoencoder's decoder.
We present extensive experimental results on the ImageNet-Ik, CC12M, and S320M datasets, demonstrating the benefits of LPL in boosting the model's quality by 6% to 20% in terms of FID. We show that LPL. is effective for variety of generative model formulations including DDPM and conditional flow matching approaches.
RELATED WORK
Diffusion models. The generative modeling landscape has been significantly impacted by diffu- sion models, surpassing previous ofate-of-the-art GAN-based methods (Brock et al., 2019; Karras et al., 2019; 2020; 2021). Difffusion models offer advantages such as more stable training and better scalability, and were successfully applied to a wide range of applications, including image gener- ation (Chen et al., 2024; Ho et al., 2020), video generation (Ho et al., 2022b; Singer et al., 2023), music generation (Levy et al., 2023; San Roman et al., 2023), and text generation (Wu et al., 2023). Various improvements of the framework have been proposed, including different schedulers (Lin et al., 2024; Hang & Gu, 2024), loss weights (Choi et al., 2022; Hang et al., 2023b), and more re- cently generalizations of the framework with flow matching (Lipman et al., 2023). In our work evaluate the use of our latent perceptual loss in three different training paradigms: DDPM under noise and velocity prediction, well based training with the optimal transport path.
2

---

Published as a conference paper at ICLR 2025
Latent diffusion. Due to the iterative nature of the reverse diffusion process, training and sam- pling diffusion models is computationally demanding, in particular at high resolution. Different approaches have been explored to generate high- high-resolution content. For example, Ho et al. (2022a) used a cascaded approach to progressively add high-resolution details, by conditioning on previously generated lower resolution images. A more widely adopted approach is define the generative model in the latent space induced by a pretrained autoencoder (Rombach et al., 2022), as previously explored for discrete autoregressive generative models (Esser al., 2021). Different architectures have been explored to implement diffusion models in the latent space, including convolutional UNet- based architectures (Rombach et al., 2022; Podell et al., 2024), and more recently transformer- -based ones (Peebles & Xie, 2023; Chen et al., 2024; Gao et al., 2023; Esser et al., 2024) which show better scaling performance. Working in a lower-resolution latent space accelerates training and inference, but training models using a loss defined in the latent space also deprives them from matching high- frequency details from the training data distribution. Earlier approaches to address this problem include the use of a refiner model (Podell et al., 2024), which consists of second diffusion model trained on high-resolution high-quality data that is used to noise and denoise the initial latents, simi- lar to how SDEdit works for image editing (Meng al., 2022). Our latent perceptual loss addresses this issue in an orthogonal manner by introducing loss defined across different layers of the AE decoder in the latter stages of the training process. Our approach avoids the necessity of training on specialized curated data (Dai et al., 2023), and does not increase the computational cost of inference.
Perceptual losses. The use of internal features of fixed, pre-trained deep neural network com- pare images or image distributions has become common practice as they have been found to correlate to some extent with human judgement of similarity (Johnson et al., 2016; Zhang et al., 2018). An example of this is the widely used Fréchet Inception Distance (FID) to assess generative image mod- els (Heusel et al., 2017). Such "perceptual" distances have also been found to be effective as a loss to train networks for image-to-image tasks and boost image quality as compared to using simple & or l2 reconstruction losses. They have been used to train autoencoders (Esser et al., 2021), models for semantic image synthesis (Isola et al., 2017; Berrada al., 2024) and super-resolution (Suvorov et al., 2022; Jo et al., 2020), and to assess the sample diversity of generative image models (Schön- feld al., 2021; Astolfi et al., 2024). In addition, recent works propose variants that do not require pretrained image backbones (Amir & Weiss, 2021; Czolbe et al., 2020; Veeramacheneni et al., 2023). An et al. (2024); Song al. (2023) employed LPIPS metric function in pixel space train cas- caded diffusion models and consistency models, respectively. More closely related to our work, Kang et al. (2024) used a perceptual loss (E-Latem:LPPPS) defined in latent space distill LDMs to conditional GANS, but used a separate image classification network trained over latents rather than the autoencoder's 's decoder to obtain the features for this loss. Lin & Yang (2024) added a perceptual loss
to train LDMs that is defined over the features of the denoiser network in the case of UNet architectures. However, they found this loss to be detrimental when using classifier- free guidance for inference. In summary, compared to prior work on perceptual losses, our work is different in that (i) LPL is defined over the features of the decoder- - that maps from latent space to RGB pixel space, rather than using network that takes RGB images as input and (ii) our LPL can be regardlessly applied to train both latent diffusion and flow models and has no architecture constraint in the denoiser model.
3 USING THE LATENT DECODER TO DEFINE A PERCEPTUAL LOSS
In this section, we analyze the impact of the decoder-diffusion disconnect on the LDM training, and then, we follow with the definition of our latent perceptual loss.
3.1 LATENT DIFFUSION AND THE & OBJECTIVE
We use Fg to refer to an autoencoder that consists of two modules. The encoder, Fý, maps RGB images Xo E TR. x3 to latent representation 2u E #//dd//xxCC where d is the spatial down- scaling factor, and C the channel dimension of the autoencoder. The decoder, Fy. maps from the latent space to the RGB image space. In LDM, the diffusion model, De, with parameters θ is de- fined over the latent representation of the autoencoder. We follow a tyyiccal setting, see e.g. (Peebles & Xie, 2023; Chen et al., 2024; Rombach et al., 2022), where we use a fixed pre-trained autoencoder with downsampling factor of d 8 and channel capacity of C 4.
3

---

Published as a conference paper at ICLR 2025
Training Objective. The diffusion formulation results in objective function that is lower-bound on the log- log-likelihood of the data. In the DDPM paradigm (Ho et al., 2020), the variational lower bound can be expressed as sum of denoising score matching losses (Vincent, 201 I), and the objec- tive function can be written as L = Ee Lt, where Lt = Dxa. [q (x--1 |x,, xo) lIpo (xx-1[x+]] =
Exo.e:t [T- - ||e - (x,, =x( Ho et al. (2020) observed that disregarding the time step specific weighting resulted in improved sample quality, and introduced a simplified noise reconstruction objective, known as epsilon prediction, where the objective is the average of the MSE loss between the predicted noise and the noise vector added to the image, Csimlee Exo,,t |||e - co (x,, +)] The underlying idea is that the better the noise estimation, the better the final sample quality. An equivalent way to interpret this objective is through reparameteriza- tion of the target to the original latents, Lsimple = E, de= Exo.e,t [[|x0- - Ex( 1;0) [²]. where
Lo(x,, t; 0) = (x, - o,,r)/ ar and A = =/00?.
We note that the presence of lg in the LDM objective has two important implications. First, the & norm treats all pixels in the latents as equally important and disregards the downstream structure induced by the decoder whose objective is to reconstruct the image from its latents. This is problem- atic because the autoencoder's latent space has a non unifornfo structure and is not equally influenced by the different pixels in the latent code. Thus, optimizing the lz distance in the diffusion model latent space could be different from optimizing the perceptual distance between images. Second, while an 22 objective is theoretically justified in the original DDPM formulation, generative models trained with an l2 reconstruction objective have been observed to produce blurry images, as is the case e.g. for VAE models (Kingma & Welling, 2014). Such an effect can lead to exposure bias in diffusion (i.e. denoiser input mismatch between training and sampling), which has been studied through the lens of the sampling algorithm but not the training objective Ning et al. (2024). The problem of blurry images due to &2 reconstruction losses has been addressed through the use of perceptual losses such as LPIPS (Zhang et al., 2018), which provide a significant boost to the image quality in settings such as autoencoding (Esser et al., 2021), super-resolution Ledig et al. (2017) and image-to-ing totittggeeeaeao generative models (Isola et al., 2017; Park et al., 2019).
To address these two implications of the latent &2 optimization, we design a new loss, namely Latent Percetual Loss (LPL), that directly tackles the mismatch between the structures of the latent and the pixel space by matching the target and the predicted latents when partially decoded with the AE decoder. Compared to existing perceptual losses, e.g., LPIPS, that usually rely on features from pre-trained classifier networks, the use of AE decoder features is not standard. However, for LDMs training. the AE decoder provides a perceptually meaningful signal as the intermediate features of the decoder, being closer to pixel space. have a structure more similar to the pixel space. Moreover, these features have higher resolution than the diffusion latent features, which helps in modeling high-level details that reduce blurriness of the predicted image.
3.2 LATENT PERCEPTUAL LOSS
We propose a loss function that operates on the features at different depths in the autoencoder's decoder, Fd. Let 2, Zg + ofe, be a noisy sample in the diffusion model latent space at time 4, and 20 = (z,- D(z, +))/ar /a, the corresponding estimated noise-free latent at time = 0. To enhance the accuracy of the reconstruction process in image space, we propose augmenting the diffusion objective with penalty term that encourages the predicted forward process distribution to match the reconstructions 2n at a lower level of compression. This is achieved by introducing an image-level projection penalty on the forward process, which can be expressed
2.e0 = By [DKL (q(xc-1/xx, xn) pe (
(1)
In Appendix 1. we show that such penalty, under certain conditions, can be approximated by computing a pairwise distance between the outputs of the projection decoder, which maps latents into samples in image-space. Guided by this result, our loss term should resemble reconstruction term between the original image and the predicted/denoised image obtained from the decoder of the autoencoder. However, this does not address the problem of blurriness exposed in the previous section, which is why we opt for a loss structure that is similar to LPIPS, where the loss term is defined over the intermediate feature spaces of the decoder.
4

---

Published as a conference paper at ICLR 2025
Li co
mil
00.0
Figure 2: Overview of our approach. (a) Latent diffusion models compare clean latents and the predicted latents. (b) Our LPL acts the features of the autoencoder's decoder effectively aligning the diffusion process with the decoder. F Fg. autoencoder encoder and decoder, De: denoiser network, CN: cross normalization layer. OD: outlier detection.
Table 1: Summary of the formula for the estimate of the clean image correspond- ing the different formulations. Using the following parameterization, vt,x, = ax or6t.
Framework
DDPM-cx
(xi D(x,, 9))
DDPM-18g vig
Vaz
VI
Hlw-OT
-;;
We compute two sets of hierarchies of decoder features, (Φi) and, (6) by decoding both the original, 2o, and estimated latents, 2o (where for brevity we drop the dependence of 2n on l):
(Q...., oz (F (20))
2 $ d1 (Fg.) (20))
Using these intermediate features, we can define our training objective. Our LLPL, is a weighted sum of the quadratic distances between the feature representations the different decoding scales, obtained after normalization:
C.PL B.EET.co-N SansT
Wi
( o (01, -
(3)
where &f is the standardized version of Φi across the channel dimension, (t,c( is a binary map masking the detected outliers in the feature map di.e. wi is depth- weighting and C the channel dimensionality of the feature tensor. Note that we better explain these terms later in this section. Moreover, to reduce both LPL. computational complexity and memory overhead, we only apply our loss for high signal-to-noies ratios (SNR). In particular, we impose hard threshold and only apply the loss if the SNR higher than it oa Sr. (a.). The LPL loss is applicd in conjunction with the standard diffusion loss, resulting in the training objective Lix LpiF + WLPL LLPL-
Depth-specific weighting. Empirically, we find the loss amplitude at different decoder layers to differ significantly it grows with factor of two when considering layers with factor two increase resolution. To balance the contributions from different decoder layers, we therefore weight them by the inverse of the upscaling factor w.n.f. the first layer, iee. wi = 2-n//r where ri the resolution of the (-th layer.
Normalization. Since the features the decoder can have significantly varying statistics from each other, we follow Zhang et al. (2018) and normalize them per channel so that the features in every channel every layer are zero mean and have unit variance. However, normalizing the induce
feature maps corresponding the original and denoised latents with different statisties nonzero gradients even when the absolute value has been correctly predicted. To obtain a coherent normalization, the feature statistics from the denoised latents normalize both tensors.
5

---

Published as a conference paper at ICLR 2025
3.3 GENERALIZATION FOR LATENT GENERATIVE MODELING
While the bulk of our experiments have been conducted on models trained under DDPM (Ho et al., 2020) for noise prediction, we can generalize our method to different frameworks such as diffusion with velocity prediction (Salimans & Ho, 2022) and flow matching (Lipman et al., 2023). To do this, the only requirement is to be able to estimate the original latents from the model predictions. Under general frameworks such DDPM and flows, can write the forward equation in the form vt, 22 = Xx + o,e,, where a and of are increasing (resp. decreasing) functions of t. In Table 1, provide summary for these different formulations.
4 EXPERIMENTAL EVALUATION
In this section, first present experimental setup, and then go present our main results, as well as qualitative results and a number of ablation studies.
4.1 EXPERIMENTAL SETUP
Datasets. We conduct an extensive evaluation on three datasets of different scales and distributions: ImageNet-Ik Ik (Deng et al., 2009). CC12M (Changpinyo et al., 2021), and S320M: large internal dataset of 320M stock images. We note that for both ImageNet- Ik and CC12M. human faces were blurred to avoid training models on identifiable personal data. For both CC12M and S320M, we recaption the images using Florence-2 (Xiao al., 2024) obtain captions that more accurately describe the image content. For each of these datasets, we conduct evaluations at both 256x256 and 512 x 512 image resolution.
Architectures. All experiments are performed using the Multi-modal DiT architecture from Esser al. (2024). We downscale the model size be similar to Pixart-o (Chen al., 2024) and DiT- XL/2 (Peebles & Xie, 2023), which corresponds to 28 blocks with a hidden size of 1,536, amounting to total of 796M parameters. For ImageNet-IK 1k models we condition on class labels, while for the other datasets we condition on text prompts. For our main results, we perform our experiments using the asymetric autoencoder from Zhu et al. (2023). For ablation studies, we revert to the lighter autoencoder from SDXL (Podell et al., 2024).
Training and sampling. Unless specified otherwise, we follow the DDPM-c training paradigm (Ho et al., 2020), using the DDIM (Song et al., 2021) algorithm with 50 steps for sampling and classifier-free guidance scale of X = 2.0 (Ho & Salimans, 2021). Following Podell et al. (2024), we use quadratic scheduler with Sstant 0.00085 and Bend 0.012. For the flow experiments, the conditional OT probability path (Lipman et al., 2023) with the mode sampling with heavy tails paradigm from Esser al. (2024). Under this paradigm, the model is trained for velocity prediction and evaluated using the Euler ODE solver. For all velocity models, zero terminal SNR schedule is enforced following Lin et al. (2024).
use
Similar to Chen et al. (2024), we pre-train all models at 256 resolution on the dataset of interest for 600k iterations. We then enter a second phase of training, in which we optionally apply our perceptual loss, which lasts for 200k iterations for 256 resolution models and for 120k iterations for models at 512 resolution.
When changing the resolution of the images, the resolution of the latents changes by the same factor, keeping the same noise threshold T, yields inconsistent results across resolutions. To ensure consistent behavior, we follow Esser et al. (2024), and scale the noise threshold similarly to how the noise schedule is scaled in order to keep the same uncertainty per patch. In practice, this amounts to scaling the threshold by the upscaling factor. The kernel sizes for the morphological operations in the outlier detection algorithm are also scaled to cover the same proportion of the image.
Metrics. To evaluate our models, we report results in terms of FID (Heusel et al., 2017) to assess image quality and to what extent the generated images match the distribution of training images, and CLIPScore (Hessel et al., 2021) to assess the alignment between the prompt and the generated image for rext-conditioned models. In addition, we report distributional metrics precision and recall (Saj- jadi al., 2018) well density and coverage (Naeem al., 2020) to better understand effects on image quality (precision/density) and diversity (recall/coverage). We evaluate metrics with re- spect to ImageNet- k and, for models trained on CC12M and S320M, the validation set of CC12M.

---

Published as a conference paper at ICLR 2025
weo LPL
with LPL
Goot the toker place within with Backgroudd.
Mother of momment day.
The building behind
These blueberry mudfar fugfy
Mosill in car thow-
bites of
Figure 3: Samples from models trained with and without our latent perceptual loss on S320M. Samples from the model with perceptual loss (bottom row) show more realistic textures and details.
Pre-training
Post-training
Results
Table 2: Impact of our perceptual loss for models trained on differ- ent datasets and resolutions for DDPM-c model. All models use the same ImageNet-256 pretraining for 600k iterations before perform- ing coparing the effect LPL during post-training. Using LPL boosts FID and CLIP score for all datasets and resolutions considered.
Res.
Iters
Res.
Iters
LPI.
FID (1)
CLP (t)
x
2.98
256
200k
ImageNet-IA
256
600k
/
2.72
x 4.88
3.79
256
200k
x
7.81
25.06
6.22
25.12
CC12M
256
600k
x 8.79
24.88
120k
7.27
25.12
S320M
256
600k
512
120k
x 8.81
24.39
8.30
24.41
For FID and other distributional metrics, we use the evaluation datasets the reference datasets and compare an equal number of synthetic samples. For CLIPScore, evaluation datasets and the corresponding synthetic samples. Following previous works (Rombach et al., 2022; Peebles & Xie, 2023), we use guidance scale of 1.5 for resolutions of 256 and 2.0 for resolutions of 512, which we also found to optimal for our baseline models trained without LPL.
the prompts of the
4.2 MAIN RESULTS
LPL applied across different datasets. Table we consider the impact the LPL on the FID and CLIPScore for models trained on the three datasets and two resolutions. We observe that the LPL loss consistently improves both metrics across all three datasets. Most notably, FID is improved by 0.91 points ImageNet-11 1 512 resolution and by 1.52 points CC12M at 512 resolution. The CLIPScore is also improved for both resolutions on CC12M, by 0.06 and 0.24 points respectively. Similarly, for the S320M dataset, we observe that FID is improved by 0.51 points while CLIP score improves (marginally) by 0.02 points. Samples of models trained with and without LPL on CC12M and S320M are shown in Figure and Figure 3, respectively.
Generalization to other frameworks. We showcase the generality of the LPL by applying it to different generative models, experimenting with DDPM for both epsilon and velocity prediction, and flow matching with optimal transsort (OT) path similar to Esser et al. (2024). In Table 3 we report experimental results for models trained on ImageNet- 1 at 512 resolution. The DDPM-based models perform very similar (except perhaps for FID, where they differ by 0.16 points), and we we find significant improvements across all metries other than density when using LPL. Density remains similar to the baseline for DDPM models, but improves from 1.14 to 1. 1.29 for the Flow-OT model, where all metrics are improved relative to the DDPM trained ones. We posit that this is due to the mode sampling scheme in (Esser et al., 2024). which emphasizes middle timesteps that could better control the trajectories of the flow path towards having more diversity and not improving the quality (precision/density), Hence, applying LPI. to Flow-OTT solve this by considerably boosting

---

Published as a conference paper at ICLR 2025
DDPM-
DDPM
OF
Table 3: Effect of LPL on ImageNet-Ile models at 512 res- olution trained with different methods. We observe con-
RID
4.88
5.79
4.72
5.84
1.54
3.61
Conrgge
0.50
0..82
0.80
0.83
0.82
0.85
sistent improvements on all metrics when incorporating the
Deuilty (1)
1.14
1:15
1.15
1.11
1.14
1.29
Precision (1)
0.74
0.77
0.73
0.78
0.73
0.70
LPL, except for density metric for which we observe very
Recall 112
0.49
0.51
0:49
0.50
0.32
0.54
slight degradation when using DDPM training.
100
2X
106
30
005
30
(a) FDuL)) F (Doss)
(b) F(Duse)
F(Dret)
(c) F F (Draa)
Figure 4: Power spectrum of real and generated images. Difference in (log) power spectrum between image generated with and without LPL. Using LPL strenghtens frequencies at the extremes (very low and very high).
quality. Notably, considering DDPM baselines, LPL provides boost as significant as the one provided by using flow matching (scores of DDPM w/ LPL in 2nd and 4th columns are on par or better than Flow-OT w/o LPL in 5th column). Moreover, the provided boost is orthogonal to the training paradigm, leading overall best results when using LPI. with the flow model.
Frequency analysis. While the metrics above provide useful information on model performance, they do not specifically provide insights in terms of frequencies at which using LPL is more effective at modeling data than the baseline. To provide insight on the effect our perceptual loss w.r.. the frequency content of the generated images, compare the power spectrum profile of images generated with a model trained with and without LPL on CC12M 512 resolution as well as a set of real images from the validation set.
In Figure 4, we plot the difference between log-power spectra between the three image sets. The left-most panel clearly shows the presence of more high frequency sig- nal in the generated images when using LPL to train the model, confirming what has been observed in the qual- itative examples of Figure 1 and Figure 3. Moreover, the very lowest frequencies are also strengthened in the samples of the model with LPL. We posit that using the LPL makes it easier to match very low frequencies as they tend be encoded separately certain channels of the de- coder. In Figure 5, we report the error when comparing the power-spectrum of synthetic images and real images, averaged across the validation set of CC12M. For this, we compute the average of the power spectrum across a set of 10k synthetic images from each model and the reference images for the validation set of CC12M. Our experiments indicate that the model trained with LPI. is consistently more accurate in modeling high frequencies (f > 150), the expense of somewhat larger error at middle fre- quencies (75 < f < 150).
L.PL.
W/o. LPI.
104
10a
0 50 100 150 200 250 300 350
Frequency f
Figure 5: Frequency comparison. We compare the power spectrum of the im- ages obtained with or without LPL with real reference images from the valida- tion set of CC12M.
4.3 ABLATION RESULTS
LPL depth. Using decoder layers to compute our perceptual loss comes with increased computa- tional and memory costs. We therefore study the effect of computing our perceptual loss using only subset of the decoder layers, well baseline using the RGB pixel output of the decoder. We progressively add more decoder features, so the model with five blocks contains features from the first up the fifth block. From the results in Figure 7, we find that earlier blocks do not significantly
8

---

Published as a conference paper at ICLR 2025
232
1.10
2.00
0.00
Figure 6: Ablation study on the impact of the noise threshold Ta We report FID, coverage, density, precision and recall. The dashed line corresponds to the baseline without LPL, note the logarithmic scaling of the noise threshold on the horizontal axis.
Bunelin
LPL
7.0
40
F1De512
6.5
FID0512
FID@512
6.0
5.5
5.0
4.5
5.0
3.5
5
10
25
50
100
250
0
2
Hlacks used
NFE
Figure 7: Exploration of LPL depth. Influence of decoder blocks used in LPL on FID, corresponds to not using LPL. Disk radius shows GPU memory usage: w/o LPL=64.9 GB, LPL. blocks=83.4 GB.
Figure 8: Impact of LPL for different number of sampling steps. With higher numbers of sampling steps, the difference be- tween the baseline and the model trained with LPL inereases.
Figure 9: Influence of the LPL loss weight on model perfor- mance. The curve shows a sharp decrease FID hefore going back up for larger weights.
improve FID and can even negatively impact performance. Deeper layers, on the other hand, sig- nificantly improve the performance. The most significant gains are obtained when incorporating the third and fourth decoder layers which both improve the FID by more than one point w.zf. the model incorporating one block less. Finally, the last decoder layer improves the FID only marginally and could be omitted to reduce resource consumption. We perform an additional ablation where the loss operates directly on the RGB image space (without using internal decoder features), which results in degraded performance compared to the baseline while inducing considerable memory overhead.
Feature normalization. Before computing our perceptual loss, we normalize the decoder features. We compared normalizing the features of the original latent and the predicted one separately, or normalizing both using the statisties from the predicted latent. Our experiment conducted on ImageNet- 1k 512 resolution. While the model trained with separately normalized latents results in a slight boost of FID (4.79 vs. 4.88 for the baseline w/o LPL), the model trained with shared normalization statistics leads much more significant improvement and obtains FID of 3.79.
SNR threshold value. We conduct an experiment on the influence the SNR threshold which determines which time steps perceptual loss is used for training. Lower threshold values cor- respond to using LPL for fewer iterations that are closest to the noise-free targets. We report results across several metrics in Figure and illustrated with qualitative examples in the supplementary material in Figure 20. We find improved performance over the baseline without LPL for all metrics and that the best values for each metric are obtained for a threshold between three and six, except for the recall which is very stable (and better than the baseline) for all threshold values under 20.
Reweighting strategy. We compare the performance when using uniform depth-specific weights to combine the contributions from different decoder layers in the LPL. We find that using depth- specific weights results in significant improvements in terms of image quality w.t.. using uniform weights. While the depth-specific weights achieve an FID of 3.79. the FID obtained using uni- form weights 4.38. Hence, while both strategies improve image quality over the baseline (which achieves an FID of 4.88), reweighting the layer contributions be approximately similar further boosts performance and improves FID by 0.59 points.
LPL and convergence. As the LPL loss adds non-negligible memory overhead, by having to backpropagate through the latent decoder, is interesting to explore at which point in training should be introduced. We train models ImageNet-1k 1k at 512 resolution with different durations of the post-traine posst-raainig stage. We use an initial post- cost-training phase - of zero, 50k, or 400k iterations in which I. IPL. is not used, followed by another 120k iterations in which either apply IPL. not.

---

Published as a conference paper at ICLR 2025
Initial -rait itens
50k
400%
Table Effect of post-training with LPL. In each col-
FID
-0.58
-0.78
-0.97
umn, we report the difference in metrics after post-training
A coverage (1)
+44.9
4-3.51
+3390
for 120k iterations with or without LPL. All metries im-
density (1)
+0.14
+.012
+0.21
A precision (1)
4-4.01
4-4.55
45.89
prove when adding LPL in the post-training phase.
recall (1)
+1.99
+2.32
+4.22
The results in Table 4 indicate that in each case I.I.. improves all metrics and that the improvements larger when the model has been trained longer and is closer to convergence (except for the coverage metric where we see the largest improvement when post-training for only 120k iterations). This suggests that better models (ones trained for longer) benefit more from our perceptual loss.
Influence on sampling efficiency. We conduct an experiment to assess the influence of the percep- tual loss on the sampling efficiency. To this end, we sample the ImageNer 512 model with different numbers of function evaluations (NFE) then check the trends for the baseline and the model trained with our method. For this experiment, we use DDIM algorithm. Results are reported on Figure 8, where we find that for very low numbers of function evaluations, both models perform similarly. The improvement gains from the LPI. loss start becoming considerable after 25 NFEs, where we observe a steady increase in performance gains with respect to the number of function evaluations up to 100, afterwards both models stabilize at a point where the model trained using LPL achieves an improvement of approximately 1.1 points over the baseline.
Impact on EMA. Since the LPL has the effect of in- creasing the accuracy of the estimated latent during every timestep, it reduces fluctuations between successive iter- ations of the model during training. Consequently, when training with LPL the EMA momentum can be reduced to obtain optimal performance. In Figure 10 we report the results of a grid search over the momentum parameter TEMA- We find that the model trained with LPL achieves better results when using slightly lower momentum than the baseline. From the graph, it's clear that better FID is obtained closer to "EMA 0.99975 for the IPI. model, which corresponds to half-life of approximately 2750 iterations, while the non-LPL model achieves its opti- mal score at "Iema = 0.9999 corresponding to a half life of 6930 iterations, more than twice as much as the LPL model, thereby validating our hypothesis.
Mrh LPM.
L.M.
4.5
4.0
3.0
0.99
0,999
amps
0.sans
appn
esppss
0.990ns
o 0.999999
YEMA
Figure 10: Impact of EMA decay rate. Training with LPL is more stable, and allows for a smaller decay parameter.
Relative weight. We conduct grid search over different values for the weight of the LPL loss WLPL- We report FID after training for 120k iterations at 512 resolution, all models are initialized from the same 256 pretrained checkpoint. Our results are reported in Figure 9. Introducing LPL. sharply decreases FID for lower weights before going back at higher weights. We find the model to achieve the best FID for WLPL 3.0 which roughly corresponds to fifth of the relative contribution to the total loss.
5 CONCLUSION
In this work, we identified a disconnect between the decoder and the training of latent diffusion models, where the diffusion model loss does not receive any feedback from the decoder resulting in perceptually non-optimal generations that oftentimes lack high frequency details. To alleviate this disconnect we introduced a latent perceptual loss (I.P..) that provides perceptual feedback from the autoencoder's decoder when training the generative model. Our quantitative results showed that the LPL is generalizable and improves performance for models trained on a variety of datasets, image resolutions, as well as generative model formulations. We observe that our loss leads to improve- ments from 6% up to 20% in terms of FID. Our qualitative analysis show that the introduction of leads models that produce images with better structural consistency and sharper details com- pared to the baseline training. Given its generality, we hope that our work will play an important role in improving the quality of future latent generative models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dan Amir and Yair Weiss. Understanding and simplifying perceptual distances. In CVPR, 2021.
Jie An, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Zicheng Liu, Lijuan Wang, and Jiebo Luo. Bring metric functions into diffusion models. In IJCAI, 2024.
Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adri- ana Romero Soriano, and Michal Drozdzal. Consistency-diversstt-relll Pareto fronts of con- ditional image generative models. arXiv preprint, 2406. 10429, 2024.
Tariq Berrada, Jakob Verbeek, Camille Couprie, and Karteek Alahari. Unlocking pre-trane pre-trained image backbones for semantic image synthesis. In CVPR, 2024.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
Ricky T. Q. Chen. torchdiffeq.
torchdiffeq, 2018.
URL
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In CVPR, 2022.
Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Igel. A loss function for generative neural networks based on Watson's 's perceptual model. In NeurlPS, 2020.
Xiaoliang Dai, Hou, Chih- Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Rade- novic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh. Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Va- jda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint, 2309.15800 2023.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
ImageNet: A large-scale
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
Patrick Esser, Robin Rombach, and Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICMI. 2024.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In ICCV, 2023.
Tiankai Hang and Shuyang Gu. Improved noise schedule for diffusion training. arXiv preprint, 2407.03297, 2024.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) pp. 7441-7451, October 2023a.
11

---

Published as a conference paper at ICLR 2025
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via Min-SNR weighting strategy. In ICCV, 2023b.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In NeurlPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research. 23(47), 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In NeurlPS, 2022b.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Younghyun Jo, Sejong Yang, and Seon Joo Kim. Investigating loss functions for extreme super- resolution. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun- Yan Zhu, and Taesung Park. Distilling diffusion models into conditional GANs. In ECCV, 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In CVPR, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurlPS, 2021.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurlPS, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.
Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Control- lable music production with diffusion models and guidance gradients. In NeurIPS Workshop Diffusson Models, 2023.
Shanchuan Lin and Xiao Yang. Diffusion model with perceptual loss. arXiv preprint, 2401.001 10, 2024.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024.
12

---

Published as a conference paper at ICLR 2025
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. ICML, 2023.
Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijndee and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers, 2024. URL hwwpt::tt//... /ttp:////xxvvvvvo org/ abs 2200 08740.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. ICML., 2020.
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. In ICLR, 2024.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Mchdi S. M. Sajjadi, Olivier Bachem, Mario Lučic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In NeurIPS, 2018.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Defossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In NeurIPS, 2023.
Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In JCML, 2023.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In WACV, 2022.
Lokesh Veeramacheneni, Moritz Wolter, Hildegard Kuehne, and Juergen Gall. Fréchet wavelet distance: A domain-agnose agnostic metrie for image generation. arXiv preprint, 2312. 2023.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- fation, 23(7):1661-1672 2011.
13

---

Published as a conference paper at ICLR 2025
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-Diffusion: Auto-regressive diffusion model for text generation. In NewrIPS, 2023.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for a variety of vision tasks. In CVPR, 2024.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric VQGAN for StableDiffusion. arXiv preprint, 2306,04632, 2023.
14


Reviews:
[{"review_id": "0Y5ujrbwMX", "reviewer": "Reviewer_sAd7", "summary": "The paper studies the perceptual loss function in the training of diffusion models and proposes to compare the features between $z_0$ and $\\hat{z}_0$ by sending them into the latent autoencoder's decoder model. The loss is only applied during the post-training stage and only applied in the time steps that have a SNR higher than the preset threshold. The results on CC3M, ImageNet-1k and S320M show that the introduction of this training strategy could help improve the generation quality of the model.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and straightforward. And the proposed method is simple and can be easily applied to the training of other diffusion models.\n2. Under the same training iterations during the post-training stage, the method can improve the FID over the baseline method which only adopts the MSE loss.", "weaknesses": "1. The paper only shows the performance increase over the baseline model. I feel like it's better to clearly demonstrate the effectiveness and performance gain over the previous state-of-the-art methods, to show that the perceptual loss can achieve what the widely used MSE loss cannot achieve.\n2. The introduction of the perceptual loss would increase the computation cost during the training stage. Could the authors provide a clear comparison on this? \n3. Following the last one, what would be the performance comparison for the same training time instead of the same training iterations?\n4. The authors mention the outliers in the features of the autoencoder's decoder which is not ideal for the computation of the perceptual loss. I'm wondering if the authors have tried other ways instead of simply masking those features out as this might cause information lost. Or has the authors tried using some other models to compute the perceptual loss to avoid those outliers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuWeCgKXZ7", "reviewer": "Reviewer_iXeA", "summary": "The paper proposes applying perceptual losses to latent diffusion models during training to improve the quality of generated images. Their proposed perceptual loss uses features from the VAE decoder, with additional contributions such as outlier filtering being proposed to enable this setup to work. The authors extensively validate the benefits of their approach over standard training of MMDiTs in different diffusion formulations and datasets.", "questions": "In addition to the addressing the weaknesses, I'd appreciate the authors answering the following questions:\n\nWhat is the difference in training FLOPS per step of using LPL vs. just the normal objective? I assume it is a minor difference, but if it is not, just comparing to normal objectives at the same step count would not be fair. Similarly, the effect of the substantial increase in VRAM usage on the training speed (via the maximum achievable local batch size) should be quantified properly.\n\nWhy do you use a CFG scale of 2? Standard choices for latent diffusion are typically much lower for ImageNet class-conditional generation and much higher for text-to-image.\n\nWhy is the frequency cut off at 360 in Fig. 6?\n\nSuggestion: while not technically required according to ICLR's rules, as it hasn't been published yet (and therefore also not taking an influence on my rating of this paper), I think a discussion of the almost one year old seminal work [https://arxiv.org/abs/2401.00110] in the area of applying perceptual losses to diffusion models would help improve the paper, especially if this work had any influence on the submitted paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The comparisons with baselines are extensive and clearly show that the proposed latent perceptual loss objective improves metrics across different diffusion formulations and datasets. I really appreciate the authors demonstrating the improvements for both eps-pred diffusion and flow matching, demonstrating that the proposed loss potentially has a general significance.\n\nThe paper ablates over/explores a large number of parameters and their influence.\n\nThe paper is generally reasonably well-written and accessible.", "weaknesses": "I think framing the proposed loss as a perceptual loss is likely incorrect. Perceptual losses typically try to incorporate human perception-based invariances into the loss, such as weighting the presence of the correct texture (e.g., grass) as more important than getting every detail of the instance of the texture (e.g., the exact positions of individual blades of grass) right. This is directly opposed to losses such as the MSE in pixel space. This is typically accomplished by taking features of a deep pre-trained discriminative model. In this case, you use features of the VAE latent decoder, which, potentially even puts the features at a lower abstraction level than the original latents. As far as I could see, there was no investigation of whether these features have the qualities of a perceptual loss. In the end, optimizing this loss seems to result in improved FIDs, which makes it a valuable contribution. Still, I think calling it a perceptual loss without further investigation goes against what people expect \"perceptual losses\" to refer to, which could lead to confusion. Could it be that the improvement actually doesn't come from perceptual qualities of the loss but rather from other qualities, such as a different implicit weighting of timesteps, which has previously been shown to substantially improve FIDs as well [https://openaccess.thecvf.com/content/ICCV2023/papers/Hang_Efficient_Diffusion_Training_via_Min-SNR_Weighting_Strategy_ICCV_2023_paper.pdf].\n\nYou also claim that \"the autoencoder’s latent space has a highly irregular structure and is not equally influenced by the different pixels in the latent code\" as an important part of your motivation. However, it has been shown [e.g., in https://discuss.huggingface.co/t/decoding-latents-to-rgb-without-upscaling/23204] that, at least for the very commonly used SD VAEs, the autoencoder latents effectively correspond to downsampled colors, differing fundamentally from that statement. Similarly, noisy latents from intermediate steps of the sampling process often already show a good approximation of the image being generated. A.2 shows an investigation of the effect of spatial interpolation for some (unspecified) AE, but is inherently very limited in its scope. It would be nice to see better backing up of this central claim that goes against common assumptions using standard methods, such as introducing small (random) deviations and showing how their effect on the decoded image is disproportionally large compared to larger ones. General claims should also be verified with standard VAEs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSECegdRvB", "reviewer": "Reviewer_UNn8", "summary": "The author introduces latent perceptual loss which acts on the decoder's intermediate features to enrich the training signal of LDM. The experiments showcased the effect of the added perceptual loss.", "questions": "- Line 179, Many SOTA T2I diffusion models use a basic L2 training objective. Since only a VAE citation is given to explain blurriness, could you add citations or evidence that diffusion training leads to blurriness? Typically, adversarial losses are used in diffusion distillation, not in standard diffusion training.\n- Line 185, \"a perceptual loss cannot be used directly on the predicted latents\", there is already accepted literature [2] that train another model which efficiently compute LPIPS losses in the latent space. It would be interesting to compare and contrast to the LatentLPIPS method.\n- Line 193, it would be better if the authors could stick to the canonical DDPM/EDM [3] notations to avoid confusion.\n- Line 289, do you enforce zero-terminal SNR here? It is known that the SD diffusion schedules are flawed [4].\n---\n[2] Kang, M., Zhang, R., Barnes, C., Paris, S., Kwak, S., Park, J., ... & Park, T. (2024). Distilling Diffusion Models into Conditional GANs. In ECCV 2024\n\n[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.\n\n[4] Lin, S., Liu, B., Li, J., & Yang, X. (2024). Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 5404-5411).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The author proposed latent perceptual loss (LPL) which shows the efficacy over various tasks datasets.\n- The qualitative results and the quantitative metrics seem promising\n- The frequency analysis showcases that the methods works\n- The ablations are carried out in a systematic way", "weaknesses": "- The paper seems unpolished and rushed towards the deadline\n- Sometimes the notation is a bit confusing, see the third point in the questions section\n- Applying perceptual loss in T2I generation isn’t novel; Lin and Yang [1] calculates the perceptual loss in middle blocks to reduce computation which bypasses the computational constraint, whereas this paper requires passing results to the decoder for intermediate features. If the authors can provide evidence that the method proposed is better and distant themselves to the literature then I would consider raising my score.\n\n---\n[1] Lin, S., & Yang, X. (2023). Diffusion Model with Perceptual Loss. arXiv preprint arXiv:2401.00110.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uYlqst820U", "reviewer": "Reviewer_W3EV", "summary": "This work introduces \"latent perceptual loss\" (LPL) to improve the training of diffusion models. According to the paper, LPL enhances diffusion models performance across various datasets (Table 2) and methods (Table 3). Additionally, extensive analyses, e.g., comparing the generation quality with and without LPL loss, and ablation studies, deepen our understanding of LPL.", "questions": "**1. Efficiency**\n\n- How does the result compare when using FLOPS as the x-axis? Comparing by iteration seems somewhat unfair. If the performance improvement looks promising when compared in terms of FLOPS, I'm considering raising the score.\n\n**2. Novelty**\n\n- Is there a specific reason why LPL was only used in post-training? I feel that LPL might provide a good gradient signal early in training, but it seems that experiment wasn't conducted, which is unfortunate. I'm curious about the performance changes if LPL is applied from the start of the pretraining.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**1. Extensive analysis**\n\nThis work provides extensive analysis to demonstrate the validity of LPL. For instance, the major performance gain from LPL arises from generating more precise high-frequency components, albeit at the expense of some low-frequency details.\n\n**2. Ablation study**\n\nIt gives us various ablation study to provide the motivation behind design choices. The ablation study includes various components including feature depth for LPL, SNR threshold value, Reweighting strategy, or etc. This level of extensive ablation study is rare.", "weaknesses": "**1. Efficiency**\n\nIn the paper, it is mentioned several times that utilizing the VAE decoder features to compute the LPL loss is costly. This is why recent research has explored alternative perceptual losses, such as latent LPIPS [1]. While the paper claims that LPL incurs minimal computation since it’s only used post-training, this claim is far from acceptable, as the post-training phase in this paper involves iterations amounting to as much as one-third, or at minimum one-fifth, of the pretraining iterations.\n\n[1]: Distilling Diffusion Models into Conditional GANs ([ECCV24](https://mingukkang.github.io/Diffusion2GAN/))\n\n**2. Novelty**\n\nAlthough subjective, I believe the novelty of LPL is somewhat lacking. This loss trick slightly enhances quality during training but seems more heuristic than principled. Additionally, it doesn’t contribute much new knowledge about diffusion models, which weaken the paper's novelty. Furthermore, it does not appear that any non-trivial trick was devised in the process of introducing the perceptual loss to the diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)