Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
Authors: Cristina Nader Vasconcelos
Abdullah Rashwan
Austin Waters
Trevor Walker
Keyang Xu
Jimmy Yan
Rui Qian
Yeqing Li
SHIXIN LUO
Yasumasa Onoe
Zarana Parekh
Ivana Kajic
Mandy Guo
Wenlei Zhou
Sarah Rosston
Roopal Garg
Hongliang Fei
Jordi Pont-Tuset
Su Wang
Henna Nandwani
Andrew Bunner
Kevin Swersky
David J. Fleet
Oliver Wang
Jason Michael Baldridge
Venue: Accepted by TMLR ()

Abstract: We address the long-standing problem of how to learn effective pixel-based image diffusion models at scale, introducing a remarkably simple greedy method for stable training of large-scale, high-resolution models. without the needs for cascaded super-resolution components.The key insight stems from careful pre-training of core components, namely, those responsible for text-to-image alignment vs. high resolution rendering. We first demonstrate the benefits of scaling a Shallow UNet, with no down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to improve alignment, object structure, and composition. Building on this core model, we propose a greedy algorithm that grows the architecture into high resolution end-to-end models, while preserving the integrity of the pre-trained representation,stabilizing training, and reducing the need for large high-resolution datasets. This enables a single stage model capable of generating high-resolution images without the need of a super-resolution cascade. Our key results rely on public datasets and show that we are able to train non-cascaded models up to 8B parameters with no further regularization schemes.Vermeer, our full pipeline model trained with internal datasets to produce 1024×1024 images, without cascades, is preferred by 44.0% vs. 21.4% human evaluators over SDXL.

Published in Transactions on Machine Learning Research (10/2024)
Greedy Growing Enables High-Resolution Pixel-Based Diffu- sion Models
Cristina N. Vasconcelos, Abdullah Rashwan, Austin Waters, Trevor Walker, Keyang Xu, Jimmy Yan, Rui Qian, Shixin Luo, Zarana Parekh, Andrew Bunner, Hongliang Fei, Roopal Garg, Mandy Guo, Ivana Kajic, Yeqing Li, Henna Nandwani, Jordi Pont-Tuset, Yasumasa Onoe, Sarah Rosston, Su Wang, Wenlei Zhou, Kevin Swersky, David J. Fleet, Jason M. Baldridge, Oliver Wang Google
Reviewed on OpenReview: https: // // openreiiew. net/forunt id- GU-UUUULAANN
Abstract
We address the long-standing problem of how to learn effective pixel-based image diffusion models at scale, introducing a remarkably simple greedy method for stable training of large- scale, high-rcsolution models without the nceds for cascaded super-resolution components. The key insight stems from careful pre-training of core components, namely, those respon- sible for text-to-image alignment benefits of scaling a Shallow UNet, with no down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to improve alignment, object structure, and composition. Building on this core model, we propose a greedy algorithm that grows the architecture into high res- olution end-to-end models, while preserving the integrity of the pre-trained representation, stabilizing training, and reducing the need for large high-resolution dalasets. This enables a single stage model capable of generating high-resolution images without the need of a super-resolution cascade. Our key results rely on public datasets and show that to train non-cascaded models up to 8B parameters with no further regularization schemes. Vermeer, our full pipeline model trained with internal datasets to produce 1024 x 1024 images, without cascades, is preferred by 44.0% us. 21.4% human evaluators over SDXL.
high resolution rendering. We first demonstrate the
able
1 Introduction
Training large-scale Pirel-Space teat-lo-image Diffusion Models (PSDM) to generate high-resolution images has been challenging due to optimization instabilities arising when growing model size and/or target image resolution, and due to the increasing demand for computational resources and high resolution training corpora. The predominant alternatives include cascaded models, comprising a sequence of diffusion models each targeting progressively higher resolution and trained independently (Ho et al., 2022a; Saharia et al., 2022a; Nichol et al., 2022), and latent diffusion models (LDMs), where generation is performed in a low- dimensional latent representation, from which high resolution images are generated via a pre-trained latent decoder (Rombach et al., 2022).
In the development of cascaded models, it challenging identify sources of quality degradation and distortion resulting from design decisions at specific stages of the model. One well-known issue of cascades is the distribution shift between training and inference, where inputs to super-resolution or decoder models during training are obtained by down-sampling or encoding training images, but during inference they are generated from other models, and hence may deviate from the training distribution. This can cause amplification unnatural distortions produced by models early in the cascade. The generation of realistic small objects such as faces or hands is one such challenge that has been difficult to diagnose in such models.
Beyond image generation per se, diffusion models serve as image priors for myriad downstream tasks, in- cluding inverse problems (Jalal et al., 2021; Kadkhodaie & Simoncelli, 2021; Kawar et al., 2022; Song et al.,

---

Published in Transactions on Machine Learning Research (10/2024)
O
START
NY
08
XI
IIII
VIII
Figure 1: Images generated with our model Vermeer. (See Appendix A for the prompis.)
2023; Chung et al., 2023; Graikos et al., 2022; Tang et al., 2023; Jaini et al., 2023; Zhan et al., 2023; Song et al., 2024), or other generative tasks (Ho et al., 2022b; Levy et al., 2023; Poolc et al., 2023; Tan et al., 2023; Bar-Tal et al., 2024; Chen et al., 2023; Tewari et al., 2023). Cascaded diffusion models are not readily applicable such tasks, and as a consequence, many such applications rely solely on the score function from relatively low resolution. A high resolution end-to-end model would
the base model of cascade, often alleviate these issues, but model development and effective training procedures have been clusive.
Key barriers to training high resolution models include prohibitive resource requirements in both memory and computation. Existent recipes require large batch sizes during training to avoid instabilities, and as a consequence, intractably large amounts of memory for high-resolution images. Another issue concerns the need for high quality, high resolution training data. Existing training methods require large, diverse corpora of text-to-image pairs at the target resolution, while in practice, such data are not readily available high resolution.
This paper introduces framework for training high resolution, large-scale text-to-image diffusion models without the use of cascades. To that end we explore the extent to which one can decouple the training of 'visual concepts' associated with lextual prompts, from the resolution al which one aims to render the image. Such disentanglement has two goals. It aims at a better understanding of alignment, composition and image fidelity (espocially for well-known hard cases like generating consistent hands, text rendering, scene composition, etc.) as a function of model scaling (e.g., see Figure 3). Second, and of equal importance, our framework yields robust and stable recipe for training large-scale, non-cascaded pixel-based models
2

---

Published in Transactions on Machine Learning Research (10/2024)
targeting high-resolution generation. A bonus is that our recipe allows us to jointly train single model with data comprising multiple resolutions, even if high-resolution text-image pairs are relatively scarce.
The contributions of this paper can be summarized as follows:
We introduce novel architecture, Shallow-UViT, which allows one to pretrain the PSDM's core layers on datasets of text-image data (subsection 3.2), eliminating the need to train the entire model with high resolution images. This also allows to investigate the emergent properties of PSDM representation scaling in isolation from layers targeting generation at the final resolution.
We present a greedy algorithm for training the Shallow-UViT architecture that allows us to success- fully train a high-resolution text-to-imas model with small batch sizes (256 versus the typical 2k used in end-to-end solutions) (section 3).
We show that one can significantly improve different image quality metrics by leveraging the repre- sentation pretrained at low-resolution, while growing model resolution in greedy fashion. Scaling the core components of the Shallow-UVIT architecture alone leads to significant improvements in image distribution, quality and text alignment (section 5).
We demonstrate that these principles work at scale by presenting Vermeer (Figure 1), model Irained with our greedy algorithm on large-scale corpora, in conjunction with other well-known methods like asymmetric aspect ratio finetuning, prompt preemption and style tuning (section 6). Vermeer is shown to surpass previous cascaded and auto-regressive models across diffcrent metrics. In human evaluation study with 500 challenging prompts and 25 annotators per image, Vermeer is preferred over SDXL (Podell et al., 2024) by 2 to margin.
2
Related work
Current high-resolution image generation with diffusion models presents a trade-off between architectural complexity and efficiency. Cascaded diffusion models (Nichol et al., 2022; Dhariwal & Nichol, 2021; Sa- haria al., 2022b; Ramesh et al., 2022; Balaji et al., 2022) were originally introduced to circumvent the difficulty of training single stage, end-to-end model. Cascaded models employ multi-stage architecture that progressively up-scales lower-resolution images to address the computational challenges of generating high-resolution images directly. Nevertheless, they entail significant complexity and training overhead, as the stages of the cascade are trained independently.
Simple Diffusion (Hoogeboom et al., 2023b) sought to simplify the process by targeting the high resolution generation with a single stage model, introducing a novel UVTT architecture and several useful modifications to training methods that improve stability. While this approach is shown to be effective, stability issues remain when targeting large-scale models, and high resolution images, due in part to their dependence large batch sizes. In this work we adopt similar UVIT architecture, and some of their techniques for scaling, extending the model to much higher resolutions through greedy training. Through scaling the core backbone of the model, and with our greedy training procedure, we find with can scale to much high resolution models (2x to 8x higher than Simple Diffusion), with excellent alignment, and much smaller batches when training high resolution layers of the model.
Another line of work proposed Matryoshka Diffuusio Models (MDM) (Gu et al., 2023) that denoises multiple resolutions using proposed Nested UNet architecture. They progressively train the network to preserve the representation at higher resolutions. We show in this work an alternate and simpler approach where demoising multiple resolutions is not required, but instead it is crucial to preserve the representation by freezing the pretrained weights as we grow the architecture up to its final design.
On another front, latent diffusion models (LDMs) (Rombach et al., 2022; Jabri al., 2022; Betker et al., 2023) reduce computational costs by operating within a compressed latent representation. However, LDMs still require separate super-resolution latent decoder networks to produce final high-resolution images.
3

---

Published in Transactions on Machine Learning Research (10/2024)
The model we introduce also resembles progressive GAN training (Karras et al., 2018) in which layers of increasing resolution are added at each stage. Our work can be thought of as an extension of progressive growing for diffusion models, where we evaluate different growing configurations, and come up with two-step recipe that arrives at a good trade-off of training efficiency, robustness, and generation quality. Specifically, while all layers remain trainable in progressive GANs, and a sequence of growing operations performed before reaching the final architecture, we pretrain core representation that remains frozen when training all grown layers at once up to the target resolution. We find that this is crucial to preserve the quality of the representation learned at lower resolutions.
3 Method
Our goal is to create straightforward, stable methodology for training large scale pixel-space diffusion models that operate revisit the UNet architecture, aiming to decouple layers that have a major impact on text-to-image alignment (core components) from those responsible for rendering at the target image resolution srncoder-decoder or super-resolution compoments). Next, we focus on pre-training the core components pretraining and on representation scaling (subsection 3.2). Finally, we present a greedy algorithm to grow the initial architecture core by adding encoder-decoder layers while protecting core layers' representation. This yields a single-stage model at inference time (subsection 3.3).
single stage model, i.e, non-cascaded, at inference time. To this end, we first
3.1 Text-to-image core components
UNet the architecture of choice for diffusion models. Two architecture families are common. In one, con- volutional networks comprise a stack of convolutional blocks alternated with pooling or downsampling layers in the encoder, and upsampling layers in the decoder. More recently; the UViT family emerged (Hoogeboom et al., 2023a), in which convolutional blocks are used at the higher layers of the encoder and decoder but augmented with transformer layers at the bottom of the UNet. In both architectural familics, text con- ditioning is accomplished via cross-altention layers, also at the bottom, low-resolution layers of the UNet. In doing so, these layers are responsible for conditioning the models' deepest representation on the textual and/or multi-modal inputs. At these low-resolution layers, the text conditioning signal able to influence the global image composition while the computational cost of attention kept relatively low.
Our search for a methodology that allows stable training of large models starts by identifying and isolating core layers responsible for text-to-image aligument. Our main conjecture is that it is possible to reduce the instability typically observed during training large-scale PSDMs by warming up layers responsible for text-to-image alignment in isolation from layers responsible for target resolution encoding/decoddigg
Specifically, we define the core components as those that directly interface with text conditioning signals and those that are crucial in the diffusion process. They can be described as:
Text encoding layers combine one or more textual, character, and/or multimodal pretrained repre- sentations (such as those from Raffel et al. (2020b); Xue et al. (2022a); Liu et al. (2023); Radford et al. (2021a)), and project them into the embedding space of the UNet. Typically composed of MLP on top of pooling layers.
Core representation layers comprise hidden layers the main backbone interfacing with cross- attention layers. They include the bottom layers of the UNet architecture whose features are directly combined with the embedded text by the cross attention operation and layers between them.
Time encoding layers map the diffusion time step into the model's embedding space. Typically designed as sinusoidal positional encoder, followed by shallow MLP. Despite not participating directly in the cross-attention operation, it is core component of the diffusion process.
We isolate these core components of PSDM text-to-image model order to study their effect on the final model's properties. Next, we propose an architecture that enables the pretraining of these layers, and also supports the study of the properties emerging from scaling them.
4

---

Published in Transactions on Machine Learning Research (10/2024)
3.2
Shallow-UVe
time
MLP
MLP
AttnPool
Text embeddings
cross- attention
Input 64x64 pixels
Tnneino C. 64x64 pixels
3x3 subpixel
5x3 subpixer conv
7x7 Conv Stride 4
Shallow encoder
decoder
Shallow
VIT
conv
16x16
32x32
Figure 2: Shallow-VViT architecture: The imput image grid is quickly reduced at the entry convolution, while a single residual block with no subsampling layers used as a shallow encoder and decoder. The layers within the core components (in light green) are reused in the final end-lo-end architecture, increasing its training stability, while remaining layers are discarded.
To assist the pretraining of the core components and, at the same time investigate the emerging properties from their scaling, we isolate the core components training and scaling from other confounding factors in the specification of the UNet's encoder-decoder layers. To that end, we simplify the UNet's conventional hicrarchical structure, which operates on multiple resolutions, and define the Maalllo-UV11 (SU), a simplified architecture comprising shallow encoder and decoder operating on a fixed spatial grid (Figure 2). Its encoder and decoder have single residual block each, containing two layers of 3 convolutions with swish activations Ramachandran et al. (2017), and no upsampling downsampling layers. As result, they share the same spatial grid as the core representation layers at the bottom. The first convolutional layer at the entry of the architecture projects the input image into the fixed size grid used by its core layers. A corresponding upsampling head at the model's output reverses this operation. These input/output layers facilitate quickly projecting input images with larger resolution into the core representation with fixed and lower resolution.
As second simplification, we restrict our investigation to the core components from the UVIT model family owing to the uniform structure of its core represenlation layers. In contrast, the corresponding layers of convolutional UNets present broader spectrum of design and hyperparameter choices, owing to their non- uniform yet hierarchical structure, rendering their analysis more complex.
An alternative to the proposed use of the Shallow-UVis architecture, might be to train the core components directly as an augmented ViT, as previously explored latent diffusion models (Peebles Xie, 2023). Our attempt explore this approach proved not be straightforward. A crucial difference between PSDM and LDM becomes highly relevant here. In the case of LDM, the transformer operates latent tokens, and the diffusson model captures the latent token distribution. Our task, on the ht hand, is to pretrain rich representation directly from the raw pixels, for subsequent reuse as deep features within higher-resolution pixel-space model. We conjecture that in such approaches the initial layers that are closer to the raw data do not transfer as well when reused within the final model.
Instead, our Shallow-UVis includes proxy additional layers that help with closing the gap between core components feature pretraining and their later use. That is, the auxiliary, yet shallow, input (output) and encoding (decoding) layers help adding expressiveness to the transformations between the input (output) and the models' hidden representation. Across the variations explored, the input convolution expands the number input channels up to 256 (we observed no improvement with more channels).
Beyond ablations on scaling (see section 5), we also found that certain variations for the Shallow- UVIT composition tend to degrade performance in comparison to our best architecture. In particular, these include the removal of the shallow encoder encoder/decoder blocks; the use of smaller /larger filters (4 x 4,5 5 x 5, 9 x 9) and strides (from 1 up 8) at the entry convolution; and the use of single output head with subpixel
5

---

Published in Transactions on Machine Learning Research (10/2024)
convolution upsampling by a factor of 4. We also experimented with convolutional core representation layers, but like Dosovitskiy et al. (2021), we find they under-perform their transformed-basee counterparts.
3.3 Greedy growing
Here we describe a greedy approach to learn PSDMs for high-resolution images. Our process consists of two distinct stages, where we first pretrain the core representation layers at low resolution using a Shallow- UViT architecture. Then, in the second phase, we replace the encoder aucoder/ecoode /ccooder layers with more expressive set of UNet layers and train at the larget resolution. This lwo-stage process in contrast. to progressive growing, which seeks to add one layer at time. With this approach, we aim to mitigate the well-known instabilities observed during training of large models (Saharia et al., 2022b; Hoogeboom et al., 2023b), while making the best use of the available training corpora.
The greedy growing algorithm can be described as follows.
Phase In this phase, the core components of the chosen architecture are identified (see subsection 3.1), and a Shallow-UViT model build on top of them. The Shallow-UViT trained on the entire training collection of text-image pairs, as is not limited to high resolution training images.
Phase 2 The second phase greedily grows the SVallow-UViT's encoder encoder/decoder /decoder (namely, throwing away the lower-resolution blocks and adding higher-resolution blocks) to obtain the final model. More specifically, this phase adds encoder and decoder layers at different resolutions, while preserving the core representation layers at the spatial resolution used during the first phase. In other words, the cone components contimue operating on a 16 x 16 grid. The added layers are randomly initialized, while the core components are initialized with the weights obtained on the first phase. The remaining components of the Shallow-UViT model are discarded.
Next, the grown model trained. As it common practice for the generation of high fidelity images, at this point filter the training data to remove text-image pairs with either image dimension is lower than the final model's target resolution. The teal encoding layers and the core representation layers are kept frozen, to preserve the richness of the pretrained representation. The time encoding layers, on the other hand, are further tuned, jointly with the new encoder and decoder layers introduced in the second phase, which allows it to adapt to changes in the diffusion noise schedule. We adjusted the diffusion logSNR shift for high resolution images as suggested by Hoogeboom et al. (2023b), by factor of log(64/d). An opt iotal third defrosting phase, may be applied in which all layers are jointly tuned, and seeks benefit from the full capacity of the end-to-cnd architecture, but in practice we find that the first two phases sufficient to obtain good PSDM.
We empirically investigate the behaviour of the proposed algorithm in models of increasing size in subsec- tion 5.2. We investigate the effects of splitting the training of the two tasks in phase one and phase two (i.e., for alxt-aligmment and high-resolution generation), and we compare with models jointly trained from scratch, end-to-end. During these ablations, we constrain the greedy growing phase to use considerably smaller batch sizes than previous work, with no further regularization to demonstrate the optimization stability.
4 Experimental settings
The proposed Shallow-UViT provides proxy architecture for pre-training the com- ponents of a larger PSDM. The ablation studies below specific instantiation of the model, but we expect Shallow-UViT to be flexible enough to be used with other component parts. In particular we adopt a combi- nation of (wo pretrained lext encoders for text conditioning: T5-XXL (Raffel et al., 2020a) with 128 sequence length and CLIP (VIT-H14) (Radford et al., 2021b) with 77 sequence length. Given a text prompt, we first tokenize and encode the text using the two encoders independently, and then concatenate the embeddings, yielding a final embedding with sequence length of 205. They are projected into model's hidden size by the text encoding layers. We keep the Shallow-UViT design fixed, except for changing the capacity by increasing
Shallow-UViT:
6

---

Published in Transactions on Machine Learning Research (10/2024)
its width (hidden size) and depth (number of transformer's blocks), as detailed in Table 1. That produces
set of models varying from 672M up to 7.7B trainable parameters, mostly dedicated to the core components.
model
transf. blocks
hidden size
MLP channels
heads
params
Shallow- UVIT Small
6
1536
6144
12
672M
Shallow-UUT UViT Large
8
2048
8182
16
1.3B
Shallow- UViT Huge
12
3072
12288
24
3.5B
Shallow-UViT XHuge
16
4096
16384
32
7.7B
Table 1: Shallow-VViT variants explored. Transformer layers operating at 16 16 grid. The components
within the shallow encoder and decoder block operate at same spatial resolution and hidden size.
End-to-end model
channels per layer
residual blocks
"params
UVIT Small
256-384-768-138 256-512-1024-2014 384-768-153663072
1-1-1-1
707M
UVIT Large
1-1-1-1
1.4B
UVIT Huge
1-1-1-1
3.6B
UVIT XHuge 512-1024-2048-1
1-1-1-1
7.9B
Table 2: Composition of the encoder-decoder layers grown on top of corresponding Shallow-UViT variants. core components identical to the corresponding shallow variant.
We stress that do not claim that these specific core components are optimal. For instance, it is widely recognized that larger pretrained text encoders and longer token sequence lengths increase image quality (Saharia et. al., 2022b; Balaji et al., 2022; Podell et al., 2024). Investigating the optimal design of each core component is beyond the scope of this work. Instead, the variations of the -VVllow-UViT were intentionally designed to explore the performance benefits gained by increasing core components's 's capacity independent of the remaining model components.
Greedy growing: In the experiments that follow we consider several different model sizes. Table 1 specifies the Shallow-UVii variants, while Table specifies prcoder/decodee parameterizations.
To ablate our hypothesis that greedy growing helps the model learn strong representations with larger, diverse corpora, we also train the full model on a high resolution subset of data used to train the Shallow-UViT. i.e., we simply removed all samples with resolution lower than the target model resolution. To that end, beyond greedy growing, we explore the three training baselines: 1) We create a bascline with all layers trained from scratch on this subset; 2) As an alternative to the frozen phase in the greedy growing, we fine-tune the core components on this smaller high resolution subset. jointly with the grown components (randomly initialized); and 3) A third baseline adds the optional phase of unfreezing the core components after warming up the random weights for 500k steps. Models are trained for 2M steps in total.
The greedy growing algorithm aims to make training large-scale PSDMs at high resolutions more stable. Tn the case of Simple Diffusson (Hoogeboom et al., 2023b), large batch sizes and regularizers like dropout and multi-scale losses enable end-to-end training from scratch. To stress test the stability and convergence of our greedy growing algorithm, restrict the batch size 256 instead of the standard 2k, and we use no other explicit form of regularization. Under that restriction, our largest model (UVit-XHugo presented numerical instabilities when trained from scratch or fine-tuned, multiple numerical issues occurred during training, Thus, the results of this large model are presented only for the frozen, and freeze-unfreeze methods. This behaviour confirms observations in previous work and their need for large batch sizes.
Rigorous evaluation of generative image models is challenging when models are trained on pro- prietary datasets. To avoid this issue, we first demonstrate our key findings through extensive empirical evaluations on a publicly available dataset, namely, Conceptual 12M (or CC12M) (Changpinyo et al., 2021).
Dataset:
Number trainable parameters after ignoring text. encoders.

---

Published in Transactions on Machine Learning Research (10/2024)
To evaluate the hypothesis that the greedy algorithm allows one to make good use of available corpora, we trained Shallow-UViT on the entire CC12M training set, while corresponding end-ta-end models were trained with CC12M's subset of 8.7M images whose dimensions are equal or larger than 512 pixels. Those end-to-end models were therefore trained on 27.5% less data than the corresponding Shallow-UVil model. We do not explore more aggressive reduction of the corpora as the CC12M dataset already a relatively small datasct for the models tested, and the variations tested already show overfitting characteristics under this setting, as discussed below. Thus, in what follows, the Shallow-UViT models were trained on 64 64 images, by resizing the smallest dimension of the images to 64 and random cropping along the remaining dimension as needed. The end-to-end models are trained at a target resolution of 512 512 as CC12M does not contain images at resolutions above 1024 pixels.
Full pipeline model: With those findings in place, we then explore the generation of larger images and train on a much larger curated datasets in order to show that the approach scales to Seate-of-the-ar models (section 6). The resulting model, named Vermeer, is used to generate 1024 x 1024 images, well beyond the scale for which quantitative metrics are readily available. As such, with Vermeer we rely on human evaluation, comparison to other recent models, like SDXL.
Sampling: Unless mentioned, the images and metrics were produced using 256 steps of a DDPM sampler (Ho et al., 2020) with classífier-free guidance (Ho & Salimans, 2021). We tune the guidance hyper-parameter by a FD-Dino//IIi (VIT-L14) trade-off as described in subsection 5.3.
4.1 Metrics
The evaluation of generative models poses considerable difficulties and constitutes an active research area (Kirstain et al., 2024; Xu et al., 2024; Hessel et al., 2021; Serra et al., 2023; Kim al., 2024; Lee et al., 2023). Tn light of its inherent complexity, we utilize multi-faceted evaluation strategy that combines image distribution metrics, text-alignment metries and semantic question and answering metrics to validate our intermediary results, but the overall performance of our final model evaluation, Vermeer, is delegated to human evaluators (subsection 6.2). The following criteria are considered:
Image distribution metrics: We evaluate models on three key metries, namely, the Fréchet Inception Distance (FID) (Heusel et al., 2017), the Fréchet Distance on Dino-v2 feature space (FD-Dino) (Stein et al., 2023; Oquab et al., 2023) and the Clip Maximum Mean Discrepancy (CMMD) distance (Jayasumana et al., 2023). FID widely used to assess generative image models and select model prree-pprametess but our findings corroborate its known limitations: it fails to reflect model improvements through training, it does not capture readily apparent distortions in individual images, and it does not correlate well with human perception (Stein et al., 2023; Otani et al., 2023; Jayasumana et al., 2023). Thus, in our study, we do not select training or sampling hyper-parameters solely on the basis of FID but, as described in Appendix 5.3, we review the trade-offs between the observed set of metrics.
We also note that metrics derived from image features vary considerably with image resolution. In what follows we compute metries using the same resolution as the reference papers. The exception is for CMMD Shallow-UViT outputs; the original metric taken at 336 336 pixels is dominated by up-sampling effects, obscuring differences between models, Thus, replaced the original Vir - L14 operating at 336 x 336 by its version at 224x x 224 pixels.
We adopt CLIP Score metric for text-image alignment, as it is widely used, and it complements image distribution metrics above, reflecting the consistency of the generated image with the given prompt. Unlike the original formulation based on ViT-B with path size 32 (Hessel et al., 2021) and previous papers in the area Saharia et al. (2022a); Hoogeboom et al. (2023b), we adopt the ViT-L (patch 14) embedding due to its improved representation. This choice results in lower absolute values of our CLIP Scores compared to previous results, however we noticed that these scores better correlate with the presence of absence of observed distortions.
Multimodal metrics:
8

---

Published in Transactions on Machine Learning Research (10/2024)
Shallowr-VVil Bosc
Shallow- UVIT Large
HAARD DRRDAPP STTTRDDAY
Happy Birthday
Shallow- UVIT Huge
HAPPY BIRTHEAY
Shallow- UVIT XHugge
Figure 3: Qualitative comparison of models with core components of increasing size Shallow- UVITs trained at 64 x 64 pixels using CC12M dataset only. Prompts: 'A sloth running a marathon, sur- prisingly outrunning all competitors.' 'A hand spread out on a wall. DSLR photograph.' 'Close-up portrait of ballerina in mid- performance, with high motion and dramatic lighting.", 'Word art of 'happy birthday". with a smiling panda wearing a party hat, surrounded by gift bores and birthday cake.' 'Four dogs on the street.'
Semantic QG/A frameworks: One can also automatically generate question-answer pairs with a lan- guage model, and then compute image faithfulness by chocking whether cxisting VQA models can answer the questions from the generated image (Hu et al., 2023; Cho et al., 2024). They were intended to address the shortcomings of existing metrics. Despite their effectiveness in evaluating color and material aspects, they often struggle in assessing counting, spatial relationships, and compositions with multiple objects. Such evaluation measures are naturally dependent the quality of the underlying question generation (QG) and answering (QA) models. Here we adopt DSG (inage-text alignment metric) and its set of 1 prompts (Cho et al., 2024). The DSG-Ik test-prompts cover different challenges (e.g., counting correctly, correct color totr//saap//txt rendering, etc.), semantic categorics, and writing styles. description of the QG, QA used, with qualitative and detailed results, are included Appendix B.
5 Experiments
5.1
Pretraining and scaling the core components
We next use Shallow- UVIT as a proxy architecture investigate the effect of scaling PSDM's core compo- nents. We train UVaIlow-UViT variants 64 x 64 images from the CC12M dataset for 2k steps. Image

---

Published in Transactions on Machine Learning Research (10/2024)
modelsgeix61
FIDa0k J
FD-Dinosok J
CMMD J
CLIP acore
Shallow-UViT Base
16.97
356.25
0.197
0.234
Shallow-UVE Large
14.80
236.24
0.156
0.240
Shallow-UViT Huge
8.81
133.51
0.139
0.244
Shallow-UVE UVIT XHuge
8.41
116.83
0.136
0.246
Table 3: Shallow-UVil variants with core components of increasing size trained CC12M at resolution
64 x 64: Image distribution metrics evaluated on 30k samples from MSCOCO captions dataset. Scaling induces performace improvements on image distribution (FID, FD-Dino, CMMD) and text-image alignment (CoLPscorre metrics simultaneously.
DSG VqVa Question Types Attributes
Entities
Relations
Global
DSG(t)
#questions:
3378
1485
1722
649
Shallow- Small
54.38
33.32
43.70
39.98
48.08
Shallow-UViT Large
59.93
39.36
48.75
43.68
52.54
Shallow-UVe Huge
69.18
48.52
54.36
43.30
60.25
Shallow-UVIT XHuge
70.66
51.61
57.38
44.14
61.91
Table 4: Shallow-UVIT evaluated on 1k samples from DSG-1k dataset. Scaling core components improves performance across all semantic categories. Fino-grained results in Appendix B
distribution metrics and Clip-Score obtained using 30k prompts from the MSCtCO-saptions validation set (Chen et al., 2015), while the semantic metrics are extracted on the 1k prompts from DSG-1k (Cho et al., 2024). A summary of the impact of scaling the Shallow-UViT model is given in Tables 3 and 4, while fine grained results on semantic categories are reported in Appendix B. All performance measures indicate significant improvements due to model scaling. smaller numerical gain is observed in the comparison of the larger two models, but the difference is reflected in qualitative comparisons of the models below.
Figure 3, presents qualitative comparison of the results the Shallow-UViT variants challenging prompts. They illustrate the impact of scaling objects structure, composition and alignment (c.g., with numbers of objects depicted). Despile of the small training dataset, the larger models show significant improvement in generating intricate shapes like hands, body parts and text.
We obscrved further quantitative improvements across the metrics when training our larger models for longer Shhallow-VVTT-EE and Shallow-UViT--HHI XHuge), but longer training also exhibits overfitting to the CC12 training samples. Figure illustrates images generated using the Shallow-UVil XHuge model with increasing numbers of training steps. As training progresses, the model diverges from the original prompt to produce images that are closer training samples from the CC12M dataset, and/or representing parts of the prompt only. This hidden phenomena was not associated with changes in the adopted metrics. We conjecture that this effect is largely aggravated by the small size of the training dataset.
Considering the complexity associated with evaluating improvements in representation and the limitations of automatic performance measures, we also ablate the effect of scaling the core components under a semantic task that is evaluated by human annotators. In this experiment consider simple counting task, defined here as the task of generating images of up to 5 objects based subset of text prompts from the numerical split of the Gecko benchmark (Wiles et al., 2024). We explore this task proxy for gauging both prompt consistency and the model's understanding of objects composition and shapes. It allows less subjective interpretation and noise in human judgments of the model's performance than other image qualities that are influenced by individual preferences. The task of counting under an open set would ultimately imply the ability to keep track of objects. Thus, this ablation emulates a much simpler version of the problem. Figure 5 shows the accuracy improvement associated with scaling observed over 59 prompts. Random condition uses random number between 1-5. The detailed description of this experiment is presented Appendix C.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Refereuce
250k
COOK
1M
1.25M
1.5M
1.75M
Figure 4: Overfitting and memorization of Shallow-UViT XHuge trained on CC12M. Prompts: (top) A group of construction workers in the style of 'The Night Watch' by Rembrandl.; (middle) A dynamic rendition of a racing cyclist leading their team through a mountain pass, rendered in the style of 'Napoleon Crossing the Alps' by Jacques- Louis David.; (bottom) group of friends enjoying a summer day al a riverside restaurant in the style of 'A Sunday Afternoon on the Island of La Grande Jatte' by Georges Seurat.
0.35
Accuracy
0.25
6:15
Base
Large
Huge
XHuge
0.29
Base
Large
Huge
XHuge
Random
Figure 5: Measuring the impact of scaling on the counting task. Using 59 systematic prompts describing 1-5 objects. Five human annotators reviewed each image (95% bootstrapped confidence intervals are shown). Models with larger core components are observed to perform better on counting. Sample prompt: 3 apples.
Given the shallow encoder-decoder structure of the Shallow-UViT architecture, we conjecture that the per- formance improvements observed here, on multiple metrics, are a direct consequence of scaling the core components. This hypothesis is further investigated via the reuse of their representation in the next section.
5.2 Experiments on Greedy growing
We next explore greedy growing of Shallow-UVil models to high resolution, non-cascaded models. We compare training models from scratch on the subset of the CC12M dataset filtered by the target resolution (512 pixels) with alternatives for reusing of the core components pretrained on the full dataset. They validate our main intuitions behind the greedy growing algorithm, ie.. that the introduction of new, untrained layers, as well as shifts the distribution the training data are known causes of the catastrophic forgetting phenomena Vasconcelos al. (2022); Kuo et al. (2023); Yu et al. (2023) possibly damaging the pre-trained representation.
Tables 5 and 6 summarize performance function of model scale for greedy growing, along with various ablations of the training procedure. Our greedy growing recipe with frozen components's 's and its optional defrosting phase lead to the best results across the metrics. The optional defrosting phase is required for improving the performance of the smallest model ablated (UViT-Base). Its frozen counterpart showed signs of underfitting during training, as it has a small number trainable parameters (217M) in the added layers. Under this low-capacity scenario, the defrosting phase offers a balance between protecting the core components representation and the use of the model's full capacity, as it reduces the degradation of the pretrained representation by warming up the growth layers. Other than this special case, the defrosting
11

---

Published in Transactions on Machine Learning Research (10/2024)
phase did not appear to benefit larger models. These quantitative results agree with our hypothesis that the final model benefits from protecting the pretrained representation in our greedy growing algorithm.
Figure 6 qualitatively compares generations obtained by finetuning and freezing the core components. Ad- ditional qualitative comparisons are shown in Appendix D. They illustrate the the benefits of protecting the core components from the noise introduced when back-propagating through the randomly initialized growth layers. We observe that the low-resolution images produced by the use of the same representation under their original SVallow-UViT models produce objects whose shapes and parts are correctly defined.
The high-resolution images generated from early steps (20k) of finetuning the core components under the U'iit architecture present objects with correct shapes superimposed with the diffusion noise. Soon after that (around 50k-100k steps) the quality of object shapes and structure decays as the training backpropagates the noise introduced by the growth layers through the pretrained representation.
Under the greedy growing regime and same number of training steps (20k steps) the frozen model is able to produce objects with correct shapes and parts, and maintain their composition training progresses. Another direct side effect of maintaining the core components representation is the fast reduction of the diffusion noise carly in training.
model
fr. params
steps
FID 4
PD-Dinogos
CMMD
CLIP
UViT-Base
scrotch
707M
2M
27 27.90
624.34
1.355
0.241
fineturing frozen core
2M
23.67
554.99
1.450
0.211
217M
24.68
563.35
1.614
0.235
freeze-unfreeze
217M/707M
2M
21.13
503.16
1.196
0.247
UVIT-Large
scratch
L.4B
2M
21.73
498.82
1.156
0.217
finetuning frozen core freeac-unfreeze
2M
21.89
414.42
1.160
0.253
351M
2M
17.68
195.80
0.752
0.264
351M/L4B
2M
18.37
362.58
0.952
0.256
Huge
scratch
3.6B
2M
18.58
382.17
1.053
0.256
finetuning
2M
17.52
302.28
0.988
0.264
frozen core
723M
2M
15.21
156.24
0.663
0.268
freeze-unfreeze
723M/3.6B
2M
16.17
231.94
0.683
0.262
UViT-Xlluge
freeze
1.2B
2M
15.32
152.12
0.571
0.269
freese-unfreese
1.2B/779B
16.58
222.38
0.620
0.266
Table 5: End2end variants trained on CC12M dataset at 512 x 512 pixels and batch size 256: image distri- bution metrics (FID, FD-Dino and CMMD). Smaller models benefit from finetuning all their parameters. Larger models have more capacity in the encoder-decoder layers, and benefit from freezing the pretrained representations, under such a small batch size regime.
DSG VqVa Question Types Attributes
model
steps
Entities
Relations
Global
DSG
UVIT-Base
scrutch
2M
73.16
53.91
62.31
$5.55
64.83
fineluning
2M
70.23
49.90
58.89
53.24
62.75
frozen
2M
69.57
49.36
58.22
53.39
61.16
freeze-unfreeze
2M
73.40
53.54
62.83
56.86
66.13
LargeLarge
scratch
2M
73.31
52.02
62.95
58.01
66.02
finetuning
2M
75.01
54.11
65.82
57.86
67.39
frozen
2M
78.97
61.55
67.19
61.40
72.13
freeze-unfreeze
2M
74.67
55.45
64.08
58.78
67.79
UViT-Huge
scratch
2M
74.33
55.02
62,98
58.63
66.90
finetuning
2M
77.29
56.40
67.13
62.56
69.67
frozen
2M
82.59
64.65
70.35
61.86
75.15
freeze-unfreee
2M
79.04
58,11
65.97
60.86
71.50
UViT-XHuge
frozen
2M
83.70
66.77
70.01
62.94
75.70
frecze-unfreeze
2M
81.14
60.44
69.40
60.25
73.53
Table 6: E2e variants at 512 512 pixels trained on CC12M dataset. Metrics evaluated on 1k samples from DSG-1k dataset. DSG results aggregated across semantic categories. Fine-grained results in Appendix B.
12

---

Published in Transactions on Machine Learning Research (10/2024)
20k steps
50k stops
100k steps
20k steps
50k steps
100k steps
Figure 6: On catastrophic forgetting during early steps of finetuning: the pretrained representation quickly deteriorates due to noise introduced by the random weights from newly added layers. (from left to right) 64 image produced by the pretrained Shallow-Unet-Hugc: followed by 512 512 images (in green) produced early steps of finetuning (ft.) the core representation in an E2e model; and (in blue) freezing the core layers. Differences better observed zooming ćn. Prompts: 'A loving mother kangaroo carrying her joey in her pouch.;; 'A close-up portrait of butterfly, revealing the intricate patterns and textures its wings in exquisite detail'; 'A playful wolf pup chasing its own tail'; 'A graceful hummingbird hovering near a bright pink flower': 'A dark and gothic illustration of raven perched on a skull':; 'A delermined sea turlle swimming against the ocean current?.
5.3 Guidance tuning
Diffusion model hyper-parameters affect both training and sampling quality. is common practice tune the sampler guidance weights using FID-CLIP score trade-off curves (Saharia et al., 2022a; Hoogeboom al., 2023b; Podell et al., 2024). In doing one aims to strike balance between images quality (by minimizing FID) and alignment with the text prompt (maximizing the CLIP score score). That said, it is well known that FID docs not correlate particularly well with human perception (Stein et al., 2023; Otani et al., 2023; Jayasunana et al., 2023), and large guidance weights are known to increase CLIP-Score but tend to produce over-sharpened, high-contrast images and unrealistic objects (Ho & Salimans, 2021; Saharia et al., 2022b). Due to such limitations, despite widespread use of FID-CLIP score scores for performance comparisons, practice they are adopted loose measure of performance, and guidance weights are typically set through qualitative inspection.
13

---

Published in Transactions on Machine Learning Research (10/2024)
000 233
60 0000
CM
0.270
200
-.85-0325
0.265
0.300
-15
180
0.260
-075 00315
Clip L14
177
0.255
FD-Dire
-0.70
0.310
DMMD
Clpp
0.250
+1
150
0.305
gu dence
Figure On the FID-CLIP tradcoff and the use of SOTA feature spaces for image and aext-allinment distributions. (left) In the process of optimizing guidance, there are clear tradeoffs between different metrics (Fréchet distance and Maximum Mean Discrepancy) taken under different feature spaces (Inception, Dino, and Clip) used to represent the image distributions. (right) qualitative illustrations: sample images taken with increasing guidance values from left-to-right and top-to-bottom. A sample taken under the guidance value that minimizes FID is highlighted with a red bounding box, whereas those using guidance optimum values for minimizing FD-Dino and CMMD are distinguished with green and yellow highlights, respectively. In cyan: the saturation /oaioooocccatoiihh effect of increasing CLIP score further in detriment of the other metrics. Differences better observed zooming in. Prompt (from MSCOCO captions): 'Two huskies hanging out of the car windows.'
Here explore alternative metrics for dyper-parametee tuning, aiming to better reflect their deployment use, and ultimately human perception. These include recent measures with alternative feature spaces that exhibit better robustness in classification tasks, and align somewhat better with human judgements of image quality and alignment. More specifically, we investigate the use of FD-Dino and CMMD as alternatives to FID in the calibration of the guidance hyper-parameter. Figure plots the response curve of different metrics as a function of guidance weight. They were measured using our UVIT-XHuge frozen model taken over 30k samples from the MSCOCO-caption validation set. It illustrates that the three image distribution metries minimized by very different guidance values. Similar curves are observed on the other models and training modalities, in which the best guidance value for minimizing FID, FD-Dino and CMMD are in increasing order. Figure 8 further illustrates samples obtained at the optimal values for each metric, and also when using the maximum guidance tested (16) for increasing CLIP score even further.
A qualitative analysis shows that by minimizing FID, one favors the generation of natural colors and textures, but under closer inspection, it fails to produce realistic object shapes and parts. We conjecture that this matches prior observations on the existence of texture vs shape bias by image classifiers (Geirhos et al., 2019). Guidance values minimizing Dino-v2 features, on the other hand, appear to produce natural color distributions and objects with natural shapes and composition. We adopt the value at this minimum as our new lower bound. Increasing guidance from that value tends to increase color-contrast and sharpening.
Images produced with guidance weights minimizing CMMD tend to produce images with initial signs of saturated colors and over-sharpening. Given its use of Clip features for image distribution comparison, this agroes with previous observations on CLIP score. But unlike CLIP score curves, CMMD curves present an inflection point within the range investigated. We use this inflection point to define a closed range for our search of reasonable guidance weights. That is, the range of guidance weights between FD-Dino and CMMD minimums was observed to strike a balance between producing correct shapes and aosthetically pleasing images characterized by enhanced color contrast and sharp edges.
All results presented in this section have their image generated using guidance weights within the FD- Dino/ Dino/CMMD CCMDD tradc-off range. The specific value selected was taken the intersection of the optimal ranges of models under the same comparison. Following this approach, our Shallow-UViT results were obtained with guidance weights fixed at 1.75, and their corresponding UViT models with guidance 4.0.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Di-Dino
CMMD
Figure 8: On the use different metric spaces for calibrating guidance. First three rows: guidance values taken for minimizing respectively FID, FD-Dino and CMMD. The use of robust features correlated with better shape and composition of images. Last illustrates the side-effects of further increasing guidance Prompts from MSCOCO caption dataset: 'A bathroom with sink and shower curtain with map print,;; 'A person holds a flip phone displaying the screen."; 'A motorcycle is parked dirt road in a forest.;; 'A stainless shiny serrated knife sits in front of sliced loaf.; 'A restroom hanging off the side of a building over a mounlain.'
and
6 A full diffusion pipeline: Vermeer
Vermeer is an 8B parameter model grown from 256 to 1024 pixel resolution. The UVIT architecture similar to our UVge-Huge model (Table 2), except that its bottom layers operate at a grid of 32x32 and with 32 transformer blocks total. We found that allocating transformer blocks at 32x scale improves details (like small faces). For Vermeer's text encoding, in addition to T5-XXL (Raffel al., 2020a) and Clip (Radford et al., 2021b) embeddings previously mentioned, also include By (Xue et al., 2022b) encoder with 256 sequence length, resulting in final embedding with sequence length of 461.
The baseline version (Vermeer raw model) is trained with 2k batch size at 256 resolution for 2M iterations, and grown to 1k resolution and finetuned for an additional 1M steps. As illustrated in Figure 1, it supports aspect ratios, i.c., 1024 x 1024, 768 x 1376, and 1376 x 768 thought aspect ratio bucketing (Anlatan, 2022). Once the raw model is trained, we apply the following extra steps to improve the aesthet of the generated images:
Style finetuning. We train an image classifier based on images that conform to aesthetic and compo- sitional attributes like those described in (Dai et al., 2023), and it to select 3k images from
15

---

Published in Transactions on Machine Learning Research (10/2024)
training data as fine-tuning set. We then fine-tune for 8K steps with mixture of the original data and the aesthetic subset. We condition the model on the aesthetic subset by adding a token to the text prompt. We found that finetuning the pixel model with a mixture of pretraining and finetuming data is needed to avoid catastrophic forgetting and to avoid the introduction of additional artifacts.
Distillation. The vanilla Vermeer model adopts 256-step sampling process, making it computation- ally expensive for real-world use. We employed the multistep consistency model (MCM) (Heek et al., 2024) to distill style-tuned Vermeer to 16 steps, achieving a substantial 16x speedup while maintaining high visual quality.
6.1 Vermeer results
We ablated four steps of Vermoer's dovelopment: (i) its raw model resulting from training on large dataset; (ii) the result of applying prompl engineering at inference to the same model, adding words to improve aesthetic image quality, but with no further training; (iii) the final model, after style finetuning on curated subset of 3k aesthetically pleasing images; and finally, (iv) its distilled, fast inference variation. Table 7 reports key performance metrics for all four variants, along with Stable Diffusion XL v1.0 (SDXL) (Podell et al., 2024). One can see that the raw model minimizes image distribution metrics that use state of the art feature space, i.e., FD-Dino and CMMD, while CLIP-score suggests a minor drop compared to SDXL. These metrics also highlight a significant shift away from the distribution of MSCOCO-captions (Chen et al., 2015), after augmenting the prompts prompl engineering) that further increased when combined with the finctuning of the model for aestheties plcasing imago( style finetuning).
The MSCOCO-captions. dataset comprises reference image-caption pairs covering diverse set of object categories and scenes. Thus, it offers interesting distribution for measuring image quality and text alignment due to the complexity and diversity of the compositions. At the same time, its use for visual quality preference assessment is spurious as its images were not curated with human aesthetics preferences. On the contrary, many the images have relatively poor aesthetic appeal. Thus, aiming to improve image acsthetics and composition, during Vermoer's prompt enginocring and style tuning phases we intentionally
model
D'D
F DD-Dinosos 4 CMMDa0v
CLIP acore
SDXL,1.0
13.19
185.57
0.898
0.279
Verneer
raw model
16.26
185.25
0.631
0.270
prospt engineering
17.33
216.01
0.867
0.269
+style tuning
24.51
336 25
1.167
0.262
distilldd
25.97
347.19
0.885
0.261
Table 7: Image distribution metrics evaluated on 30k samples of MS-COCO. The raw Vermeer model minimizes distribution metrics that adopt feature spaces from SOTA models (FD-Dino uses Dino-v2 while CMMD adopts Clip features), while tuning it to produce aesthetically pleasing images intentionally diverges from MSCOCO distribution.
DSG1 Attributes
model
Entitics
Relations
Global
DSG
SD2.1
75.44
53.06
69.66
68.49
71.23
Muse
77.65
60.64
75.61
67.18
73.09
Imagen Cascade
79.94
62.73
75.73
69.34
75.93
SDXL.1.0
88.04
73.00
78.48
75.19
81.47
Vermeer
raw model
86.92
76.36
76.48
68.49
80.77
+promp eng
87.94
74.92
76.31
67.41
80.99
+style tunning
88.04
74.21
77.38
69.57
81.16
+ disttlattion
84.71
69.23
72.68
65.49
76.88
Table 8: Vermeer. Broad and fine- fine-grained results
16

---

Published in Transactions on Machine Learning Research (10/2024)
Wemeerr prefrrn
Assthetic
61.4%
20.7%
17.9%
Consistecyy
26.5%
48. 48.7%
24.8%
41-
02
Aggregated
44.0%
34.7%
21.4%
70%
60%
50%
40%
30%
20% 10% ok 10% Percentage of Responses
20%
30%
40%
50%
60%
70%
Rade
(a)
aesthetics
(b) consistency
(c) final model 3-point Likert
Figure 9: Human evaluation results: Likert plot across 495 prompts, two tasks with 13 users each. Vermeer aesthetic is preferred during 61.4% of all comparisons, while its image-text consistency is marginally preferred. Aggregating the 1 annotations, Veermer is preferred during 44.0% of all comparisons, against. 21.0% from SDXL, Prompt engineering and style tuning aligned with human preference for visual aesthetics. Side by side qualitative comparisons in Figure 12 and Figure 13.
move the distribution of images generated by Vermeer away from MSCOCO-ccption distribution. To validate this we rely on human evaluation (in the next section).
The effect of the changes on the raw model with the CLIP-score and on semantic metrics on the other hand is minimal, aligned with our observation that the consistency of the model is not much affected by these two procedures. Semantie VqVa results presented on Table 8. The references to Imagen (Saharia et. al., 2022b) and Muse (Chang et al., 2023) models in this table are versions trained on internal data sources thus of similar resources and training pipelines than Vermeer. It shows that Vermeer presents competitive performance with SDXL, and surpassing the other models, inchuding auto-regressive and cascade models.
Finally, we also develop a distilled version of our model, in order to offer an alternative version with faster inference time that similar to the other models presented in this paper operates as a single, non-cascade end-to-ent end model at inference time. Figure 1 illustrates Vermeer outputs and additional qualitative results including comparison of samples from the full and distilled versions is presented in Appendix F.
6.2 Human evaluation
Assessing the performance of text-to-image models, ideally, depends on human evaluation, this complex cognitive process necessitates profound understanding of text and image relationships. Prior research has demonstrated that many recent works rely exclusively automated metrics, such as the Fréchet Inception Distance (FID). However, it has been observed that the current automated measures are not fully consistent with human perception in assessing the quality of text-to-image samples (Otani et al., 2023). Thus, to objectively access the quality of images generated by Vermeer, we conduct a side-by-side human evaluation comparing our model with SDXI. (Podell et al., 2024).
Setup. In this human evaluation, we ask annotators to evaluate generated images by Vermeer and SDXL based on the same prompt. For this, we collected 495 prompts covering a range of skills: 160 are from TIFA v1.0 targeting measuring the faithfulness of generated image its text input covering 12 categories (object, attributes, counting, eHu. et al., 2023); 200 are sampled from the 1600 Parti Prompts (Yu et al., 2022), selecting for both complexity and diversily of challenges; and 150 others are created fresh for, or are sourced from, more recent prompting strategies targeting challenging cases.
We create two tasks in which we instruct annotators to consider cither image quality (aesthetics) or fit to the prompt (consistency), and indicate their preferences using 3-point Liker scale: Vermeer is preferred, Unsure, and SDXL preferred (the model names are anouymized). The neural response includes cases that both images are equally good and bad. In the annotation UI, the annotators are shown a prompt along with two images that are randomly shuffled. We collected 13 human ratings per prompt for both aest and consistency (26 ratings per image).
We first sampled 510 prompts, 495 of them were usable after filtering incomplete samples.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Results. Prompt engineering and style tuning confirmed to have positive effect on human aesthetics preference (Figure 9, left), and small impact on text consistency (Figure 9, middle). They confirm our conjecture that the decrease on Vermeer's performance based on metrics grounded on the appearance of MSCOCC--cation dataset induced by these two stops are in alignment with the ultimate goal of human preference (Table 7).
Figure 9 (right) plots the Likert scale for our final model in each task (aesthetics or consistency) as well the aggregated responses (shown in in the bottom bar). Overall, annotators prefer Vermeer 44% of the time, while they select SDXL 21.4% of the time, with relatively fewer Neutral responses (34.7%). Vermeer is clearly preferred for its aesthetics, with a win rate of 61.4%, while the gap in consistency between the two models is small, with a difference in the win rate of just 1.7%. Krippendorff's a for aesthetics and consistency are 0.27 and 0.41, respectively, indicating moderate agreement among annotators.
7 Conclusion
We propose novel recipe for training non-cascaded large scale pixcl-space text-to-image diffusion models. It benefits from splitting their training in two phases representing different tasks: learning image-text condition alignment and learning to generate images at high-resolution.
We identified the model core components as those responsible for the first task and propose a proxy architec- ture (Shallow-UViT) to supports its pretraining. The second task is learned with greedy growing algorithm that stacks encoder-decoder layers of the final architecture top of the pretrained core components. When learning the second task, our training recipe preserves the core components representation from the noise introduced by the grown layers and their random initialized weights
Existing non-cascaded models training recipes struggle with scale, not supported with large batch size and further regularization like dropout and multi-scale loss. Our approach is able to train models up to 8B parameters with small batch size (256) and no further regularization, by pretraining the core components and preserving it during the second training phase targeting high-resolution generation.
Compared with training from scratch and finetuning, the greedy growing procedure is more stable, and improves performance on set of different metrics. Qualitative analysis shows that while keeping the core components representation stable it helps to preserve objects shape and overall structure, improving the definition of body parts. Our method allows use of data at different resolutions; the first phase benefits from the larger corpora with minimal requirements on image resolution, while the second phase learn to produce sharp images from the set filtered by the target resolution while reusing the representation learned from the larger set. We also explore models with increasing size, and show the benefits from scaling under different aspects and metrics.
In practice, the non-cascaded solution removes the oft-o--distiibution shift existent between training and deploying super-resolution phascs. Based on that, we present Vermoer, an 8B parameter Pixel based Test-to- Image Diffusion Model that produces high-resolution high-quality images using a single non-cascaded model. By training it on a larger dataset, and incorporating a final style tuning phase, Vermeer able to surpass SDXL v1.0 in human preference study.
References
Anlatan. Novelai improvements on stable diffusion, 2022.
URL https://bbog.noveel novelai net/
novelai- wtw.rrewweeeeeeoeataaaaeod--aaae
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mikk Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXio preprint arXiv:2211.01324, 2022.
Omer Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation, 2024.
18

---

Published in Transactions on Machine Learning Research (10/2024)
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et. al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall---8 pdf, 2(3):8, 2023.
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuaa Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Texl-to-image generation via masked generalive transformers. In Andreas Krause, Emnn Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, wtwpeeinnn..rr...ess/.e/e.e..eee ppess/v205 cch2g/2cca22bbbb/ html.
pp. 4055 4075. PMLR, 23 29 Jul 2023. URL https: https://proceeee http://///ooceeg...oio.e .lr.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Singlo-stage diffusion nerf: A unified approach to 3d generation and reconstruction. Tn ICCV, 2023. URL https:
//arxiv.org/aaaee 2rg/abs//304.067//
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. CoRR, abs/ 011/1544.0035,, 2015. URL http://ablp.miittrie...ee ChenFLVGDZ15.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Post-Tusct, and Su Wang. Davidsonian Scene Graph: Improving Reliability in Fino-Grained Evaluation for Text-to-Image Generation. In ICLR, 2024.
Hyungji Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL https: https://openrevviet net/fforrmnAe--0DD94AACTT.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Van- denhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2300.1500 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NewrIPS, pp. 8780-8794, 2021.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL attps://openreviee
Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A. Wichmann, and Wicland Brendel. Imagenet-trained CNNs biased Lowards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL httpss//openreee//
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and- play priors. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL / hwtp:///////oorrrrrr/eea///aaaaa
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint axXiv:2144008718, 2021.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6629 6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generalive Models and Downstream Applications, 2021. URL https://openreviiew.nn http:///ppparrrreeeeeeeetforrooo nddtrrruudd qw8AKxfYbI.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabillstic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Newral Information Processing
Systems, volume 33, pp. 6840 6851. Curran Associates, Inc., 2020. URL wtt.ttts:/pprroedangg neurips.
pdf.
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.
J. Mach. Learn. Res., 23(1), jan 2022a.
ISSN 1532-4435.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022b. URL https://
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In JCML, 2023a.
Emicl Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023b.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and intorpretable text-to-image faithfulness evaluation with question answering. In Proceed- ings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 20406 20417, October 2023.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint Xrxi:22212.11 11972, 2022.
Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. 2023.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 14938 14954. Curran Associates, Inc., 2021. URL https: http///proeeeenn..///e///e neurips.cc/paper files/ Paper. paf.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and San- jiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. arXiv preprint arXiv:2401.06600 2023.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13242-13254. Cur- ran Associates, Inc., 2021. URL. http://////o.c.e.....e... wtwpsssnuurrsc.ppppprrrrr cc/paper ftless/aperr/021//// pdf..
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL http://ppweerreeeeeeeeerrrrrtttp
20

---

Published in Transactions on Machine Learning Research (10/2024)
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Informa- tion Processing Systems, 2022. URL http://pppeeeeeeeeeeeeorrrrdddde
Kyuyoung Kim, Jongheon Jeong, Minyong An, Mohammad Ghavamzadeh, Krishnamurthy Dj Dvijotham, Jinwoo Shin, and Kimin Lee. Confidence-aware reward optimization for fine-tuning texl-to-image models. In The Tweifth International Conference on Leorning Representations, 2024. URL https://openrrviiei net/fforummid-Ieet000e00000
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object de- tection upon frozen vision and language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreriee..eert htt.:eewww....eetooooooddddd/tta
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupla, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang, Holistic evaluation of text-to-image models. In Thirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track; 2023. URL https: htap::://ppeee http///ppp/eeeeeee//e/e/ee////e/ http:/iwwwww.eeeeeeeeeeeeeeeeeee wwt.:ee///ooooeeee ceet/forum?id-iee /tt.formmmdd-gyyL77002222
Mark Levy, Bruno Di Giorgi, Floris Weers, Augelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. In NewrIPS, 2023. URL http://arxiv.org/ abs/2311.00613.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16270-16297, Toronto, URL
Canada, July 2023. Association for Computational Linguistics. doi: 10. /1633//1/2023..clooonn https://aclanthologg. org/2023.acI-oong 900.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πlya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. Xrv preprint (rXiv2211..07//4 2022.
Andreas Nieder and Stanislas Dehaene. Representation of number in the brain. Areual review of neuro- science, 32:185 208, 2009.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin'ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image genera- tion. In Proceedings 2023 IEEE/CVF Conference on Computer Vision and Pottern Recognition, CVPR 2023, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition, pp. 14277-14286. IEEE, 2023. doi: N0.1109/CVPR52229.00 STT.PR55729.222..1111 2022.01322222 Publisher Copyright: 2023 IEEE.; IEEE/ /CV Conference on Computer Vision and Pattern Recognition ; Conference date: 18-06-2023 Through 22-06-2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 1172-4182 2023. doi: 04110//TCCV55100.2222.0000
21

---

Published in Transactions on Machine Learning Research (10/2024)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXT; Improving Latent Diffusion Models for High-Resolution Image Synthesis. In ICLR, 2024.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference Learning Representations, 2023. URL httpss:/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 18-24 Jul 2021a. URL attps://proceedinn mlr. mlr. (aees//v199/aadforod22 html.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9748-8763. PMLR, 18-24 Jul 2021b. URL https: http:://proceeddngss mlr. html.
Desss//v39/radfofr
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21(140):1 67, 2020a. URL http://jmlr. org arg/papers/ v21/20-074. ht.mm..074.htm..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21(140):1-61 67, 2020b. URL http://jmlr. org/poperr.. v21/20-074.hte html.
Prajit. Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. orXiv preprinl arXiv:1710.05441 2017.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar 2xx::2224.06122 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Essor, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognétion, pp. 10684 10694, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Topes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understand- ing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL stttss///ppereviiie net/forumPid=00Y7 1512AA.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorcalistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 36479-36494 Curran Associates, Inc., 2022b. URL https://prooeeedingg weurips.cc/paper. cctpappr_fileseeeeeecceeeeeaaeee oo755ealta00722000ss55aaa0oo. watp:440co0---Pee---e Conferrr-oooeeeencee pdff.
Alessio Serra, Fabio Carrara, Maurizio Tesconi, and Fabrizio Falchi. The emotions of the crowd: Learning image sentiment from tweets via eross-modal distillation. arXiv preprint ar EXx::2944.19943 2023.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference Learning Representations, 2024. URL http::/ppperreeeeeeeeeooorrraaaa het::oormm...ddddhhcdddd..
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guideed diffusion models for inverse problems. In International Conference on Learning Representations, 2023. URL https:// openreview.oo www.ieewweeffooooo nww//eeuutidda-33 __g.A---K__MMAKKK
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Tiu, Anthony Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Advances in Neural Information Processing Systems, volune 36, 2023.
Vanessa Tan, Junghyun Nam, Juhan Nam, and Junyong Noh. Motion to dance music generation using latent diffusion model. In SIGGRAPH Asia 2023 Technical Communications, SA '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703140. doi: 10. (0.1145/361054433626168 URL https://doi 11111/00111////1553 36261644
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent corre- spondence from image diffusion. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttp:///////ereeeeeeemmomee www..neeoooonmddoooo.oo.oo
Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B. Tenenbaum, Fredo Durand, William T. Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL attps://
net pet/foruu/idrgg/xkkoop
Cristina Vasconcelos, Vighnesh Nandan Birodkar, and Vincent Dumoulin. Proper reuse of image classification features improves object detection. In Proceedings of the IEFE/CVP Conference Computer Vision and Pattern Recognition, pp. 13628 13637, 2022. URL https://openaccest thecvf.... com/
Proper_rpprrrr Reuse_of. of_ SaPFR_222___aaa___ paper pdf.
Shagee_Classfiicaitoo_eeeee Satrriiioneeeuueseeeeeeeee Ittr__nrprrv__o__eee
Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, and Aida Nematzadch. Revisiting text-to- image evaluation with gecko: On metrics, prompts, and human ratings. Under review (ECCV), 2024.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li, Ming Ding, Jic Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Linting Xue, Aditya Barua, Noah Constant, Rami A1-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel B5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291-306, 03 2022a.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-300 2022b.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Gen- eration, 2022.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chiel Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional CLIP. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (ods.), Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,
23

---

Published in Transactions on Machine Learning Research (10/2024)
New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers. nips. cc/paper.ppp.....pppp cctppape_ffiles/ files/ http:////ssctecceaoaoaaaaaaaaaaa
htmml
Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. What does stable diffusion know about the 3d scene?, 2023.
24


Reviews:
[{"review_id": "8poB2oOTlz", "reviewer": "Reviewer_cYDt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the general idea quite well motivated, and results convincing to see that the growing strategy works well. In addition, the authors have run many different experimental ablations to leave no doubt of its efficacy. \n\nThere are a few concerns, however. The underlying base model - what they call 'core' is a bit facile or obvious. They claim that it is analogous to the latent space of the VAE, but I find that an exaggeration. It looks like a generic extension of a pretrained backbone. Moreover, cascading or using hierarchical layers of pretrained backbone would only help (indeed, that is also done in latent versions of these models - e.g. hierarchical VAE [1]). \n\nI also think we should examine the time dimension, what happens inside the various diffusion steps. \n\n[1] https://arxiv.org/abs/2007.03898", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a way to implement super-resolution in diffusion models (text to image) through progressive growing strategies. The work is empirical, in that the algorithmic changes are at the network and training level. However, novelties in training strategy is important in complex setups such as diffusion models. \nThe training method is as follows. One first learns a 'core' components backbone that is common to all training setups. Following this, 'growing' is carried out for higher resolutions. The core is common to all trainings and skews, and does away with the cascading of multiple networks used generally in diffusion model training. \n\nThey train on the publicly available CC12M dataset. Following they present results on internal datasets trained with 8B parameters.\n\nEvaluations are carried out to show the effectiveness of pieces involved - backbone size, greedy growing, evaluation metrics (e.g. FID, CLIP-score), prompt engineering and style tuning, diffusion guidance. They also show different tasks - e.g. counting, VQA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above - It would be good to show what happens in the various diffusion steps. \nA latent version of this work would be useful to examine. I see no reason why we cannot implement this idea at the latent level. \nOn guidance - perhaps it might be possible to optimize all the scores in a learned way?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vCq9q8UkSR", "reviewer": "Reviewer_BAm2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n* A method which enables more efficient training of diffusion models, both in terms of computation as well as data, has clear interest and value to the community\n* Overall the text is clear and well written\n* The related works does a good job highlighting the current approaches and how the proposed method differs/builds on them.  \n* The variations and ablation studies explored are well motivated and thorough.\n* Use of, motivation for, and limitations of metrics are clear\n\nWEAKNESS:\n* The number of users involved in the evaluation is relatively small.  It would be good to significantly increase this to give the results more credence since this is the only comparative metric that exists.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method for training high-resolution, pixel-based image diffusion models.  The approach allows a single stage model to generate high-resolution images without a large high-resolution dataset or cascaded super-resolution components.  The generated images are preferred by human evaluators 2:1 over SDXL.  In comparison to cascaded models, this approach avoids the domain shift and propagation of irregular distortions in higher levels.  Their architecture, Shallow-UViT enables pretraining of the PSDM's core on text-image data as opposed to requiring a massive high resolution image dataset.  Their greedy algorithm enables the text-to-image model to be trained with small batch sizes.  The approach to frow the architecture produces better image distribution, quality, and text alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Typo in intro in first sentence- period between \"models. without\"\nTypo in \"Full pipeline model: ... \" reply --> rely\nBackwards quotes throughout\nFigure 3, Step 1.75M is not aligned in the top row\n\nNot a request, but a suggestion.  I would try to work in a picture from the full model earlier into the work.  When the reader sees the output of Figures 2/3, even for the xhuge models, it's not particularly overwhelming.  Once Figure 8 is presented, it's clear what the model is capable of.  I might move this forward (or make a smaller, related one), which captures what the model/approach is able to achieve.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Yy1c14S8G", "reviewer": "Reviewer_Mr9y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper clearly stated the issue of existing methods, including the requirement for large batch size, demand for high-quality high-res data, and inapplicability for tasks using diffusion models for image priors.\n    \n- The paper is well-structured, with nice figures and tables.\n    \n- The paper provides comprehensive ablation studies on the model size.\n    \n\nWeaknesses\n\n- The presentation of the methodology part is somewhat hard to follow. It could be improved with more use of formal notations and equations.\n    \n- The proposed Vermeer only beats SDXL v1.0 on human evaluation, which can be very subjective and random, especially considering there are only 13 ratings for each prompt. I also wonder why the authors do not show the generated images by Vermeer and SDXL on the same prompts.\n    \n- In the introduction section, the authors claim that cascaded diffusion models are not applicable to tasks where diffusion models serve as image priors. So, I wonder how the proposed model performs when serving as image priors for downstream tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with the challenge of training large-scale pixel-space text-to-image diffusion models (PSDM). The key idea is to decouple the alignment of image-text concepts from generating high-resolution images. Based on this insight, the authors propose a novel framework that enables pretraining PSDM’s core layers without requiring many high-res images. The authors further propose an algorithm to train the PSDM to generate high-res images. The resulting approach overcomes issues of existing methods and is able to achieve comparable performance with the current state-of-the-art models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses stated above.\n\nMinor:\n\n- Abstract: “…a remarkably simple greedy method for stable training of largescale, high-resolution models. without the needs for …”: remove the dot before “without”.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the response, all the reviewers lean toward acceptance. This paper has some merits: the proposed growing strategy is well-motivated and works well; the experimental settings are thorough; the presentation is decent. The AC has read the paper and agrees with the opinions of the reviewers. However,  the paper still should undergo a minor revision, as the reviewers suggest the authors include more side-by-side comparisons against the current SotA methods (with the same prompts).
Decision: Accept (poster)