Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models
Authors: name: Cristina N. Vasconcelos, name: Abdullah Rashwan Austin Waters, name: Trevor Walker, name: Keyang Xu, name: Jimmy Yan, name: Rui Qian, name: Shixin Luo, name: Zarana Parekh, name: Andrew Bunner, name: Hongliang Fei, name: Roopal Garg, name: Mandy Guo, name: Ivana Kajic, name: Yeqing Li, name: Henna Nandwani, name: Jordi Pont-Tuset, name: Yasumasa Onoe, name: Sarah Rosston, name: Su Wang, name: Wenlei Zhou, name: Kevin Swersky, name: David J. Fleet, name: Jason M. Baldridge, name: Oliver Wang

We address the long-standing problem of how to learn effective pixel-based
image diffusion models at scale, introducing a remarkably simple greedy growing
method for stable training of large-scale, high-resolution models. without the
needs for cascaded super-resolution components. The key insight stems from
careful pre-training of core components, namely, those responsible for
text-to-image alignment {\it vs.} high-resolution rendering. We first
demonstrate the benefits of scaling a {\it Shallow UNet}, with no
down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to
improve alignment, object structure, and composition. Building on this core
model, we propose a greedy algorithm that grows the architecture into
high-resolution end-to-end models, while preserving the integrity of the
pre-trained representation, stabilizing training, and reducing the need for
large high-resolution datasets. This enables a single stage model capable of
generating high-resolution images without the need of a super-resolution
cascade. Our key results rely on public datasets and show that we are able to
train non-cascaded models up to 8B parameters with no further regularization
schemes. Vermeer, our full pipeline model trained with internal datasets to
produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4%
human evaluators over SDXL.