Ultra-Resolution Adaptation with Ease
Authors: Ruonan Yu
Songhua Liu
Zhenxiong Tan
Xinchao Wang
Venue: ICML 2025 poster (2025)
Keywords: Ultra-Resolution Generation
Efficient Tuning
Synthetic Data
Diffusion Models
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We present key factors for adapting a pre-trained text-to-image diffusion model to ultra resolution with data and parameter efficiency.

Abstract: Text-to-image diffusion models have achieved remarkable progress in recent years. However, training models for high-resolution image generation remains challenging, particularly when training data and computational resources are limited. In this paper, we explore this practical problem from two key perspectives: data and parameter efficiency, and propose a set of key guidelines for ultra-resolution adaptation termed URAE. For data efficiency, we theoretically and empirically demonstrate that synthetic data generated by some teacher models can significantly promote training convergence. For parameter efficiency, we find that tuning minor components of the weight matrices outperforms widely-used low-rank adapters when synthetic data are unavailable, offering substantial performance gains while maintaining efficiency. Additionally, for models leveraging guidance distillation, such as FLUX, we show that disabling classifier-free guidance, i.e., setting the guidance scale to 1 during adaptation, is crucial for satisfactory performance. Extensive experiments validate that URAE achieves comparable 2K-generation performance to state-of-the-art closed-source models like FLUX1.1 [Pro] Ultra with only 3K samples and 2K iterations, while setting new benchmarks for 4K-resolution generation. Codes are available [here](https://github.com/Huage001/URAE).

Ultra-Resllutin Adaptation with Ease
Ruonan Yu
Songhua Liu
Zhenxiong Tan
Xinchao Wang'
Figure 1: High-resolutinn results by our method.
Abstract
trices outperforms widely-used low-rank adapters when synthetic data unavailable, offering sub- stantial performance gains while maintaining effi- ciency. Additionally, for models leveraging guid- ance distillation, such as FLUX, we show that disabling classifier-free guidance, i.e., setting the guidance scale to during adaptation, is crucial for satisfactory performance. Extensive exper- iments validate that URAE achieves compara- ble 2K-generation performance state-of-the-art dlseddsource models like FLUXI. [Pro] Ultra with only 3K samples and 2K iterations, while setting new benchmarks for 4K-resolution gener- ation. Codes are available here.
Text-to-image diffusion models have achieved remarkable progress in recent years. However, training models for high-resolution image genera- tion remains challenging, particularly when train- ing data and computational resources limited. In this paper, we explore this practical problem from two key perspectives: data and parameter efficiency, and propose a set of key guidelines for ultra-resolution adaptation termed URAE. For data efficiency, we theoretically and empirically demonstrate that synthetic data generated by some teacher models can significantly promote training convergence. For parameter efficiency, find that tuning minor components of the weight ma-
Eqqua contribution 'National University of Singapore, Singapore. Correspondence to: Xinchao Wang <xin- chao@ edd.e...g.
1. Introduction
Recent years have witnessed remarkable progress in text-to- image generation with diffusion models (Nichol Dhariwal, 2021; Dhariwal & Nichol, 2021; 1; Rombach et al., 2022; Ho et al., 2020). From -Nsee-based architectures (Ronneherger
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ultra-Resolutiin Adaptation with Ease
et al., 2015; Rombach al., 2022) to latest state-of-the- art Diffusion Transformers (DiTs) (Peebles & Xie, 2023; Bao et al., 2023; Chen et al., 2023; Esser et al., 2024; Li et al., 2024; Gao et al., 2024; Chen et al., 2024). these mod- els leverage powerful backbones and multistep denoising schemes to generate high-quality and diverse images from textual prompts effectively, solidifying their leading position in this field (Croitoru et al., 2023; Yang et al., 2023a).
it surpasses previous models in 4K generation performance and remains highly compatible with existing training-free high-resolution generation pipelines (Du et al., 2024b; Meng et al., 2021), enabling further performance improvements. In summary, the contributions of this paper are:
We are the first to delve into the problem of ultra- resolution adaption to the best of our knowledge;
Nevertheless, extending current diffusion models ultra- resolution generation, such as 4K, remains significant challenge. The process typically demands massive amounts of high-quality data and substantial computational resources, making training at such resolutions daunting and accessi- ble only to industry-scale efforts. Although recent attempts have been made train 4K-resolution text-to-image mod- els (Chen al., 2024; Xie et al., 2024), they rely on in- ternal datasets containing millions of high-resolution im- ages to fine-tune base low-resolution models. In practice, collecting such large-scale datasets for training is highly cumbersome if not infeasible at all. Meanwhile, tuning the entire diffusion backbone introduces an intensive GPU memory footprint, especially for state-of-the-art models like FLUX (Black Forest Labs, 2023) and Stable Diffusion 3.5 (Esser et al., 2024).
We propose URAE, a set of key guidelines focusing on data efficiency, parameter efficiency, and classifier- free guidance, to facilitate the adaptation of existing lext-to-image models to higher resolutions;
We validate that URAE achieves comparable perfor- mance in 2K generation, superior capabilities in 4K generation, and strong compatibility with existing training-free high-resolution generation pipelines.
2. Related Works
2.1. Text-to-Image Diffusion Models
The diffusion model (Ho et al., 2020) has emerged powerful class of generative models. Unlike traditional ap- proaches such as GANs (Goodfellow et al., 2014), diffusion models iteratively refine noisy maps with a UNet back- bone (Ronneberger et al., 2015) to produce high-quality and detailed images (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), which fuels significant advancements in large-scale text-to-imase diffusion models (Rombach et al., 2022; Podell et al., 2023; Balaji et al., 2022; Ding et al., 2022; Nichol et al., 2021; Ramesh et al., 2022; Razzhigaev et al., 2023; Xu et al., 2023; Saharia et al., 2022). Leveraging billions of image-text pairs, they demonstrate remarkable semantic understanding and the ability to generate diverse and photorealistic images aligning with text prompts.
Focusing on these drawbacks, we are curious about one prac- tical question: Can this ultra-resolution adaptation process be made easier? In this paper, we answer the question pos- itively by proposing URAE, set of key guidelines, under which ultra-resolution adaptation is achievable with merely thousands of training samples and iterations.
Specifically, we initiate our exploration from two key as- pects: data and parameter efficiency. On the one hand, we provide theoretical and empirical evidence that synthetic data produced by some teacher models can largely enhance training convergence. However, despite recent advance- ments in text-to-image generation, -httearf models still face significant challenges in acquiring high-quality synthetic training data for ultra-resolution adaptation, such as 4K. We thus, on the other hand, investigate such scenar- ios where synthetic data are unavailable and identify that tuning minor components of the pre- are-trained weight matrices more effective than commonly used parameter-efficient adaptation strategies like LoRA (Hu et al., 2022).
Most recently, Transformer (Vaswani et al., 2017) has been introduced as an alternative backbone to UNet (Peebles & Xie, 2022) in diffusion models, known as Diffusion Trans- former (DiT). Then, text-to-image models based on it have progressively demonstrated dominant performance (Chen et al., 2024; Esser et al., 2024; Gao et al., 2024; Li et al., 2024; Zheng et al., 2024). We thus focus on DiT-based models and conduct experiments mainly on FLUX, which yields state-of-the-an text-to-image performance, in sake for superior ultra-resolution adaption results.
Furthermore, we delve into the principles of fine-tuning guidance-distilldd models like FLUXX and discover that dis- abling classifier-free buidance-by setting the guidance scale to 1-is essential, regardless of the availability of synthetic data. Backed up by the above guidelines, we conduct extensive experiments to demonstrate that URAE achieves performance comparable to state-of-the-art closed- source models like F 1 [Pro] Ultra with merely 3K training samples and 2K adaptation iterations. Meanwhile,
2.2. High- figh-Resolution Generation
Training models at high resolutions demands substantial computational resources. To address this, series of works propose training- free solutions, developing inference stage strategies that allow diffusion models trained at their native resolutions to operate effectively at higher scales (Bar-Tal
2

---

Ultra-Resolution Adaptation with Ease
al., 2023; Meng al., 2021; Du et al., 2024b; He al., 2024; Du al., 2024a; Huang et al., 2024; Wu al., 2024; Zhang et al., 2023). While effective, without looking at any high-resolution images during training, in fact, they still fall short in accurately handling detailed structures and textures inherent in ultra-resolution images. By contrast, ultra-resolttion adaptation focused in this paper dedicates on addressing this drawback through training, which is tech- nically orthogonal training-free approaches and can work as plug-and-play component to enhance their performance.
of images a and their corresponding textual descriptions y is sampled. These images are then encoded into latent map 20 using pre-trained VAE encoder, and a noise map is drawn from a Gaussian distribution. Let 2 denote the noisy version of 2g after applying e at the 1-th diffusion timestep. The flow matching loss is then formulated as:
fim yr t) = II(e - =o)
(I)
where co(-) is the denoising backbone with parameters A.
The inference process begins with a text prompt y and ran- dom Gaussian noise e, also denoted er, which is iteratively denoised using the trained backbone. After steps, the re- sulting 20 represents a clean sample in the latent space and is then decoded to a generated image using a pre- tre-trained VAE decoder. Such training and inference paradigms are also employed in our URAE framework.
There are indeed some works training for high-resolution generation like 4K (Chen et al., 2024; Xie et al., 2024; Zheng et al., 2024; Ren et al., 2024). However, millions of high-quality training data and industrial-sall computa- tional resources are required to train the whole transformer backbone. In this paper, we focus the challenges of data and parameter efficiency and demonstrate that compa- rable or even superior performance can be achieved with significantly less data and fewer trainable parameters.
3.2. Synthetic Data or Real Data?
Previous works train 4K-generation models using millions of high-quality training images (Chen et al., 2024; Xie et al., 2024), leading to significant challenges in collecting, trans- mitting, storing, and processing such large volumes of data. To alleviate these inconveniences, we target a data-efficient approach for ultra-resolution adaptation.
Another line of research has concentrated on enhancing inference efficiency through the development of efficient and scalable diffusion backbones (Chen et al., 2024; Liu et al., 2024b;a). These designs and insights are orthogonal to our work, and it is promising to combine their strengths with our approach to achieve the best of both training and inference efficiency, which lies beyond the scope of this paper and is left for future exploration.
Building recent advances in the distillation of diffusion models (Yang et al., 2023b; Kim et al., 2023; Liu et al., 2024b;a), we recognize that incorporating a teacher model for reference and a loss term for knowledge distillation (Hin- ton, 2015) can enhance training:
2.3. Parameter-Efffiien Fine- Tine-Tunigg
In many real-world scenarios, fine-tuning existing models for specific applications is often necessary. However, fine- tuning all parameters can lead to substantial computational overhead, particularly in terms of memory footprint. To address this limitation, series of works propose parameter- efficient fine-tuning strategies (Hu et al., 2022; Syyeon-WWo et al., 2021; Meng et al., 2024; Yeh et al., 2023; Wang et al., 2024a). In this paper, we aim at an effective method specifically tailored for ultra-resolution adaptation.
Eaistal (zo: y,
t. U, ||<o(2,, t, v) Cores (2e, t, v
(2)
where fref represents the parameters of the teacher model. However, this approach relies on access to the diffusion backbone of the teacher model to compute the step-wise distillation loss, which is impractical for closed- weight mod- els such as FLUXI.1 [Pro] Ultra. We therefore experiment with an alternative approach that optimizes the vanilla flow matching loss defined in Eq. I using data synthesized by the teacher model. We expect this to yield similar training benefits, as validated by the following theoretical analysis.
3. Methodology
In this section, we delve into the motivations and technical details of URAE, our proposed strategy for ultra-resolution adaptation. We begin with some preliminary concepts, fol- lowed by three key components including training with syn- thetic data, parameter-efffcient fine-tuning strategies, and classifier-free guidance.
Before presenting our main results, we first set up some nec- essary assumptions. From the model perspective:
Assumption 3.1. Let u denote the input data pair (c. v). The process from to the output can be char- acterized by a neural network [u; W) with infinite width, where W denotes the network's 's parameters.
3.1. Preliminary
State-of-the--ar text-to-image diffusion models commonly adopt the flow matching training scheme (Esser al., 2024; Lipman al., 2022). Specifically, in each iteration, batch
It has been demonstrated that neural networks with a sin- gle hidden layer and sufficient width can approximate any
3

---

Ultra-Resolutee Adaptation with Ease
complex functions (Cybenko, 1989; Homik, 1991), and that infinite-widtt neural networks trained with gradient descent are equivalent to training linear models in the space of Neu- ral Tangent Kernel (NTK) (Jacot et al., 2018). From the data perspective:
W W B- 0.04
Ww W
00.54
W W
B
1.56
1.4
pi8.25
p-2.25
1.4
pib.1
p-4.5
1.
pet.
2-
1..0
Vi
0.6
0.2
The dataset contains both real and synthetic data, with the synthetic portion denoted as p [0. 1]. The total number of data is N. Given an input u, the corresponding target in the real data distribution is characterized by Treat fw; W*) + + where & ~ No, o²) is a scaler from a noise distribu- tion and 4 denotes the optimal parameters, serving as an unknown oracle. For synthetic data, the target is given by x eyyn = I [u; Wr where Wref denotes the parameters of pre-trained teacher model for reference. Without loss of generality, we restrict our discussion to the one-onnentt deedeeisiional target case.
Assumption 3.2.
1K 2< 3K 46
Iterations tons
5K
Ieerations
2K
3K
4K
56
3 2 36 4x Iterations
Figure A toy linear regression case. There are real data with noisy labels and synthetic data generated by reference model Wres- The proportion of synthetic data is p.
used for generating synthetic data and the optimal model, shown in the 2nd and 3rd terms of Eq. 3.4 separately. Fig. provides an illustrative example to visualize this effect.
Closely examining 3.,3.4, we can discover that, by dimin- ishing label noise, synthetic data would be helpful if the reference model providing these data is accurate. Although some works reveal that synthetic data can result in model collapse (Dohmatob et al., 2024b;a) by amplifying the gap between real and synthetic distributions, we demonstrate that they useful particularly for ultra-resolution adap- tion. On one hand, large-scale real datasets such as LAION- 5B (Schuhmann et al., 2022) tend to be noisy, containing numerous low-quality images and mismatched text-image pairs. On the other hand, at 2K resolution, models like FLUX-1.1 [Pro] Ultra- aetra-hghhugh closed-weight- are avail- able to produce high-quality synthetic data. Building on this analysis, we train our 2K-generation model using only syn- thetic data in this work, demonstrating superior performance across various scenarios.
And from the training perspective:
Assumption 3.3. The model's parameters are initial- ized as Wo- The training conducted through SGD using the square error as the loss function: L 2N (w; W) ==1 The learning rate is 17.
Our main results are summarized in Theorem 3.4:
Theorem 3.4. Under the setting defined in Assump- tions 3.1, 3.2, and 3.3, the error between Wr, the parameters after T training iterations, and the optimal W" is bounded by:
3.3. Tune Major or Minor Components?
Parameter-efficient fine-tuning strategies enable adapting pre-trained model from its original domain to target domain by integrating lightweight adapters. For instance, in personalized text-to-image generation such as Dream- Booth (Ruiz et al., 2023), attaching low-rank, e.g., rank r == 4, adapters, ie.,, LoRA, to the original model's weights can achieve satisfactory performance (Hu al., 2022). Specifically, this is achieved by:
N [ W [12] Bill(T 7/M)² Do 1011
17 (p((1 - 1 + - )))
Xi
+ >> W0
(3)
where Ao = (pW nat ( (1-p)W*), M is defined as V f(U; (W) T V (U; Wo). 5 f(u; Wref) f(u;; W*), and X; is the i-th eigenvalue of M.
== XW + XAB,
(4)
where X, Y, and W are input, output, and original weight matrices respectively, A e R xr and B gx are low-rank matrices for adaptation, and Cin and Cout are input and output dimensions, output dimension separately. In prac- tice, A is initialized using a normal distribution, whereas B is set to all zeros, which makes the adapter branch output zero initially and allows tuning to begin from the original parameters. After tuning, A and B can be merged into the original weight matrix via W = W + AB, ensuring that
The proof can be found in Appendix A. Intuitively, Theo- rem 3.4 indicates that, when training data are drawn from a mixture of real and synthetic data points, the distance to the optimal parameters convergence reflects trade- off-govereed by the synthetic portion y betweneen two fac- tors: the error introduced by label noise in the real data dis- tribution and the discrepancy between the reference model

---

Ultra- Citra-eesolution Adaptation with Ease
the total number of model parameters remains unchanged. These low-rank adapters employ a small number of param- eters that focus on the major components with the largest singular values, enabling efficient adaptation to the target domain (Meng et al., 2024).
cp. + y (Egr()) - (o)
(g.
cel
Original Model
Distilled Model
Training Tarpus
Zg
Already Matched
Proper Matching
However, different from DreamBooth modifying the styles and appearances of output images, ultra-resolution adap- tation focuses on learning the arrangements of details and local textures, which may not correspond to the major com- ponents in weight matrices. Under this hypothesis, we intro- duce a method to tune the components associated with the smallest singular values instead.
c0(1. > 1)
Co =
improper Matching
Figure 3: For CFG-distilled models, classifier-free guidance should be disabled in the training time. 24 and are omitted from the inputs of eg and cØ here for simplicity.
Specifically, given a weight matrix W E R XCast and min(Cin Cout), we first conduct Singular Value Decom- position (SVD) and derive W UEV, where U E Rei xe and V E Rex are orthogonal matrices, and Σ E Rex is a diagonal matrix with the singular values arranged from large to small. Then, r components with the smallest singu- lar values and the rest c - r ones are extracted via:
ing an additional "null-condition" branch. Specifically, at each denoising step in the inference time, the current latent map is processed by both the main branch and the null- condition branch, and the final output is then guided, with a certain strength, in the direction opposing the mull-condition branch's 's prediction:
W yemalll U[: -r [ [[rrr:[]rr :,:], Wree U|: -r|=| -r,: -r][: -r,
(5)
Eo(z,, + g (co(ze,t,)) - EZ( t, 0)),
(7)
where the indexing syntax in Numpy (Harris et al., 2020) and PyTorch (Paszke et al., 2019) are used to represent the operations for extracting multiple rows/columns. Analyzing from Eq. 5, Wamaal is a low-rank matrix. Therefore, for pa- rameter efficiency, we formulate the training time behavior similar Eq. 4:
where denotes the null condition, e.g., an empty prompt, and g is hyper-parameter controlling the strength.
Although effective, the additional null-condition branch doubles the inference cost. To address this issue, models like FLUX.. 1-dev use guidance distillation to train a dis- tilled model that takes the CFG scale embedding as an ad- ditional input, encouraging its output aligns with the result in Eq. 7. Since g is typically larger than 1 in inference, in training, many works also set g to the same value used at inference time during fine-tuning (XLabs-AI, 2024; Ten- centARC, 2024). However, according to the experiments, it results in inferior performance, especially in the problem of ultra- iitra-resolution adaptation.
Y XWres + XAB,
A = U|: - 2 -r
(6)
B = V Vir
A and B are initialized using Eq. 6 and updated during fine-tuning. In terms of formulation, the approach is similar to PISSA (Meng et al., 2024); however, it fundamentally differs by tuning the components with the smallest singular values instead of the largest. Although (Wang et al., 2024a) introduce a similar approach in the field of large language model finetuming, they fail to analyze its applicability in various scenarios.
Specifically, during the distillation stage. the distilled model is trained with g > 1 as input Eq. 7, which involves the null condition. In contrast, during the adaptation stage, the target is e 20 defined in Fq. 1, which is irrelevant the null condition. If y remains larger than 1, mismatch arises between the training targets in these two stages, making the training process more challenging.
Empirically, we observe that this approach is particularly effective when no synthetic data are available to serve reliable reference, e.g., in 4K generation. We speculate that the effectiveness stems from preserving the major compo- nents in the original weight matrices, thereby safeguarding the model's ' capacity to handle semantics, layouts, and ap- pearances from label noise in the real data distribution.
To address this issue, we note that incorporating the null- condition branch during adaptation is unnecessary; simply disabling CFG at training time by setting g = 1 works well and yields consistent target across the two stages. Fig. 3 illustrates the mismatch triggered by g >> 1 and how y = addresses the problem.
3.4. Enable or Disable Classifier-Free Guidance?
During inference, CFG is still necessary by using g > 1. Although the model does not encounter > 1 during adapta- tion, we find that generalizes sufficiently well in practice.
Classifier- guidance (CFG) (Ho& & Salimans, 2022) aims to enhance the quality of the generated samples by introduc-
5

---

Adaptation with Ease
Table 1: Quantitative results of the baseline methods and our proposed guidelines. The prompts are from HPD and DPG datasets. All images are at a resolution 2048 x 2048. Here, FLUX.I-dev' is FLUX. 1-dev with scaled RoPE, proportional attention, and removing dynamic shifting strategies.
HPD Prompt
DPG Prompt
Method/Setting
(J)
LPIPS (1)
MAN-IQA (
QualiCLIP (t)
HPSv2.1 (f)
PickScore (f)
Bench (t)
FLUXI.1 [Pro] Ultra
0.4129
0.6424
29.61
22.99
84.76
Real-ESRGAN
36.25
0.6593
0.4653
0.6392
30.70
22.91
83,50
SinSR
35.09
0.6566
0,4194
0.5556
30.95
22.96
83.79
SDEdit
35.59
0.6456
0.3736
0.4480
30.92
22.86
83,56
w/ URAE
34.07
0.6419
0.3872
0.5800
32.26
23.02
84.61
I-Max
33.66
0.6394
0.3670
0.4797
31.12
23.02
83.92
w/ URAE
32.24
0.6357
0.3833
0.5736
32.37
23.18
87.88
PixAnt-Sigma-I.
36.58
0.6801
0.2949
0.4438
30.66
22.92
80.60
Sana-1.6B
33.17
0.6792
0.3695
0.6718
30.92
22.83
85.14
FLUX.1-dev
43.78
0.6530
0.3821
0.3800
26.22
21.54
80.64
FLUX. I-dev"
34.86
0.6036
0.4110
0.5468
28.73
22.68
80.15
w/ URAB
29.44
0.5965
0.4730
0.7191
31.15
23.15
83.83
100
4K model, utilize 30K images with at least 4K resolution from the LAION-5B dataset (Schuhmann et al., 2022) and fine-tune the base model FLUX. Idev for 2K iterations on H100 GPUs, which takes days. In terms of training convergence, our method requires significantly fewer itera- tions compared with state-of-the ttfthh-art methods, such as 10K for SANA (Xie al., 2024).
94.7
$7.5
75.0
$1.0
25.0
WinRatee (%)
60
40
For baseline, apply URAE on the FLUX. 1-dev model and compare the performance with Airrt--iiga-X- (Chen et al., 2024), Sana- 1.6B (Xie et al., 2024), and FL.UX series models. In order to further demonstrate the effectiveness of URAE, we also apply URAE to the existing training-free high-resolution generation pipelines, i.e., SDEdit (Meng et al., 2021) and I-Max (Du al., 2024b). These pipelines require the base text-to-image model, e.g., FLUX. 1-dev, to generate low-resolution, e.g., 1024 1024, images as the guidance, and upscale these images to higher resolutions through image-to-ines image pipelines. For comparison, we also include the et al., 2021) and SinSR (Wang et al., 2024b), based on GAN and diffusion, respectively. We conduct quantitative exper- iments 2048 x 2048 samples generated with prompts from the HPD (Wu et al., 2023) and DPG (Hu et al., 2024) datasets. We evaluate FID and STLPIPS (Ghildyal & Liu, 2022) with the reference dataset generated by the FLUXI.1 [Pro] Ultra model. Furthermore, the quality of the generated images is assessed using metrics MAN-IQA (Yang et al., 2022) and QualiCLIP (Agnolucei et al., 2024). We adopt DPG Bench to measure the semantic consistency and coher- ence between the generated image and the corresponding prompt. Additionally, we use the HPSv2. (Wu et al., 2023) and PickScore (Kirstain et al., 2023) as human preference
20
Proepe Allganest
Vaszl Acathtics
PisAn--igna--
BLUX- der
PLUN- Lebe
PLAIKE (Pio)
Figure 4: GPT-4o preferred evaluation against current SOTA T2I models. We request GPT-4o to select a better image regarding overall quality, prompt alignment, and visual acs- thetics. Our proposed method are preferred against others.
methods Real-ESRGAN (Wang
4. Experiments
4.1. Settings and Implementation Details
In this paper, we adopt the open-source text-to-image FLUX.1-dev model (Black Forest Labs, 2023) as the base model to demonstrate the effectiveness of our proposed URAE guidelines, thanks to its superior performance. For our 2K-generation model, we collect 3K synthetic sam- ples with various aspect ratios generated by the F [Pro] Ultra model as the training dataset, and fine-tune the L.UXXI-dev on it for merely 2K iterations with a batch size of 8, which takes only day on 2 H100 GPUs. For our
6

---

Adaptation with Ease
Real-ESRCOE ESRGAN
SDEIL
SDEI (W URAE)
(w/ URAI)
Figure 5: Visualizations of our proposed method apply to training-free high-resolution generation pipelines. The prompt is A giraffe stands beneath a tree beside a marina.
1LUX1.1 [Pro] Ultra
PixArt: Sigma-XL
Sana-1.68
FLUX.1-dev
FLUX.1-dev*
URAE
Figure 6: Qualitative comparisons with baseline methods. All the images are of 2048 x 2048 size.
metrics to further evaluate the quality and aesthetic appeal of the generated images. Following prior works like Art Sigma and I-Max, we also utilize the GPT-4o to assess the generated images from prompt alignment, visual aesthetics, and overall quality key perspectives, at both 2K and 4K resolutions. These AI preference scores are derived from 300 randomly selected prompts in the COCO30K (Lin et al., 2014; Chen et al., 2024) dataset.
the state-of-the-art model FLUXI. [Pro] Ultra in terms of image quality, demonstrating its superiority in generating visually refined images. Moreover, the remarkable improve- ments in image quality further underscore the strength of our method in achieving state-of-the-ar visual results for all quality metrics, making it highly effective solution for nigh-resolution image generation tasks. In addition, our method also achieves a substantial improvement in the per- formance of the base model in terms of prompt alignment, improving the original LLUXX.--de by 3.19 in DPG Bench score, and 3.96 for I-Max pipeline.
4.2. 2K Resolution
Here, we evaluate the performance of the proposed methods on 2K images generated with prompts from HPD and DPG datasets. The results are shown Table 1. The quantita- tive results indicate that our proposed method is capable of significantly enhaneing the ability of models to generate high-resolution images and demonstrates its versatility and adaptability across different methods. Our method surpasses
For human preference study, adopt HPSv2.1 and PickScore to benchmark the human preference score. The samples are generated with prompts from the HPD dataset and the resolution is 2048 x 2048. The results are shown in Table 1. The results show that our method improves the human preference score of the base model, indicating that

---

Adaptation with Ease
9yn- Nagor Wb
1yn- Magor HFCG
Jei Mgor WeGGG
Radl Major w/CF3
8yn- Mina- WoCRG
9ym- Minr W/
Hool -inor- Wb
Hest Minrr ve CHG
Figure 7: Visualization results of ablation studies. The prompt is Imogen Poots portrayed as D&D Paladin in fantasy concept art by Tomer Hanuka.
Sixmat-
Sana-1.6B
F-Ux.1-dev*
URAE (Major4K)
URAE (Minor-IK)
Figure 8: Visualization results for ultra-resolution image generation task. All the images are of 4096 x 4096 size.
our proposed guidelines are capable of generating images that better align with human preferences.
minor components achieves outstanding performance when no synthetic data are available to serve as a reliable refer- ence for the altra-resolution image generation task, while the commonly adopted LoRA may fail on the overall semantics. Moreover, our proposed method also demonstrates excep- tional performance compared with other methods, further validating its competitiveness to existing approaches.
We also conduct preference studies with GPT-4o pair comparison regarding overall quality, prompt alignment, and visual aesthetics aspects. The results shown in Fig. 4. For the prompts for GPT-4o to assess the quality, prompt alignment, and visual aesthetics, please refer to Appendix B. The results demonstrate that our proposed method excels and is preferred in all three aspects. Please refer to Appendix C.I for more quantitative results.
Given that there is no well-defined benchmark specifically for 4K-ultra-resohlutonn generation, we conduct a user study using randomly generated prompts listed in Appendix B.2. For each prompt, we present results generated by the can- didates to and let them select the best one from three aspects including overall quality, prompt alignment, and visual aesthetic aspects. We collect 020 votes altogether. The results in Table 3 are coherent with the above analysis.
4.3. 4K Resolution
Here, we evaluate the performance of our proposed method in 4K-ultra-reae rultrui-eoolution image generation. The results are shown Table 3. From the experimental results, fine-tuning
8

---

Adaptation with Ease
Table 2: Ablation studies on three key guidelines, using real (Real) or synthetic (Syn) data, whether to adopt CFG in training, and tuning of major or minor components. Evaluations are on the 2048 x 2048 images generated from HPD prompts.
Method/Setting
FID (4)
LPIPS (J)
MAN-IQA (1)
QualiCLIP (t)
HPSv2.1 (1)
PickScore (1)
Syn w/o CFG
29.44
0.5965
0.4730
0.7191
31.15
23.15
Syn w/ CFG
76.07
0.6388
0.3992
0.5890
24.80
21.87
Major
Real w/o CFG
31.39
0.6076
0,4262
0,5953
29.33
22.86
Real w/ CFG
133.68
0.5978
0.3254
0.3645
16.55
19.92
Syn w/o CFG
27.90
0.5779
0.4558
0.6616
30.40
22.87
Syn w/ CFG
65.34
0.5858
0.3852
0.5312
23.77
21.64
Minor
Real w/o CFG
32.09
0.6000
0.4485
0.6098
28.71
22.61
Real w/ CFG
133.32
0.6026
0.3387
0.3672
16.36
19.68
Table 3: Evaluation on ultra resolution image generation task. The images are of 4096 x 4096, and generated with prompts randomly selected from COCO3OK. For user study, the prompts are randomly generated as listed in Appendix B.2.
COCO3OK Prompt
User Study
Method/Setting
MAN IQA (1)
Rank
QualiCLIP (1)
Rank
Overall Quality
Rank
Prompt Alignment
Rank
Visual Aesthetic
Rank
Pixart-Sgma-XLL
0.2935
0,2308
31.18%
30.88%
2
30.00%
Sama-1.6B 1.6B
0.3288
3
0.4979
10.29%
10.00%
3
12.06%
FLUX.1 1-DX..1dev"
0.3673
2
0,2564
4
3.24%
4
3.24%
4
3.24%
4
w/ URAE (Major-4K)
0.3280
0,2700
2.06%
1.76%
5
1.76%
(Minor-4K)
0.3999
0.5118
53.24%
54.12%
52.94%
4.4. Ablation Study
data and parameter efficiency, and provides a of useful guidelines. First, by incorporating synthetic data generated by some teacher models, we demonstrate the potential to promote training convergence and achieve high-quality out- comes even under data-scarce conditions. Second, for the cases where synthetic data are unavailable, we introduce a parameter-efficien fine-tuning approach to tune the minor components of weight matrices, which outperforms stan- dard low-rank adapters. Additionally, for models employing guidance distillation, e.g., FLUX, setting the guidance scale to 1 during adaptation proves crucial for achieving favorable results. Extensive experiments reveal that URAE matches the 2-ggnnertioo performance of leading closed-source solutions such as FLUXI. [Pro] Ultra using only 3K sam- ples and 2K iterations, and further sets new milestones for 4K-resolution generation.
To evaluate the effectiveness of our proposed method, we earry out ablation studies three key guidelines that we proposed in URAE, the source of training data, tuning major or minor components, and the adoption of CFG in training. The experiments are conducted on the 2048 x 2048 images generated from HPD prompts. The results are shown in Table 2, and the visualization examples are shown in Fig. 7. For the source of training data, the results demonstrate that high-quality synthetic data can provide better performance than noisy real data. When the model is fine-tuned with real data, tuning minor components can bring more vivid details as shown in Fig. 7. As for CFG, although it is necessary in the inference stage, it can lead to significant performance degradation in the training stage. According to these results, we by default use synthetic data and tune major components, i.e. adopt LoRA, 2K resolution, while use real data and tune minor components 4K resolution. In both cases, CFG is disabled in training and enabled during inference.
Nevertheless, the models presented in this work fall short of matching the inference-time efficiency exhibited by recent high-resolution text-to-image generation methods (Xie et al., 2024; Liu et al., 2024b;a; Chen et al., 2024), as we have not introduced architectural optimizations specifically targeting this aspect. In the future, we envision research aimed at streamlining the ultra-resolution generation process to bal- ance quality and efficiency requirements in practice. It is also meaningful to integrate our methods into multi-modal large language models to unlock even broader and more versatile capabilities.
5. Conclusions and Limitations
In this paper, we focus on the challenge of adapting text- to-image diffusion models from their native scales ultra- resolution settings with limited training data and computa- tional resources. Our proposed framework, URAE, tackles the problem from two complementary perspectives, i.e.,

---

Iltra-Resolution Adaptation with Ease
Acknowledgements
training of diffusion transformer for 4k text-to-image generation, 2024.
This project supported by the Ministry of Education, Sin- gapore, under its Academic Research Fund Tier (Award Number: MOE- T2EP20122-0006) and partially supported by NUS IT's Research Computing group using grant num- bers NUSREC-HPC-00001
Croitoru, F.A.,, Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Impact Statement
Our work on ultra-resolution text-to-image generation has broad implications for both research and real-world applica- tions. By pushing resolution boundaries, enable richer and more detailed visual content, benefiting domains such as digital art, virtual reality, advertising, and scientific vi- sualization. At the same time, these advancements high- light important ethical considerations. High-fidelity images could be misused for creating deceptive content, and the computational demands of large-scale generation can have environmental impacts. We therefore emphasize responsible development practices, including efficient training strategies and transparent model documentation, to help ensure that the benefits of ultra-resolution text-to-image models are realized while mitigating potential risks.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Ding, M., Zheng, W., Hong, W., and Tang. J. Cogview2:
Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing 2022.
Systems,
Dohmatob, E., Feng, Y., and Kempe, J. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07713 2024a.
Dohmatob, E., Feng, Y., Subramonian, A., and Kempe, J. Strong model collapse. arXiv preprint arXiv:2410.04844 2024b.
References
Du, R., Chang, D., Hospedales, T., Song, Y--Z., and Ma, Z. Demofusion: Democratising high-resolution image generation with no $$S. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6159 6168, 2024a.
Agnolucci, I.., Galteri, I.,, and Bertini, M. Quality-aware image-text aligmment for real-world image quality assess- ment. arXiv preprint arXiv:2403. 176,, 5(6), 2024.
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al. ediffi: Text-to-imaae diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
Du, R., Liu, D., Zhuo, L.,, Qi, Q. Li, H., Ma, Z., and Gao, P. I-max: Maximize the resolution potential of pre-trained rectified flow transformers with projected flow. arXiv preprint arXiv:2410.07536, 2024b.
Bao, F., Nie, S., Xue, K., Cao, Li, C., Su, H., and Zhu, All are worth words: vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-226677 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Bar-Tal, O., Yariv, L., Lipman, Y., and Dekel, T. Multidiffu- sion: Fusing diffusion paths for controlled image genera- tion. In Internntional Conference Machine Learning, pp. 1737-1752. PMLR, 2023.
Gao, P., Zhuo, L.., Lin, Z., Liu, C., Chen, J., Du, R., Xie, E., Luo, X., Qiu, L., Zhang, Y., et al. Lumina- -txinaa-txx::aa Transform- ing text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.055: 05945, 2024.
Black Forest Labs. Flux. https://gitain black seree-oorssteeee laasstllabs//e flux, 2023.
com/
Ghildyal, A. and Liu, F. Shift-tolerant perceptual similarity metric. European Conference on Computer Vision, pp. 91-107. Springer, 2022.
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang. Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310. 10..0226, 2023.
Goodfellow, I., Pouget- Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-a: Weak-to-strong
10

---

Aitra-Resolution Adaptation with Ease
Harris, C. R., Millman, K. J., Van Der Walt, J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., al. Array programming with numpy. Nature, 585(7825):357-22 -62, 2020.
Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang, Y., Liu, X., Huang. M., Xiao, Z., et al. Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine- tine-graneed chinese understanding. arXiv preprint arXiv:2405 20:5:44050000 00778, 2024.
He, Y.. Yang, S., Chen, H., Cun, X., Xia, M.. Zhang. Y., Wang, X., He, R., Chen, Q. and Shan, Y. Scalecrafter: Tuning-free higher-resolution visual generation with dif- fusion models. In The Twelfth International Conference on Learning Representations, 2024.
Lin, T.-Y., Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02533 2015.
Lipman, Y., Chen, R. T., Bam-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2100002747, 2022.
Ho, J. and Salimans, T. Classifier- free diffusion guidance. arXiv preprint arXiv:2277.12596 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851 2020.
Liu, S., Tan, Z., and Wang, X. Clear: Conv-like linearization
revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412. 2024a.
Homik, K. Approximation capabilities multilayer feed- forward networks. Neural networks, 4(2):251-251 1991.
Liu, S., Yu, W., Tan, Z., and Wang, X. Linfusion: gpu, minute, 16k image, arXiv preprint arXiv:2409.02099 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview net/ forum? dww.mmmeededdeeeeeeee..
Meng, C.,, He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. arXiv preprint arXiv: 1rXiv:210.001077 2021.
Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.055 135,, 2024.
Meng. F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404 02948, 2024.
Huang, L., Fang, R., Zhang, A., Song, G., Liu, S., Liu, Y., and Li, H. Fouriscale: A frequency perspective on training-free free higl-resouution image synthesis. arXiv preprint arXiv:2403. 12963, 2024.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint a1X1v:211..10741, 2021.
Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H. Fedpara: Low- rank hadamard product for communication-effiienn fed- erated learning, arXiv preprint 2 200. 000.666 2021.
Nichol, A. Q. and Dhariwal, Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162 8171. PMLR, 2021.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, I., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Kim, B.-K., Song, H.-K., Castells, T., and Choi, S. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. In Workshop on Efficient Sys- tems for Foundation Models@ ICML2023, 2023.
Peebles, W. and Xie. S. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748. 2022.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user prefer- ences for text-to-in to-ittggeeeaaaa generation. Advances in Neural Information Processing Systems, 36:36652-3666 -36623, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.

---

Ultra- Stra-Reesolutin Adaptation with Ease
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, H., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter- efficient IIm finetuning. arXiv preprint arXiv:2406.09044 2024a.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Razzhigaev, A., Shakhmatov, A., Maltseva, A., Arkhip- kin, V., Pavlov, I., Ryabov, I., Kuts, A., Panchenko, A., Kuznetsov, A., and Dimitrov, D. Kandinsky: an improved text-to-image symthesis with image prior and latent diffu- sion. arXiv preprint arXiv:2310.03500 2023.
Wang, X., Xie, I.., Dong, C., and Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure syn- thetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1905-1914, 2021.
Wang. Y., Yang, W., Chen, X., Wang. Y., Guo, L.,, Chau, L.-P., Liu, Z., Qiao, Y., Kot, A. C., and Wen, B. Sinsr: diffusion- based image super-resolution in single step. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25796-25805, 2024b.
Ren, J., Li, W., Chen, H., Pei, R., Shao, B., Guo, Y., Peng, L., Song, F., and Zhu, L. Ultrapixel: Advancing ultra- high- high-resolution image synthesis to new peaks. arXiv preprint arXiv:2407.02155, 2024.
Wu, H., Shen, S., Hu, Q.. Zhang, X., Zhang, Y., and Wang, Y. Megafusion: Extend diffusion models towards higher- resolution image generation without further tuning. arXiv preprint arXiv:2408. 1001, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICC. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Xie, E., Chen, J., Chen, J., Cai, H., Lin, Y., Zhang, Z., Li, M., Lu, Y., and Han, S. Sana: Efficient high- high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410. 10629, 2024.
Ruiz, N., Li. Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image dif- fusion models for subject-driven generation. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recogmition, pp. 22500-22511 2023.
XLabs-AL.
x-flux.
https: http:/gttthhc..ommm
XLabs AI/x AI/x-flux, 2024.
Xu, X., Wang. Z., Zhang, G., Wang, K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7754-7765, 2023.
Saharia, C., Chan, W. Saxena, S., Li, L., Whang. J., Denton, E. L.,, Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479-36494, 2022.
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023a.
Yang, S., Wu, T., Shi, S., Lao, S., Gong, Y., Cao, M., Wang, J., and Yang, Y. Maniqa: dulti-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1191-1200, 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
Yang, X., Zhou, D.,, Feng, J., and Wang, X. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 22552-22562. 2023b.
TencentARC. Fluxkits. https://githut. com/ TencentARc/Fluxxits. 2024.
12

---

Ultra-Resolution Adaptation with Ease
Yeh, S.-Y., Hsieh, Y.-G., Gao, Z., Yang, B. B., Oh, G., and Gong, Y. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023.
Zhang. S., Chen, Z., Zhao, Z.,, Chen, Z., Tang, Y., Chen, Y., Cao, W., and Liang, J. Hidiffusion: Un- locking high-resolution creativity and efficiency in low- resolution trained diffusion models. arXiv preprint
11X152 2023.
Zheng, W., Teng, J., Yang, Z., Wang, W., Chen, J., Gu, X., Dong, Y., Ding, M., and Tang, J. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint 4:X1112033.051 2024.
13

---

Ultra-Resolution Adaptation with Ease
A. Theoretical Proof
supplement the proof of our main theoretical results in Theorem 3.4 here.
Theorem A.1. Under the setting defined in Assumptions 3.1, 3.2, and 3.3, the ervor between WI, the parameters after T training iterations, and the optimal W is bounded by:
(1 (1-1)))) ?),)T)²
WT 11
<BII(1 ~ Do (p(((1))) (1 p)o
A
(8)
+ +p'lWres - 1
where Do Wo (p-- M is defined as Vw Vw 8 = ((u: and X4 is the i-th eigenvalue of M.
-1
Proof. Under the assumption of -iitt neural networks, the network output f(u; W) can be viewed as the following linear form with respect to the parameter W:
f(u W) R Wo) W (lu; - -W)-
We denote W (WW; u) as & for simplicity. According to the loss function L of L with respect to W is:
2N [u: W) - ==13 and Eq. 9, the gradient
1
VwC 1 N
x
Σ 1 [D( Wo) f(u:Wo x)]
(10)
Training is conducted using SGD:
W++ = We - WW We
N Σ + [WW,, - Wo) + f(u; Wo) - =x]}}.
(11)
Due to the linearity, the optimal parameter W when training on a mixture of real and synthetic data given by:
w pWref + (1 p)W
(12)
The target x can be viewed as the output of (w; W with noise term gl:
f(u; W + +
(13)
Then analyze the mean and variance of Ege. According to Eq. 13, g satisfies:
f(u; W-);, ((u, - f(u; w*'),
with probability p,
(14)
with probability 1 - p.
Given the linearity and Eq. 12,
flu; ((;:W") = pf Wref) (1 p)((u:
(15)
Denote [1; Wres) [l; W*) as 6. Then,
5' 8'= (1 - p)o,
with probability p. with probability - p.
(16)
-pó +
Since & o2), the expectation of g is:
E[g"] p(1 - p)8 + p)(-p5)
(17)
14

---

Adaptation with Ease
And the variance, denoted as Eg,, is computed as:
Ee' = EE("2 = p1 [) + p)p E) p(1 p) +1 -
(18)
Let M denote & ① DDD, where D is the number of parameters in the network, and A, denote We W Combining with Eqs. 11 and 13, we obtain:
W4+1 = (W* + Dc) - "
Σ f 11oT1014 [A (W* - Wo)) + f(u; - (f(u; =
(19)
Note that:
(w): - f(u; Wo) D(W" -Wo).
(20)
We then have:
W411 = (W + A1) - 17 T 100, + E7 = + 11I - rpb (apTT)A, + t706
(21)
Thus,
4 = (I - nMMA, 7
(22)
Starting from Ag, after 7 iterations, according to Eq. 22, we can obtain the explicit expression of Ay:
T-1 -M-n
AT (I - 1--MM)a D +1
(23)
Initial caror docay
k==
Label noise accumulation
Now, we are interested in 1T which contains quadratic terms of initial error decay and label noise accumulation as well as their cross term. Since label noise is independent of error caused by initializing model parameters and E[s'] = 0, the cross term 0. Thus,
B - V 1113 = BiI(II - (7M))Doll + FE
(1
7 Tel
(24)
The first term is related to the initial error. For the noise term, let Ar be the cumulative sum of the updates:
T-1 AT = Σ n)-
(25)
k=
Then:
T-17-1 C TE 4(I nM)s Ar T k= 0=
T
(26)
Since the noise covariance Eg is scalar, using M T tb, get:
E 2 Te
Eg (AT
(27)
Using the matrix geometric series sum formula:
Ar (r=
(28)
where denotes the Moore-Penross pseudoinverse. Substituting.
My - -I- nat))aart
(29)
Taking the trace:
= (T (T-
(30)
15

---

Ultra- Reraa-eeoottion Adaptation with Ease
Let M have eigenvalue decomposition:
M VAVT, A = d (.,, Ap)-
(31)
Note that for infinite-wide width network, D > N. Thus, X, = 0 for N < <is D. Then:
(1 - (--nM)ar = - nARTT..
(32)
Therefore:
N
Tr((--M)) = Σ 11 - TT
(33)
1=1
Similarly,
Σ (1- (1-(1-nA))) n)
Tr Tr|M -I----nnnn)))])
(34)
Thus, using Eq. 18, the result of B| || is:
(1
W W 11:1 [I M² Aol + p) +1 p)o
(35)
By triangle inequality, we have:
Wr
[|]
< Wr W w+11]] + | Wo
Wr w 11]+ + re + (1 p)W - 1/
N
n),
(36)
=Bil([I -n A 3 + +1= + (1 2)
A
+ + - W
B. More Experimental Details
B.1. Prompts for AI Preference Study
To better compare the quality of generated images, we employ GPT-4o as the evaluator, assessing methods from three aspects: overall quality, visual aesthetics, and prompt alignment. The evaluation involved both pairwise comparisons and quantitative analysis. During the evaluation, for pairwise comparison, GPT-4o compares our method with the baseline methods, selecting the more preferred image. For quantitative analysis, GPT-4o assigns scores (0-100) to each image generated by each method. The prompts used in our testing are listed below, designed following the previous work PixArt-Sigma (Chen et al., 2024). For pairwise comparison, the designed prompt to evaluate the overall quality of images is as follows:
As an Al visual assistant, you are an evaluator specialized in image quality analysis for high-resolution text-to-image generation models. Given a specific caption, please evaluate the overall quality of the image by considering both content alignment and technical excellence. For content alignment, assess the key information including object identities, properties, spatial relationships, object numbers and caption-specified style. For technical quality, evaluate the image's photorealism and aesthetics, focusing on clarity, richness of detail, artistic quality, and overall visual appeal. Please analyze how well the image performs in both aspects to determine its comprehensive quality, the prompt is "your prompt". Please output [Image 1] if the first image is better, [Image 2] the second image is better, and give me the reason.
The designed prompt to evaluate visual aesthetics of images as follows:
16

---

Ultra-Resolution Adaptation with Ease
As an Al visual assistant, you are an evaluator specialized in image quality analysis for high-resolution text-to-image generation models. When presented with specific caption, is required to evaluate and determine which image exhibits greater photorealism and aesthetical, in terms of clarity, richness of detail, and overall quality. Please pay attention to the key factors, including image style, the artistic quality of the image, realism, etc., the prompt is "your prompt". Please output [Image 1] if the first image is better, [Image 2] if the second image is better, and give me the reason.
The designed prompt to evaluate the prompt alignment of images is follows:
As an Al visual assistant, you are an evaluator specialized in image quality analysis for high-resolution text-to-image generation models. Given a specific caption, you need to judge which image aligns with the caption more closely, Please pay attention to the key information, including object identities, properties, spatial relationships, object numbers and image style, etc., the prompt is "your prompt". " Please output [Image 1] if the first image is better, [Image 2] if the second image is better, and give me the reason.
For quantitative analysis, the designed prompt to evaluate the overall quality as follows:
As an AI visual assistant, you specialize in evaluating image quality for high-resolution text-to-image generation models. Given a specific caption, please evaluate the overall quality of the image by considering both content alignment and technical excellence. For content alignment, assess the key information including object identities, properties, spatial relationships, object numbers and caption-specified style. For technical quality, evaluate the image's photorealism and aesthetics, focusing on clarity, richness of detail, artistic quality, and overall visual appeal. Please analyze how well the image performs in both aspects to determine its comprehensive quality. The prompt is: "your prompt". Please output strictly the score from 0 to 100. Do not provide any explanation or additional text beyond this numeric score.
The designed prompt to evaluate the visual aesthetics of images is follows:
As an AI visual assistant, you specialize in evaluating image quality for high-resolution text-to-image generation models. When given specific caption, you are required to assess the image and assign 0-100 score, reflecting its photorealism, aesthetic appeal, clarity, richness of detail, and overall quality. Key factors to consider include image style, artistic quality, and realism. The prompt is: "your prompt". Please output strictly the score from 0 to 100. Do not provide any explanation or additional text beyond this numeric score.
The designed prompt to evaluate the prompt alignment of images is as follows:
As an Al visual assistant, you are an evaluator specialized in image quality analysis for high-resolution text-to-image generation models. Given specific caption, you need to determine the score 0-100 that the image aligns with the caption. Please pay attention the key information, including object identities, properties, spatial relationships, object numbers and image style, etc., the prompt is "your prompt". Please output strictly the score from 0 to 100, reflecting how accurately the image aligns with the caption. Do not provide any explanation or additional text beyond this numeric score.
B.2. Prompts Used in User Study
1. Craft image depicting surreal dreamscape with a majestic unicor floating amidst tumultuous waves, viewed from an aerial perspective akin to observing through a bird's eyes soaring above the sea. This scene captures both serene beauty and chaotic turbulence in one fantastical landscape. Utilize vibrant, contrasting colors, featuring deep blues for the stormy sea and fiery oranges and purples for the swirling clouds overhead, creating an emotional gradient that evokes wonder, danger, and ethereal grace simultaneously.
captivating Art Nouveau Nouveau-inspired image showcases a celestial enchantress gracefully dancing amidst a swirling vortex
17

---

Ultra- Aitra-Resolution Adaptation with Ease
of shimmering stardust, her ethereal gown intricately woven with delicate silver threads reminiscent of cosmic nebulae. Captured from an elevated perspective that accentuates the vastness and grandeur of the cosmos, this scene radiates a sense of wonder, enchantment, and serenity as it invites viewers to marvel at the luminous beauty of the muse against the backdrop of the infinite expanse.
3. A surreal digital artwork depicting an enigmatic floating cityscape composed of inverted ziggurats suspended in midair. From an aerial perspective, the city appears to hover over an endless expanse of rippling water. As one draws closer to the water's s surface, the reflection reveals a mirrored image of the city, but with its architecture twisted and distorted by the shifting tides. The vibrant colors and intricate details evoke a sense of wonder mixed with unease, inviting the viewer to contemplate the relationship between reality and illusion.
4. A surreal digital artwork depicting a bustling futuristic cityscape at night, with towering skyscrapers adorned in vibrant, abstract shupes. The scene transitions from sharp clarity to soft, dreamlike almosphere as it approaches the horizon, evoking both awe and uncertainty in the viewer. Neon lights pulse within the city, seemingly melting and warping in the air, creating mesmerizing patterns and reflections on the wet streets below. From a high-altitude perspective, this enigmatic metropolis is captured in an aerial view, inviting contemplation of the convergence of reality and dreams.
5. In the style of visionary art, depict a serene female figure draped in a luminous white gown with intricate mandala patterns in deep blue and vibrant teal hues. This ethereal portrait is viewed from an aerial perspective, showcasing the subject against a cosmic background that seamlessly blends into swirling galaxies and nebulae. The radiant colors and harmonious compositions evoke a profound sense of spiritual awakening and intercoonectedness with the vast universe around us.
6. In the style of visionary art, depict a serene female figure draped in a luminous white gown with intricate mandala patterns in deep blue and vibrant teal hues. This ethereal portrait is viewed from an aerial perspective, showcasing the subject against a cosmic background that seamlessly blends into swirling galaxies and nebulae. The radiant colors and harmonious compositions evoke a profound sense of spiritual awakening and interconnectedness with the vast universe around us.
7. A dreamlike landscape emerges from a first-person viewpoint, immersing the observer in an alluring world where waterlilies of soft lavender and violet hues gracefully drift on the surface of an opalescent pond. Towering lotus blossoms stretch lowards an indigo sky embellished with celestial bodies that gleam like stars, invoking both tranquility and awe. A regal swan presides over this fantastical garden, its iridescent feathers creating captivating ripples across the water that seem to distort time itself, crafting a harmonious melody of dreams and nature, encapsulating the spirit of beauty and whimsy in one stunning tableau.
8. Envision otherworldly aquatic environment where a graceful mermaid adorned with iridescent scales reminiscent of deep-sea hues gracefully dances amidst vibrant coral formation.. Her tranquil expression mirrors state of contemplative introspection, as if she is enveloped in the enigmatic depths from an unconventional vantage point that of a diminutive seashell. Delicate intricacies emerge, such as her cascading tresses mirroring tender seaweed and how sunlight weaves through the water to cast enchanting patterns upon her skin. This captivating scene instills a sense of awe and reflection while preserving an ethereal aura reminiscent of surrealistic art.
9. Craft an enchanting surrealist scene showeasing a ''bug' s-ee view' perspective of a chess game occurring on a shifting landscape. In this scene, a majestic phoenix perches atop a black bishop, its vibrant wings casting intricate shadows across the checkerboard expanse below. The background alternates between lush tropical forests and barren deserts with each move made by the ethereal beings participating in this mysterious match, instilling feelings of intrigue and wonder as they traverse the umpredictable terrain under the eerie illumination of a full moon.
forests
10. Imagine an enchanting digital artwork depicting a dragonfly's 's perspective above a tranquil pond, reminiscent of Moner's ss captivaling water lilies. The scene showcases lush, vibrant vegetation surrounding the serene water surface, with an emotional gradient highlighting the beauty and enchantment as light gracefully dances across the composition. Merging elements of impressionism with high-resolution photorealistic textures, this piece evokes a sense of awe and wonder, creating an ethereal atmosphere that captures the dragonfly's mesmerizing flight amidst blooming floral paradise.
11. Imagine a lively digital painting capturing the exhilarating spirit of an tniim-innspired hoverboard race in neon- drenched cyberpunk cityscape at twilight. The scene pulses with action as diverse characters navigate through maze-like urban landscape of towering skyscrapers and radiant billboards, deftly maneuvering around airborne vehicles and zipping pedestrians while leaving trails of shimmering pixels behind them. As the sun dips below the horizon, its dramatic lighting casts elongated shadows that accentuate the futuristic architecture and high-tech trinkets sprinkled throughout, creating rich tapestry of cool blues, purples, and pinks that intensify the emotional stakes and anticipation as the race nears its peak. This captivating image, with its dynamic composition and vibrant use of color and light, transports viewers into an enthralling world where technology and nature intertwine in a dazzling display of innovation and style.
18

---

Adaptation with Ease
12. An enigmatic digital artwork showcases an celestial ballerina gracefully spinning across the cosmos, her luminescent dress shimmering with iridescence against the inky black backdrop of space. The scene is observed from a unique 'worm's s eye view', accentuating the grandeur and elegance of this otherworldly dancer as she whirls amidst nebulae and stars, creating an entrancing spectacle that captivates both the senses and the imagination while evoking sense of wonder and awe for the hidden beauty within the universe.
13. A mesmerizing digital artwork portraying an enchanting celestial sorceress in shimmering robes adorned with starlight- inspired patterns. Seen from above, her captivating gaze reflects the light of distant galaxies as she skillfully shapes the cosmos with her mystical staff. creating swirling nebulae and brilliant constellations in a breathtaking display. The backdrop showcases deep purple expanse of space filled with nebulous clouds, pulsating stars, and enigmatic planets, generating an emotional gradient that balances wonder and mystery within this celestial tableau.
14. Craft image that embodies spiritual realism with celestial elements, depicting an astronaut in reflective spacesuit adorned with intricate mandala patterns. This ethereal figure floats amidst the cosmic void, traversing through a luminous wormhole that connects our physical world to a higher plane of existence. The astronaut's 's journey is captured from an eye-of-the-needll perspective, enveloped by vivid colors and profound symbolism that evoke wonder, transcendence, and spiritual awakening in the vast expanse of space.
15. A celestial ballet unfolds within an ethereal nebula as galaxies collide in a mesmerizing dance of cosmic cosmic From an impossible vantage point, suspended above the event horizon, we gaze upon this breathtaking spectacle. The scene captures the chaotic beauty and sublime mystery of the universe, with swirling patterns of stars, nebulas, and cosmic dust illuminated by an otherworldly light source that casts long shadows across the celestial plane. This visually stunning composition evokes a sense of awe and wonder at the sheer scale and complexity of our cosmos, as well as the delicate balance between order and chaos in the grand design. Inspired by the visionary artistry of Escherian architecture and the cosmic explorations of space telescopes, this image invites viewers to ponder the infinite mysteries that lie beyond the stars.
16. Create a dreamlike still life scene blending impressionistic water lilies with surreal melting elements. Picture a tranquil garden pond adorned with softly ruffled blue, pink, and green flowers, glowrs, as if captured in ethereal dance of light and color. In the background, a tower melts into the distant horizon, its hands frozen in time, evoking a sense of timeless wonder. The reflection on the water's 's surface distorts the landscape into mesmerizing shapes and colors, blurring the line between reality and fantasy, enveloping the viewer in an enchanting atmosphere of surreal beauty and awe.
17. An awe-inspiring image reveals an enigmatic blend of swirling celestial patterns reminiscent of van Gogh's 's iconic brushstrokes and the surreal melting clocks synonymous with Dali's dreamscapes. Set within the opulent interior of a vast library, the scene captivates the viewer's gaze as they stand amidst towering shelves adorned with golden-hued spines. Above, an intricate celestial map unfolds across the ceiling, its constellations echoing van Gogh's 's dynamic style while clocks dissolve into nebulous forms that cast whimsical shadows over the polished marble floors. This surreal tableau masterfully combines wonder and introspection, inviting the observer to embark on journey through time and space within its harmonious visual symphony.
18. A captivating digital artwork merges Victorian-era street market with an underwater world, creating a surreal fusion of reulity and fantasy. The bustling activity of vendors selling exolic goods under a sky of swirling auroras transitions seamlessly into the vibrant depths of the ocean, inviting viewers to explore this mysterious realm. This unique blend of surfacelevell excitement and serene tranquility evokes curiosity and wonder, guiding the audience through mesmerizing journey that defies traditional boundaries.
19. An enchanting digital artwork portrays an underwater haven where fantastical creatures interact harmoniously in style reminiscent of James Gurney's 's Dinotopia. From an aerial perspective, this captivating scene reveals the gentle touch between a regal triceratops and a graceful mermaid, as they share a tender moment under the luminous sunlight filtering through the pristine waters. This mesmerizing encounter evokes wonder and tranquility, skillfully blending fantasy with prehistoric elements to create a visually stunning tableau that transcends time and imagination.
20. Craft an enchanting digital artwork that fuses surrealism with vibrant colors, depicting an underwater realm where fish gracefully perform ballet in harmony with the flowing currents. This unique perspective showcases dreamlike dance between elegant sea creatures and floating bubbles, bathed in soft pastel hues evoking sense of wonder and tranquility. Delicate brushstrokes and fantastical shapes intertwine to create a captivating visual symphony, inviting viewers into an otherworldly realm where reality and imagination seamlessly blend.
19

---

Ultra-Resolution Adaptation with Ease
C. More Experimental Results
C.1. More Results on Al Preference Study
We additionally use GPT-4o for a quantitative analysis, allowing for more intuitive evaluation of performance of cach method across different assessment dimensions. Our experimental setting and pairwise comparison remain consistent. We randomly select 300 prompts from the COCO30K dataset and generate images of 2048 x 2248. The results are shown in Table 4. The results indicate that method achieves exceptionally high level across all three dimensions and is comparable SOTA model FLUXI.1 [Pro] Ultra.
Table Results on Al preference study. Evaluation images are generated with COCO30K prompts with a resolution of 2048 x 2048.
Method/Setting
Overall Quality
Rank
Prompt Alignment
Rank
Visual Aesthetics
Rank
SDEdit
87.09
90.99
2
89.18
2
w/ URAE
88.23
92.49
90.09
I-Max
88.24
2
91.38
2
89.96
2
w/ URAE
89.12
92.58
90.86
FLUX1.1 [Pro] Ultra
90.42
93.53
2
90.42
2
ExArt-Sigma-XI.
86.13
88.71
86.31
5
Sana-1.66
86.46
90.25
87.80
3
FLUX.1 1-dev
84.23
89.05
84.88
FLUX. 1 I-dev*
86.05
5
89.48
87.02
w/ URAE
89.71
2
93.64
91.47
C.2. More Results FID and LPIPS against Real Images
In Table report the FID and LPIPS evaluation results of baseline methods against reference images generated by FLUXI.1 [Pro] Ultra model. In this section, we also provide FID and LPIPS results of baseline methods against real images. The results are shown in Table 5.
Table 5: Results on FID and LPIPS evaluated against real images. Evaluation images are generated with 2, 000 and 1, 000 prompts randomly selected from COC02014val with resolution of 2048 x 2048 and 4096 x 4096.
2048 x 2048
FID
LPIPS
4096 x 4096
FID
LPIPS
F [Pro] Ultra
47.12
0.4518
FLUX1.1 [Pro] Ultra
PixArt- PixArt-Sigma-XL.
57.02
0.5075
PiAArt-Sigmat
75.81
0.5066
Sana- 1.6B
54.57
0.5122
Sana-1.66
73.46
0.5108
Ours
52.95
0.4669
Ours
70.44
0.4647
C.3. More Results on Few-Step Diffusion Model
We also conduct experiments on FLUX.I-Schnel] to demonstrate the generalization and the strong adaptation capabilities of our proposed method. Without any additional training, a trained adapter on FLUX.1 1dee can be migrated onto FLUX.1- schnell, which can generate high-quality results with only 4 denoising steps and achieves 6x acceleration compared with FLUX. 1-dev (25.8 v.s. 36.5 sec./image). The results are shown Table
20

---

Ultra-Resolutinn Adaptation with Ease
Table 6: The quantitative results on I-Schnell. All images are generated with the 2, 000 prompts randomly selected from COCO2014wC 4val, and are of 2048 x 2048 in size. Here, FID evaluated against real images. Here, FLUX.1-SchnelII FLUXX. T.UX.1-Scchie 1-Schnell with scaled RoPE, proportional attention, and removing dynamic shifting strategies.
FID ())
HPSv2.1 (t)
PickScore (1)
FLUX. --schnell
42.42
27.97
22.07
FLUX.1 I-schnell"
42.20
28.17
22.38
w/ URAE
38.66
29.63
22.74
C.4. More Results on Ablation Studies
C.4.1. CHOICE OF SINGULAR COMPONENT RANK
We also conduct the ablation studies to analyze the sensitivity to the singular component rank r. The evaluation results are shown in Table 7. From the results, the performance remains stable when is around 16.
Table 7: The quantitative results on the choice of singular component rank (r), All images are generated with the 2, 000 prompts randomly selected from COCO2014val, and are of 2048 x 2048 in size. Here, FID is evaluated with inference images generated by FLUX1.1 [Pro] Ultra.
FID (L)
HPSv2.1 (1)
PickScore (t)
r T= 1
42.34
30.21
23,01
r = 4
38.08
30.99
23.04
r = 16
38.85
31.50
23.21
r 64
38.97
30.14
22.91
256
38.77
29.89
22.82
C.4.2. TRAINING- TIME GPU MEMORY REQUIREMENTS FOR DIFFERENT RANKS
For parameter efficiency, we conduct experiments on training time GPU memory requirement (MB) with respect to various ranks of the adapters. The results are shown in Table 8. We observe that comparing with full-rank adaptation, the low-rank adapters save GPU memory over 50%.
Table Training-time GPU memory requirements for different ranks for 2048 x 2048 and 4096 x 4096 image generation tasks. Here, we adopt r as the default setting the paper.
16 (Default)
r
256
3072 (Full)
2048 2048
35916
35958
36124
36816
39884
77880
4096 x 4096
62806
62850
63010
63704
661 14
OOM
21


Reviews:
[{"review_id": "tny5BsfiIY", "reviewer": "Reviewer_5Gfa", "summary": "This paper explores the adaptation of existing models to ultra-resolution image generation. The authors categorize the challenges into two key aspects: data efficiency and parameter efficiency. Regarding data efficiency, the authors argue that synthetic data can serve as a valuable resource for model convergence in data-scarce scenarios. Regarding parameter efficiency, the proposed approach focuses on tuning minor components when adapting existing models to ultra-resolution. This method offers a promising direction for expanding existing models to ultra-resolution image generation by leveraging a small set of synthetic data for efficient adaptation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 5Gfa's thoughtful comments and are glad that the significance and insights of our work are recognized. We would like to address the concerns as below.\n\n> 1. Theoretical analysis on synthetic data and mode collapse.\n\n* Theorem 2.4 illustrates that, **by diminishing label noise, accurate synthetic data achieve lower error than real data**, which theoretically supports the effectiveness.\n* **For mode collapse, we theoretically verify that the difference on the diversity of generated samples between the trained and optimal models are tightly bounded, highlighting its robustness against this issue.** Specifically, assume the input data $u\\sim\\mathcal{N}(0;I)$. The distance between the variance of generated samples by models after $T$ iterations and the optimal one satisfies:\n$$\n\\mathbb{E}[\\vert Var(f(u;W_T))-Var(f(u;W^*))\\vert]\\leq2\\Vert W^*\\Vert_2\\sqrt{d}+d,\n$$\nwhere the settings follow Theorem 2.4 and $d$ is the r.h.s. of Eq. 3, concerning with the accuracy of synthetic data. We will include the proof in the revision.\n\n> 2. Visual inspection and mode collapse.\n\n* Mode collapse refers to a lack of diversity in **various generative samples**, which is, in fact, not equivalent to similar patterns **within an image**. According to its definition, we do not encounter this issue as validated by the FID against 2K real images in COCO2014val below:\n|2K|FID$\\downarrow$|LPIPS$\\downarrow$|4K|FID$\\downarrow$|LPIPS$\\downarrow$|\n|-|-|-|-|-|-|\n|PixArt-Sigma-XL|57.02|0.5075|PixArt-Sigma-XL|75.81|0.5066|\n|Sana-1.6B|54.57|0.5122|Sana-1.6B|73.46|0.5108|\n|Ours |**52.95**|**0.4669**|Ours|**70.44**|**0.4647**|\n|FLUX1.1 [Pro] Ultra|47.12|0.4518|FLUX1.1[Pro]Ultra|-|-|\n* Possibly caused by the powerful spatial attention, FLUX itself tends to yield similar pattens, which are also reflected in Fig. 1 and Fig. 14 of I-Max (Du et al., 2024b) and can be inherited by models based on it.\n* Empirically, we observe that when objects or patterns are explicitly specified in prompts, the results tend to follow the prompts rather than exhibiting similarity. We validate this through GenEval scores below, which assess precisions of position, instance appearance, etc.\n||PixArt-Sigma-XL|Sana-1.6B|Ours|\n|-|-|-|-|\n|GenEval Score|0.5422|0.6892|**0.6913**|\n* Sincerely hope our responses can alleviate this concern and we will further clarify it with visualizations in our revision.\n\n> 3. It's unclear how closely FLUX-1.1 [Pro] Ultra resembles real data.\n\n* As FLUX1.1 [Pro] Ultra ranks top on multiple text-to-image leaderboards and our goal is to achieve on-par performance with it, we adopt its generated images as targets in Tab. 1.\n* We also supplement results computed against real images in COCO2014val. Please refer to *our response to Q2* for details.\n\n> 4. Reduce the resolution for quantitative evaluation.\n\n* Thanks for the suggestion. We evaluate our URAE on SD1.5 and adapt it from 512 to 1024 scale. The training data are generated by SD3. We kindly refer the reviewer to *our response to Q2 of Reviewer u5sm* for the results, which demonstrate that URAE achieve superior high-resolution generation capacity.\n\n> 5. MAN-IQA and QualiCLIP may not be reliable metrics for evaluating 4K resolution.\n\n* In fact, various metrics have varying preferences and biases, so we include diverse metrics to demonstrate the superiority of our method across various aspects. By downsampling the generated 4K images to the required resolution for evaluation, we supplement more metrics here to reinforce the conclusion:\n|4K|HPSv2.1|ImageReward|PickScore|GPT-4o Aesthetic|GPT-4o Prompt Alignment|GPT-4o Overall|\n|-|-|-|-|-|-|-|\n|PixArt-Sigma-XL|31.02|0.9342|22.76|87.66|87.00|86.28|\n|Sana-1.6B|32.00|1.0886|22.86|87.71|89.94|86.83|\n|Ours|**32.85**|**1.1484**|**23.38**|**89.65**|**90.50**|**87.58**|\n* GPT-4o scores for human-like evaluation are also included.\n|Win Rate|Aesthetic|Prompt Alignment|Overall|\n|-|-|-|-|\n|v.s. pixart|67.90%|61.60%|59.50%|\n|v.s. sana|66.60%|50.40%|57.30%|\n\n> 6. How LPIPS is measured?\n\n* In Tab. 1, similar to FID, images generated by FLUX1.1 [Pro] Ultra are used as the target for LPIPS, while the sources are images generated by various methods, following [a]. In *our response to Q2*, we also supplement the LPIPS results computed against real images.\n\n> 7. Relationship with tuning-free high-dimensional image generation methods.\n\n* The \"ultra-resolution adaptation\" in the manuscript refers to the training-based adaptation. We will explicitly clarify this in our revision. \n* The relationships are discussed in Sec. A.2 of the appendix, where we mention that the two lines of research tackle the problem from two orthogonal perspectives: model and pipeline.\n* As shown in Line 308 (left), we apply the trained adapters by our method to these training-free solution in the high-resolution stage. Results can be found in Tab. 1 and Fig. 5.\n***\n[a] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models, Li et al., CVPR 2024", "claims_and_evidence": "It is somewhat unclear whether synthetic data generated by teacher models can theoretically promote training convergence significantly, as the authors provide only empirical evidence without a formal theoretical justification. In Section 2.2, synthetic data would be beneficial only if the reference model generating these data is highly accurate. However, it is not guaranteed that this approach avoids mode collapse.\n\nFrom visual inspection, the generated images appear to exhibit **highly similar patterns**, suggesting possible mode collapse. For instance: In Figure 8 (URAE Minor-4K), the second image contains many repetitive flower patterns, whereas PixArt-Sigma-XL generates more diverse floral structures.\nSimilarly, in the giraffe example, the URAE-generated image displays repetitive mountain patterns, whereas PixArt-Sigma-XL and Sana-1.6B show greater variation. Such **repetitive patterns** are also widely noticeable in Figure 1, further supporting this concern.\n\nAdditionally, it is unclear how closely FLUX-1.1 [Pro] Ultra resembles real data. The authors appear to assume FLUX-1.1 [Pro] Ultra as real and measure FID scores relative to it in Table 1, yet it is still synthetically generated data.\n\nWhile the paper argues that existing 2K or 4K resolution benchmarks do not exist, an alternative approach could be to **reduce the resolution for quantitative evaluation**. For example, adapting a 512-resolution model to generate 1024-resolution images could provide meaningful comparative insights.", "methods_and_evaluation_criteria": "See the **Claims And Evidence** part.", "theoretical_claims": "It is commendable that the paper includes a theoretical proof in Section 2; however, the derivation does not clearly support the results claimed in the paper. Please refer to the \"Claims and Evidence\" section for further clarification.", "experimental_designs_or_analyses": "It appears that MAN-IQA and QualiCLIP may not be reliable metrics for evaluating 4K resolution, as their rankings differ significantly from user study results. For instance, while FLUX-1.dev* ranks second in MAN-IQA, it exhibits noticeable artifacts in Figure 8, raising concerns about the alignment between automated metrics and perceptual quality.\n\nAdditionally, could the authors clarify how LPIPS is measured? Specifically, which images are used as the source and which as the target in the comparison?", "relation_to_broader_scientific_literature": "The key contribution of this paper is adapting existing models to a data-scarce domain by leveraging a smaller set of synthetic data. This approach could be highly beneficial for various domains where obtaining high-dimensional data is significantly more challenging than acquiring low-dimensional data.", "essential_references_not_discussed": "The paper provides a well-discussed review of existing works, effectively situating its contributions within the broader research landscape.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper addresses the important problem of adapting existing models for ultra-resolution image synthesis.\n- The writing is well-structured and easy to follow, making the paper accessible to readers.\n- Exploring this problem from multiple perspectives (e.g., data, parameters) provides valuable insights and contributes to a broader understanding of the challenges involved.\n\n**Weaknesses**: Please address the concerns raised in the \"Claims and Evidence\" and \"Experimental Designs or Analyses\" sections.", "other_comments_or_suggestions": "Given the existence of numerous tuning-free approaches for expanding models to high-resolution image synthesis, including ultra-resolution adaptation, it is difficult to claim that this paper is the first to tackle adaptation as a primary contribution.", "questions_for_authors": "How can this approach be integrated with existing tuning-free high-dimensional image generation methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ygJVP4U5M2", "reviewer": "Reviewer_u5sm", "summary": "This paper tackles the challenge of efficiently adapting text-to-image diffusion models to ultra-high resolutions (2K and 4K). Traditional approaches demand massive amounts of 4K training data and expensive fine-tuning of the entire model, making them difficult to deploy at scale. In contrast, URAE explores two main dimensions—data efficiency and parameter efficiency—and provides guidelines that yield strong ultra-resolution results with only thousands of samples and minimal GPU resources. By combining synthetic teacher-generated data (when available) and targeted parameter-efficient fine-tuning, URAE achieves state-of-the-art 2K image quality comparable to closed-source models such as FLUX1.1. It also sets new benchmarks in 4K resolution, demonstrating its adaptability under data-scarce conditions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer u5sm for the constructive comments. We are happy that the reviewer finds our data efficiency practical, ablation detailed, and empirical evidence strong. We would like to address the concerns and questions reflected in the review below.\n\n> 1. Limited Real-World Cost Analysis: While fewer iterations are praised (2K–10K), a clearer breakdown of training time, memory usage, or energy consumption would better illustrate URAE’s resource savings.\n\n* We would like to sincerely thank the reviewer for the constructive suggestions. Following the suggestions, according to publicly available information, we summarize \"# of Training Iteration × Batch Size\" of various methods, which reflects the total number of seen samples during training, to present a clearer breakdown of the required resources:\n\n    ||PixArt-Sigma-XL|Sana-1.6B|Ours|\n    |-|-|-|-|\n    |# of Training Iteration × Batch Size|64K|≥320K|16K|\n  \n  Since different methods use varying base models and hardwares for training, we exclude training time as a direct indicator of resource savings. Nevertheless, even for FLUX—the largest open-source diffusion model with 12B parameters—our 4K model can still be trained within a day on an 8×H100 server.\n  \n* For memory usage, we conduct the following studies on training-time GPU memory requirement (MB) with respect to various ranks of the adapters:\n\n    |Rank|1|4|16 (Default)|64|256|1536|3072 (Full)|\n    |-|-|-|-|-|-|-|-|\n    |2K|35916|35958|36124|36816|39884|52102|77880|\n    |4K|62806|62850|63010|63704|66114|80332|OOM|\n  \n  We observe that comparing with full-rank adaptation, the low-rank adapters save GPU memory by 50%+.\n\n> 2. Focus on DiT-Style Models: The method’s adaptability to other architectures (e.g., UNet-based) is suggested but not deeply tested.\n  \n* Thanks for the constructive suggestion. Following the suggestion, we conduct an experiments on SD-1.5, to adapt it from 512 to 1024 resolution. The synthetic data used are 10K samples generated by SD3. Results are shown below:\n\n    ||FID$\\downarrow$|HPSv2.1$\\uparrow$|PickScore$\\uparrow$|\n    |--|--|--|--|\n    |SD 1.5|47.55|23.66|20.69|\n    |SD 1.5*|45.15|23.72|20.71|\n    |SD 1.5 w/ [a]|43.07|24.36|21.32|\n    |SD 1.5 w/ Ours|**31.06**|**28.93**|**21.98**|\n    \n  The FID is computed against 5K images in COCO2014val following [b]. SD1.5* denotes using the porpotional attention strategy similar to FLUX-1.dev* in Tab. 1. [a] is a state-of-the-art training-free high-resolution generation baseline based on resolution-aware downsampling and upsampling. The results verify the adaptability of our method to UNet-based diffusion models and its superior high-resolution generation capacity.\n\n> 3. Inference Efficiency: The paper admits it does not optimize for inference latency, which might matter for large-scale industrial use.\n\n* Thanks for pointing this out. Although this work does not specifically optimize inference latency, we would like to share our latest observation that, without any additional training, a trained adapter on FLUX.1-dev can be migrated onto FLUX.1-schnell, which can generate high-quality results with only 4 denoising steps and achieves $6\\times$ acceleration compared with FLUX.1-dev (25.8 v.s. 36.5 sec./image). The performance under this setting is shown below:\n\n    ||FID$\\downarrow$|HPSv2.1$\\uparrow$|ImageReward$\\uparrow$|PickScore$\\uparrow$|\n    |--|--|--|--|--|\n    |FLUX-schnell|42.42|27.97|0.6902|22.07|\n    |FLUX-schnell*|42.20|28.17|0.7446|22.38|\n    |w/ URAE|**38.66**|**29.63**|**0.9999**|**22.74**|\n\n    We will include these results in our revision, which suggest significant potential for acceleration.\n\n> 4. In Figure 7, it’s not immediately clear how using synthetic data differs from using real data. Could you explain why the visual differences in Figure 7 appear subtle, and what evidence in the paper supports the conclusion that synthetic data ultimately improves training and performance?\n\n* Thanks for the good question. In fact, Fig. 7 in the manuscript empirically verfies the theoretical result in Theorem 2.4 that synthetic data improve performance by diminishing label noises. Comparing results from synthetic and real data, we observe that the latter introduces many unrelated petals, whereas the former exhibits a cleaner layout. Additionally, the synthetic data produce a brighter, more vivid color tone and sharper contours with higher saturation. \n* Furthermore, the results in Tab. 2 quantitatively demonstrate the superiority of synthetic data.\n\nWe would like to thank Reviewer u5sm again for the valuable feedback. Hope our responses alleviate the reviewer's concerns and we are happy to answer additional questions if there are.\n\n***\n\n[a] HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models, Zhang et al., ECCV 2024\n\n[b] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models, Li et al., CVPR 2024", "claims_and_evidence": "1. **Claim: URAE Achieves Ultra-Resolution Adaptation with Minimal Data**  \n   - **Evidence**: The authors fine-tune a base diffusion model (FLUX.1-dev) on just **3K synthetic samples for 2K** tasks and achieve close or better results than advanced closed-source models. The theoretical analysis (Theorem 2.4) shows how synthetic data from a high-quality teacher can expedite training convergence.\n\n2. **Claim: Parameter-Efficient Fine-Tuning Is More Effective Than Full Model Tuning**  \n   - **Evidence**: Through ablation, they show that focusing on particular “minor” or “major” singular values outperforms commonly used LoRA in certain scenarios, especially for 4K adaptation when synthetic data is unavailable. Empirical benchmarks in Tables 1–3 confirm superior performance over baseline or naive approaches.\n\n3. **Claim: Disabling Classifier-Free Guidance (CFG) During Training Improves Stability**  \n   - **Evidence**: The authors discover that for guidance-distilled models like FLUX, setting the CFG scale to 1 (effectively “off”) during fine-tuning leads to better adaptation performance. Results in Table 2 and Figures 3 & 7 illustrate the negative impact of leaving CFG on during adaptation.\n\n4. **Claim: Compatibility with Training-Free High-Resolution Pipelines**  \n   - **Evidence**: URAE can be employed in conjunction with existing post-processing or upscale pipelines (e.g., SDEdit, I-Max). Figure 5 shows that URAE effectively upgrades their output from 1024×1024 to 2048×2048, surpassing conventional super-resolution baselines like Real-ESRGAN and SinSR.", "methods_and_evaluation_criteria": "- **Methods**:  \n     - URAE advocates fine-tuning on high-quality synthetic data generated by a teacher model.\n     - At 2K resolution (with synthetic data), focusing on major components (LoRA) works well. At 4K resolution (less reliable data), tuning minor singular values preserves the model’s essential capacities and avoids overfitting to noise.  \n     - For models that rely on guidance distillation, turning off CFG (g=1) eliminates mismatched training objectives.  \n- **Evaluation**:  \n     - Datasets & Benchmarks: HPD, DPG, LAION-5B for real data, and teacher-synthesized data from FLUX1.1 [Pro] Ultra for synthetic data.  \n     - Metrics: FID, LPIPS, MAN-IQA, QualiCLIP, user preference metrics (HPSv2.1, PickScore), and GPT-4-based AI preference scores.  \n     - Baselines: Includes stable or reference models like PixArt-Sigma, Sana-1.6B, Real-ESRGAN, SinSR, FLUX-1.dev, etc.  \n\nOverall, comprehensive quantitative and qualitative comparisons highlight URAE’s effectiveness.", "theoretical_claims": "The authors introduce a linearized neural tangent kernel perspective (Theorem 2.4) to show that mixing real and synthetic data can accelerate learning, provided the synthetic data come from a sufficiently good teacher. This analysis solidifies the data-efficiency claim, as it mathematically bounds the distance to the optimal solution under varying real/synthetic data proportions.", "experimental_designs_or_analyses": "- **Extensive Benchmarks**:  \n  1. **2K Results**: Detailed in Table 1 and Fig. 4–6, showing URAE outperforms baseline or SOTA methods in image fidelity and preference tests.  \n  2. **4K Results**: Evaluated in Table 3 and Fig. 8, highlighting that “minor” component tuning without synthetic data can still yield strong high-resolution outputs.  \n  3. **Ablation Studies**: Table 2 and Fig. 7 analyze the effect of (i) synthetic vs. real data, (ii) tuning major vs. minor components, and (iii) CFG on or off.  \n\n- **User Studies & AI-Assisted Scoring**:  \n  Incorporating GPT-4 preference evaluation (Fig. 4, Table 4) yields additional insights into alignment, aesthetics, and overall image quality.", "relation_to_broader_scientific_literature": "- URAE aligns with growing research on large diffusion transformers, e.g., FLUX, PixArt, SANA, focusing primarily on data-efficiency, rather than training entire massive backbones.  \n- The approach extends beyond standard LoRA, referencing PISSA, FedPara, and other recent minor-component methods.  \n- Although mainly tested on image generation, the method could integrate well with broader multimodal large language models.", "essential_references_not_discussed": "Yang, Zhuoyi, et al. \"Inf-dit: Upsampling any-resolution image with memory-efficient diffusion transformer.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "other_strengths_and_weaknesses": "**Strengths**:\n- **Practical Data Efficiency**: Demonstrates that 3K–30K images are enough to scale from 2K to 4K resolution, far below prior 4K training demands.  \n- **Detailed Ablation**: Comprehensive analysis of synthetic vs. real data usage, plus major/minor SVD component choices.  \n- **Strong Empirical Evidence**: Includes GPT-4 preference ranking, user studies, and well-known objective metrics.\n\n**Weaknesses**:\n- **Limited Real-World Cost Analysis**: While fewer iterations are praised (2K–10K), a clearer breakdown of training time, memory usage, or energy consumption would better illustrate URAE’s resource savings.  \n- **Focus on DiT-Style Models**: The method’s adaptability to other architectures (e.g., UNet-based) is suggested but not deeply tested.  \n- **Inference Efficiency**: The paper admits it does not optimize for inference latency, which might matter for large-scale industrial use.", "other_comments_or_suggestions": "In Figure 7, it’s not immediately clear how using synthetic data differs from using real data. Could you explain why the visual differences in Figure 7 appear subtle, and what evidence in the paper supports the conclusion that synthetic data ultimately improves training and performance?", "questions_for_authors": "1. **Computational Footprint**: Could you provide more details on training cost comparisons, e.g., GPU hours, memory usage, or speedups over full fine-tuning?  \n2. **UNet-Based Models**: Does URAE also apply neatly to stable diffusion–type backbones, or are modifications needed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehIocNX7LI", "reviewer": "Reviewer_wYHL", "summary": "The paper \"Ultra-Resolution Adaptation with Ease\" presents a novel approach called URAE for adapting text-to-image diffusion models to generate ultra-high-resolution images (e.g., 4K) with limited training data and computational resources. The key contributions include:\n1.\tTheoretical and empirical evidence showing that synthetic data from teacher models can significantly enhance training convergence.\n2.\tA parameter-efficient fine-tuning strategy that tunes minor components of weight matrices, outperforming widely-used low-rank adapters when synthetic data is unavailable.\n3.\tThe importance of disabling classifier-free guidance during adaptation for models leveraging guidance distillation.\n4.\tExtensive experiments demonstrating that URAE achieves performance comparable to state-of-the-art closed-source models like FLUX1.1 [Pro] Ultra with only 3K samples and 2K iterations for 2K generation, while setting new benchmarks for 4K-resolution generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer wYHL for the positive feedback on the manuscript and are very excited that the reviewer mentions the strengths of addressing a significantly practical problem with a comprehensive framework, insightful theoretical analysis, extensive experiments, and well-written manuscript. The questions are addressed below.\n\n> 1. The paper could benefit from more detailed comparisons with very recent works on efficient high-resolution generation. For example, those methods mentioned in the “Essential References Not Discussed” section.\n\n* Thanks for bringing these related works to our attention. We show comparisons and integrations with some training-free works in Tab. 1 and Fig. 5. For works mentioned by the reviewer, we would like to supplement the comparison results using consistent COCO validation prompts here:\n||FID$\\downarrow$|HPSv2.1$\\uparrow$|ImageReward$\\uparrow$|PickScore$\\uparrow$|\n|--|--|--|--|--|\n|AP-LDM[2]|48.50|30.40|0.6874|22.80|\n|FreeScale[3]|48.87|31.19|0.7494|22.66|\n|DiffuseHigh[4]|49.02|30.16|0.6182|22.77|\n|URAE(Ours)|**38.85**|**31.50**|**1.0923**|**23.21**|\n\n* We would like to include the following discussions on these reference to the revision:\n    1. [1] proposes a resolution-adaptive attention scale factor, which has **already been adopted** in a series of works including FLUX-1.dev$^*$ and I-Max in Tab. 1.\n    2. [2] proposes an attention-guidance scheme and a progressive upsampling strategy.\n    3. [3] adopts a global-local self-attention mechanism and a tailored self-cascade upscaling strategy with region-aware detail control.\n    4. [4] proposes a DWT-based structural guidance to guide the high-resolution generation with the structural information of the low-resolution images.\n    \n  These works mentioned by the reviewer tackle the problem of high-resolution image generation from **training-free** perspectives by designing effective stragies, *e.g.*, **progressive generation**, to leverage pre-trained diffusion models at their native scales, wheras our method focuses on adapting these models from a **training-based** perspective so that they can **directly** operate at a high-resolution scale. Therefore, as mentioned in Sec. A.2 of the appendix, the two lines of research work address the problem from orthogonal directions, *i.e.*, strategy v.s. model, and can be readily integrated together for better performance, as shown in Tab. 1 and Fig. 5.\n  \n> 2. The computational efficiency during inference is not specifically optimized, which could be a limitation for real-time applications.\n\n* Thanks for pointing this out. Although this work does not specifically optimize inference latency, we would like to share our latest observation that, even without any additional training, a trained adapter on FLUX.1-dev can be migrated onto FLUX.1-schnell, which can generate high-quality results with only 4 denoising steps and achieves $6\\times$ acceleration compared with FLUX.1-dev (25.8 v.s. 36.5 sec./image). The performance under this setting is shown below:\n\n    ||FID$\\downarrow$|HPSv2.1$\\uparrow$|ImageReward$\\uparrow$|PickScore$\\uparrow$|\n    |--|--|--|--|--|\n    |FLUX-schnell|42.42|27.97|0.6902|22.07|\n    |FLUX-schnell*|42.20|28.17|0.7446|22.38|\n    |w/ URAE|**38.66**|**29.63**|**0.9999**|**22.74**|\n\n\n    We will include these results in our revision, which suggest significant potential for acceleration.\n\n> 3. How would the performance of URAE scale with additional training data beyond the 3K samples used in the experiments?\n\n* Thanks for the insightful question. We are actively collecting more data from FLUX1.1 [Pro] Ultra and training new models. As scaling up data collection, preprocessing, and training requires significant resources, the experiments are still ongoing, and we will include the results in our revision.\n\n> 4. What specific architectural modifications would be needed to combine URAE with recent efficient diffusion backbone designs (linear attention, SSM)?\n\n* Thanks for the valuable question. We are continuing working on improving the architectural efficiency of the proposed URAE. Our latest exploration suggests the feasibility of replacing the original full attention with the **linearized attention** structure introduced in (Liu et al., 2024a). We find that even without further adaptation, the trained adapters in URAE are compatible with these novel attention layers. We present some examples via [this anonymous link](https://anonymous2024.s3.ap-southeast-1.amazonaws.com/data/linear.pdf). The models with linearized attention achieve $1.4\\times$ acceleration at 2K resolution (25.8 v.s. 36.5 sec./image) and $2.7\\times$ acceleration at 4K resolution (124.2 v.s. 330.4 sec./image)\n\nWe would like to thank Reviewer wYHL again for the in-depth reviews. We would definitely love to further interact with the reviewer if there are any further questions.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence. The authors provide theoretical analysis (Theorem 2.4) to demonstrate the potential benefits of using synthetic data for training convergence. They also conduct extensive experiments to validate the effectiveness of their proposed methods, including ablation studies on key components (data source, parameter tuning strategy, classifier-free guidance). The results show significant improvements over baseline methods and state-of-the-art models in both quantitative metrics and qualitative visual comparisons. The claims about the effectiveness of tuning minor components when synthetic data is unavailable are well-supported by experimental results in the 4K generation task.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem of ultra-resolution adaptation. The approach of using synthetic data from teacher models addresses the challenge of limited high-quality training data for ultra-resolution images. The parameter-efficient fine-tuning strategy that focuses on minor components of weight matrices is innovative and appropriate for scenarios where synthetic data is unavailable. \n\nThe evaluation criteria, including FID, LPIPS, MAN-IQA, QualiCLIP, HPSv2.1, and PickScore, are standard and relevant for assessing image generation quality. The use of GPT-4o for AI preference studies adds a novel dimension to the evaluation, providing insights into human-like preferences for generated images.", "theoretical_claims": "The theoretical claims are correct. The authors provide a detailed proof (Theorem B.1) for their main theoretical result regarding the error bound when training with a mixture of real and synthetic data. The proof follows standard optimization analysis for neural networks and correctly accounts for the impact of label noise and model discrepancies. The assumptions made (infinite-width neural networks, linear approximation) are standard in theoretical analyses of neural network training.", "experimental_designs_or_analyses": "The experimental designs are sound and valid. The authors conduct experiments on both 2K and 4K resolution tasks, comparing against multiple baseline methods and state-of-the-art models. The ablation studies effectively isolate the impact of different components of their approach. The user study for 4K generation provides additional validation of the practical effectiveness of their method. The experimental setup, including training details and implementation specifics, is well-documented and allows for reproducibility.", "relation_to_broader_scientific_literature": "The key contributions of this paper are well-situated within the broader scientific literature on text-to-image generation and diffusion models. The work builds upon recent advances in diffusion models, parameter-efficient fine-tuning, and high-resolution image generation. It addresses the practical challenge of adapting existing models to ultra-resolution settings with limited resources, which is a significant concern in the field.", "essential_references_not_discussed": "The paper cites relevant prior work in text-to-image diffusion models, high-resolution generation, and parameter-efficient fine-tuning. However, it could benefit from discussing more recent works on high-resolution generation[1,2,3,4], especially training-free ones.\n\n[1] Jin, Zhiyu, et al. \"Training-free diffusion model adaptation for variable-sized text-to-image synthesis.\" Advances in Neural Information Processing Systems 36 (2023): 70847-70860.\n \n[2]Cao, Boyuan, et al. \"Ap-ldm: Attentive and progressive latent diffusion model for training-free high-resolution image generation.\" arXiv preprint arXiv:2410.06055 (2024).\n\n[3] Qiu, Haonan, et al. \"Freescale: Unleashing the resolution of diffusion models via tuning-free scale fusion.\" arXiv preprint arXiv:2412.09626 (2024).\n\n[4] Kim, Younghyun, et al. \"Diffusehigh: Training-free progressive high-resolution image synthesis through structure guidance.\" arXiv preprint arXiv:2406.18459 (2024).", "other_strengths_and_weaknesses": "Strengths:\n•\tThe paper addresses a significant practical problem in the field of text-to-image generation.\n•\tThe proposed URAE framework is comprehensive, addressing both data and parameter efficiency.\n•\tThe theoretical analysis provides valuable insights into the effectiveness of synthetic data.\n•\tThe experimental validation is extensive and rigorous.\nWeaknesses:\n•\tThe paper could benefit from more detailed comparisons with very recent works on efficient high-resolution generation. For example, those methods mentioned in the “Essential References Not Discussed” section\n•\tThe computational efficiency during inference is not specifically optimized, which could be a limitation for real-time applications.", "other_comments_or_suggestions": "The paper is well-written and well-structured, making it accessible to both experts and those new to the field. The visualizations of results are clear and effectively demonstrate the quality improvements achieved by URAE.", "questions_for_authors": "1.\tHow would the performance of URAE scale with additional training data beyond the 3K samples used in the experiments?\n2.\tWhat specific architectural modifications would be needed to combine URAE with recent efficient diffusion backbone designs (linear attention, SSM)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dOOTuQQoOm", "reviewer": "Reviewer_FjKy", "summary": "This paper introduces **URAE**, a framework for efficiently adapting text-to-image diffusion models to ultra-high resolutions (e.g., 4K) while minimizing computational costs and data requirements. The approach is based on three key ideas:\n\n- **Data Efficiency**: Using synthetic images generated by a teacher model improves convergence.\n- **Parameter Efficiency**: Fine-tuning the minor singular components of weight matrices is more effective than traditional low-rank adaptation methods like LoRA.\n- **Classifier-Free Guidance (CFG) Control**: Disabling CFG during adaptation (\\( g = 1 \\)) improves training consistency.\n\nExperiments demonstrate state-of-the-art performance at 2K and 4K resolutions while requiring significantly fewer training samples and iterations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank Reviewer FjKy for the valuable comments and are glad that the reviewer finds our method practical, efficient, and empirically strong. We would like to address the concerns as below.\n\n> 1. Theoretical motivation for minor singular component tuning.\n\n* We would like to supplement the following theoretical analysis torwards tuning minor singular component, which will be included in our revision.\n* Consider the low-rank adapter:\n  $$\n  Y=XW_0+XAB.\n  $$\n  The loss is denoted as $L$. Then, the gradients of $L$ w.r.t. $A$ and $B$ are:\n  $$\n  \\frac{\\partial L}{\\partial A}=X^\\top\\frac{\\partial L}{\\partial Y}B^\\top\n  $$\n  and\n  $$\n  \\frac{\\partial L}{\\partial B}=A^\\top X^\\top\\frac{\\partial L}{\\partial Y},\n  $$\n  respectively.\n* In vanilla LoRA, $W_0$ is the original weight matrix $W$, while $A$ and $B$ and initialized as random values and zeros respectively. Thus, in the initial adaptation stage, due to the joint influence of $A$'s **random initialization and noise in data**, the gradients of $A$ and $B$ can be highly random, potentially leading to instability.\n* In our approach of tuning minor components, as shown in Eqs. 5 and 6 of the main manuscript, if $W=U\\Sigma V$ derived by SVD, $W_0=U[:,:-r]\\Sigma[:-r,:-r]V[:-r,:]$, $A=U[:,-r:]\\sqrt{\\Sigma[-r:,-r:]}$, and $B=\\sqrt{\\Sigma[-r:,-r:]V[-r:,:]}$ initially. Consequently, the gradients of $A$ and $B$ are influenced by the minor components of the original weight matrix $W$, which tend to be **numerically small and more stable** compared to standard LoRA. For ultra-resolution adaptation, where major semantics and appearances remain unchanged, tuning minor components helps  preserve knowledge in $W$ by **effectively regulating the gradients** of $A$ and $B$.\n\n> 2. No formal analysis of convergence or stability guarantees of the fine-tuning method.\n\n* We conduct the following theoretical analysis on  the upper bound of the distance between the solution after $T$ iterations $A_TB_T$ and the optimal $W^*$:\n  $$\n  \\Vert A_TB_T-W^*\\Vert^2_F\\leq(1-\\eta\\mu)^T\\Vert A_0B_0-U[:,:r]\\Sigma[:r,:r] V[:,:r]\\Vert^2_F+\\sum_{i=r+1}^c\\Sigma[i,i]^2,\n  $$\n  where $W^*=U\\Sigma V$ with SVD, and $\\mu$ is the smallest non-zero eigenvalue of the Hessian matrix. \n* The above bound indicates that the training converges to its theoretical optimal solution in a linear rate. We will include the proof in the revision.\n\n> 3. Applicability to other models.\n\n* Thanks. We conduct experiments on Infinity, a text-to-image visual autoregressive model, to adapt it from 1K to 2K scale. The following results confirm the applicability to various models.\n||QualiCLIP$\\uparrow$|MAN-IQA$\\uparrow$|HPSv2.1$\\uparrow$|\n|-|-|-|-|\n|Infinity-8B|0.5233|0.3226|32.26|\n|w/ URAE|**0.5570**|**0.3584**|**32.35**|\n\n> 4. Quantified computational efficiency.\n\n* In fact, the parameter efficiency here refers to training efficiency as it only requires to tune a small amount of parameters. As indicated in Sec. 4, this work does not focus on inference efficiency since it is orthogonal to our main contributions. The inference cost is the same as the original FLUX operating at the corresponding resolutions. On H100:\n||2K|4K|\n|-|-|-|\n|Inference Time (28 Steps/Image)|36.5 Sec.|330.4 Sec.|\n|GPU Memory|27.5 GB|39.5 GB|\n* For the quantified analysis of data efficiency and parameter efficiency, we kindly refer the reviewer to *our response to Q1 of Reviewer u5sm*.\n\n> 5. Tuning minor components in other applications.\n\n* We note that (Wang et al. 2024a) focus on LLM fine-tuning and also applies minor-component tuning. However, there lacks critical analysis on the applicability of this approach across various scenarios. In contrast, we demonstrate that the method can improve performance when (1) data contains significant noise and (2) the target distribution does not shift too much from the source, *e.g.*, 4K generation. In other cases, when clean data are available, we find that vanilla LoRA can be more effective, as shown in Tab. 2.\n* Our response to Q1 provides theoretical insights on this. Due to small singular values, the gradients w.r.t. $A$ and $B$, are numerically small, which may lead to insufficient adaptation when training data are accurate.\n\n> 6. Discussion on alternative ultra-resolution adaptation techniques.\n\n* We show comparisons and integrations with some works in Tab. 1 and Fig. 5 and include some discussions in Sec. A.2.\n* We kindly refer the reviewer to *our response to Q1 of Reviewer wYHL* for more discussions.\n\n> 7. Extend experiments to larger datasets (e.g., ImageNet, LAION-HR).\n\n* In fact, as shown in Line 263 (right), our 4K-generation model is already trained with LAION-HR data.\n\n> 8. Sensitivity to the choice of singular component rank ($r$).\n\n* We conduct the following studies to analyze the sensitivity to the rank $r$:\n|Rank|1|4|16 (Default)|64|256|\n|-|-|-|-|-|-|\n|ImageReward$\\uparrow$|0.9291|1.0150|1.0923|0.9442|0.9239|\n\n  Overall, the performance remains stable when $r$ is around 16.", "claims_and_evidence": "- **Claim**: Ultra-resolution adaptation can be efficiently achieved without large-scale data or full model fine-tuning.\n  - **Evidence**: Empirical results demonstrate that the method achieves competitive performance at 4K resolution with limited data and training iterations.\n\n- **Claim**: Synthetic data can significantly accelerate convergence.\n  - **Evidence**: Theoretical support via the bound:\n    \\[\n    E[\\|W_T - W^*\\|_2^2] \\leq E[\\|(I - ηM)^T\\Delta_0\\|_2^2] + \\eta^2 (p(1-p)E[\\delta^2]+(1-p)\\sigma^2)\\sum_{i=1}^{N}\\frac{(1-(1-\\eta\\lambda_i)^T)^2}{\\lambda_i}+p^2\\|W_{ref}-W^*\\|_2^2\n    \\]\n  - Experimental validation shows faster convergence with synthetic data augmentation.\n\n- **Claim**: Tuning minor singular components of weights outperforms traditional LoRA-based fine-tuning.\n  - **Evidence**: Ablation studies in Table 3 confirm that fine-tuning lower-rank singular values improves performance.\n\n- **Claim**: Disabling classifier-free guidance during training is necessary.\n  - **Evidence**: Table 2 shows significant degradation in performance when CFG is enabled during adaptation.", "methods_and_evaluation_criteria": "- **Data Generation**: Uses synthetic images from pre-trained models (e.g., FLUX-1.1) at lower resolutions for training guidance.\n- **Parameter-Efficient Fine-tuning**: The authors fine-tune the minor singular components of weight matrices instead of major components:\n  \\[\n  W = U\\Sigma V, \\quad W_{small} = U[:, -r:]\\Sigma[-r:, -r:]V[-r:, :].\n  \\]\n- **Classifier-Free Guidance Control**: CFG is disabled during training and only applied at inference:\n  \\[\n  \\epsilon_{\\theta}(z_t, t, \\emptyset) + g \\cdot (\\epsilon_{\\theta}(z_t, t, y)-\\epsilon_{\\theta}(z_t,t,\\emptyset))\n  \\]\n\n**Evaluation Metrics:**\n- **Quantitative**: FID, LPIPS, MAN-IQA, QualiCLIP, HPSv2.1, PickScore.\n- **Qualitative**: GPT-4o preference scores.", "theoretical_claims": "- **Strengths**:\n  - The use of synthetic data to improve convergence is supported by a well-defined mathematical bound (Theorem 2.4).\n  - The choice to tune minor singular components instead of major ones is motivated by SVD properties.\n\n- **Weaknesses**:\n  - The theoretical motivation for minor singular component tuning is not rigorously justified beyond empirical observations.\n  - No formal analysis of convergence or stability guarantees of the fine-tuning method.", "experimental_designs_or_analyses": "- **Strengths**:\n  - Comprehensive ablation studies validate key design choices.\n  - Performance comparisons with state-of-the-art methods support claims of efficiency.\n\n- **Weaknesses**:\n  - Experiments are limited to **diffusion models**, leaving uncertainty about applicability to other generative models (GANs, autoregressive models).\n  - The computational efficiency gains are not well-quantified in terms of inference latency and resource consumption.", "relation_to_broader_scientific_literature": "- The paper builds upon well-established literature in diffusion models and parameter-efficient adaptation, particularly works on LoRA and DreamBooth.\n  - Explicitly contrasts the proposed approach with previous fine-tuning strategies.\n  - Missing discussion on how this method compares with **latent-space adaptation** techniques used in recent diffusion models.", "essential_references_not_discussed": "- The paper thoroughly covers diffusion model fine-tuning literature but could benefit from:\n  - **Comparison with other LoRA modifications** (e.g., tuning minor components in other applications).\n  - **Discussion on alternative ultra-resolution adaptation techniques**, such as patch-based super-resolution models.", "other_strengths_and_weaknesses": "### Strengths:\n- The approach is **practical and efficient**, providing clear guidelines for ultra-resolution adaptation.\n- The **synthetic data augmentation strategy** is well-supported theoretically and experimentally.\n- Extensive **benchmarking** against diffusion models provides strong empirical validation.\n\n### Weaknesses:\n- This paper lack of a formal explanation, for why minor singular components are more effective than major ones. The results are compelling but require a more theoretical justification.\n- Computational overhead: While the method is designed to be efficient, there is **no detailed profiling of inference efficiency** (e.g., time per image, GPU memory consumption).", "other_comments_or_suggestions": "- Provide a **deeper** theoretical analysis on why minor singular component tuning is optimal for ultra-resolution tasks.\n- Try to extend experiments to larger datasets (e.g., ImageNet, LAION-HR) to validate scalability.\n- Analyze the **impact on inference time** to better quantify computational benefits.", "questions_for_authors": "This is a good work but I still have several general questions:\n\n1. **Theoretical Justification:**  \n   Can you formally analyze why tuning minor singular components leads to better adaptation performance in ultra-resolution tasks?\n   \n2. **Hyperparameter Sensitivity:**  \n   How sensitive is the approach to the choice of singular component rank (\\( r \\))? Would an adaptive selection method improve results?\n\n3. **Applicability to Other Architectures:**  \n   Have you considered applying this method to other generative models beyond diffusion models, such as GANs or autoregressive transformers?\n\n4. **Computational Costs:**  \n   Given that URAE is designed to be parameter-efficient, have you conducted inference speed and memory consumption benchmarks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques for adapting pretrained text2image diffusion models to high resolutions. It studies this problem from two aspects, data and parameter efficiency respectively. The reviewers have reached unanimous positive rating of the work after extensive discussions. The AC believes that this is a strong submission and would recommend accept.
Decision: Accept (poster)