UltraPixel: Advancing Ultra High-Resolution Image Synthesis to New Peaks
Authors: Jingjing Ren
Wenbo Li
Haoyu Chen
Renjing Pei
Bin Shao
Yong Guo
Long Peng
Fenglong Song
Lei Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image generation
Diffusion Models
Primary area: diffusion_based_models

Abstract: Ultra-high-resolution image generation poses great challenges, such as increased semantic planning complexity and detail synthesis difficulties, alongside substantial training resource demands. We present UltraPixel, a novel architecture utilizing cascade diffusion models to generate high-quality images at multiple resolutions (\textit{e.g.}, 1K, 2K, and 4K) within a single model, while maintaining computational efficiency. UltraPixel leverages semantics-rich representations of lower-resolution images in a later denoising stage to guide the whole generation of highly detailed high-resolution images, significantly reducing complexity. Specifically, we introduce implicit neural representations for continuous upsampling and scale-aware normalization layers adaptable to various resolutions. Notably, both low- and high-resolution processes are performed in the most compact space, sharing the majority of parameters with less than 3$\%$ additional parameters for high-resolution outputs, largely enhancing training and inference efficiency. Our model achieves fast training with reduced data requirements, producing photo-realistic high-resolution images and demonstrating state-of-the-art performance in extensive experiments.

UltraPixel: Advancing Ultaa-High-Resolution Image Synthesis to New Peaks
Jingjing Ren'; Wenbo Li Haoyu Chen', Renjing Pei?, Bin Shao", Yong Guo Long Peng', Fenglong Song?, Lei Zhu 1,4 HKKST (Guangzhou) Huawei Noah's 's Ark Lab SMPI "HKUST
Project page: attps://jaae ingren enabb.................. github. watthu..oouuuteee iouuttaiie/t
3840x5120
2560x5120
5120X3072
4096x4096
4096x2048
2560x1536
4096X4096
Figure 1: The proposed UltraPixel creates highly photo-realistic and detail-rich images at various resolutions. Best viewed zoomed in. All image prompts in this paper are listed in the appendix.
'Joint first authors
'Coresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Abstract
Ultra- Ittra-bigh-resolttion image generation poses great challenges, such increased semantic planning complexity and detail synthesis difficulties, alongside substantial training resource demands. We present UltraPixel, a novel architecture utilizzig cascade diffusion models to generate high-quality images at multiple resolutions (e.g., 1K to 6K) within single model, while maintaining computational effi- ciency. UltraPixel leverages semantics-rich representations of lower- lower-resolution images in the later denoising stage guide the whole generation of highly detailed high-resolution images, significantly reducing complexity. Furthermore, we intro- duce implicit neural representations for continuous upsampling and scale-aware normalization layers adaptable to various resolutions. Notably, both low- and high- resolution processes are performed in the most compact space, sharing the majority of parameters with less than 3% additional parameters for high-resolution out- puts, largely enhancing training and inference efficiency. Our model achieves fast training with reduced data requirements, producing photo photo-realistic high-resolution images and demonstrating state-of ofte--of-the-ar performance in extensive experiments.
Introduction
Recent advancements in text-to-image (T2I) models, e.g., Imagen [42). SDXL (37]. PixArt-a 4. and Würstchen [36). have demonstrated impressive capabilities in producing high-quality images, enriching a broad spectrum of applications. Concurrently, the demand for high-resolution images has surged due to advanced display technologies and the necessity for detailed visuals in professional fields like digital art. There is a great need for generating aesthetically pleasing images in ultra-high resolutions, such as 4K or 8K, in this domain.
While popular T2I models [37 36 excel in generating images up to 1024 x 1024 resolution, they encounter great difficulties in scaling to higher resolutions. To address this, training-free methods have been proposed that modify the network structure [I5] [2] or adjust the inference strategy [2] 21] to produce higher-resolution images. However, these methods often suffer from instability, resulting in artifacts such as small object repetition, overly smooth content, or unreasonable details. Additionally, they frequently require long inference time [12 14 2T and manual parameter adjustments [15 14 12] for different resolutions, hindering their practical applications. Recent efforts have focused on training models specifically for high resolutions, such as ResAdapter [6] for 2048 x 2048 pixels and PixArt-E Σ [ for 2880 x 2880. Despite these improvements, the resolution and quality of generated images remain limited, with models optimized for specific resolutions only.
Training models for ultra-thighresooluiin image generation presents significant challenges. These models must manage complex semantic planning and detail synthesis while handling increased computational loads and memory demands. Existing techniques, such as key-value compression [3] in attention 440 [ 38 39 35) and fine-tuning small number of parameters (6), often yield sub- optimal results and hinder scalability to higher resolutions. Thus, a computationally efficient method supporting high-quality detail generation is necessary. We meticulously review current T2I models and identify the cascade model [36] as particularly suitable for biira-high-resolution image generation. Utilizing cascaded decoding strategy that combines diffusion and variational autoencoder (VAE), this approach achieves a 42:1 compression ratio, enabling more compact feature representation. Additionally, the cascade decoder can process features at various resolutions, as illustrated in Sec- tion A in the appendix. This capability inspires us to generate higher-resolution representations within its most compact space, thereby enhancing both training and inference efficiency. However, directly performing semantic planning and detail synthesis at larger scales remains challenging. Due to the distribution gap across different resolutions (i.e., scattered clusters in the t-SNE visualization in Figure 2 existing models struggle to produce visually pleasing and semantically coherent results. For example, they often result in overly dark images with unpleasant artifacts.
In this paper, we introduce UltraPixel, high-quality ultra-high resolutsso image generation method. By incorporating semantics-rich representations of low-resolution images in the later stage as guid- ance, our model comprehends the global semantic layout from the beginning, effectively fusing text information and focusing on detail refinement. The process operates a compact space, with low- and high-resolution generation sharing the majority of parameters and requiring less than 3%
2

---

1024x1024
2048X2048
3072x3072
4096x4096
L-SNE
L-SNE
StableCascade
Ours
Figure 2: Illustration of feature distribution disparity across varying resolutions.
additional parameters for the nigh-resolution branch, ensuring high efficiency. Unlike conventional methods that necessitate separate parameters for different resolutions, our network accommodates varying resolutions and is highly shource-friendly. We achieve this by learning implicit neural representations to upscale low-resolution features, ensuring continuous guidance, and by developing scale-aware, learnable normalization layers to adapt to numerical differences across resolutions. Our model, trained on I million high-qualig, images of diverse sizes, demonstrates the capability to produce photo-realistic images at multiple resolutions (e.g., from IK to 6K with varying aspect ratios) efficiently in both training and inference phases. The image quality of our method is comparable to leading closed-source T2I commercial products, such as Midjourney V6 (32] and DALL-E 3 [34].
Moreover, demonstrate the application of ControlNet [52] and personalization techniques (20
built upon our model, showcasing substantial advancements this field.
Related Work
Text-guided image synthesis. Recently, denoising diffusion probabilistic models 45
18 have 29 46
refreshed image synthesis. Prominent text-guided generation models 37 3
36
35
42
27 have demonstrated a remarkable ability to generate high-quality images. A common approach is map raw image pixels into more compact latent space, which a denoising network trained to
learn the inverse diffusion process 4 3 37]. The use of variational autoencoders [23 has proven to
be highly efficient and is crucial for high-resolution image synthesis
[33 4T StableCascade [36]
advances this approach by learning more compact latent space, achieving compression ratio of 42:1 and significantly enhancing training and inference efficiency. We build our method on StableCascade primarily due to its extremely compact latent space, which allows for the efficient generation of high-resolution images.
High-resolution image synthesis. Generating high-resolution images has become increasingly popular, yet most existing text-to-image (T2I) models struggle to generalize beyond their trained resolution. A straightforward approach is to generate an image at base resolution and then upscale it using super-resolution methods [51] I0 28 48 8 However, this approach heavily depends on the quality of the initial low-resolution image and often fails to add sufficient details to produce high-quality high-resolution (HR) images. Researchers have proposed direct HR image generation as an alternative. Some training-free approaches [15] 12 21 22 53 26 adjust inference strategies or network architectures for HR generation. For instance, patch- based diffusion 26 employ patch- wise inference and fusion strategy, while ScaleCrafter [15] modifies the dilation rate of convolutional blocks in the diffusion UNet [37 based on the target resolution. Another method [22 adapts attention entropy in the attention layer of the denoising network according to feature resolutions. Approaches like Demofusion 22 and FouriScale [2 design progressive generation strategics, with FouriScale further introducing a patch fusion strategy from a frequency perspective.
Despite being training-free free, these methods often produce higher-resolution images with noticeable artifacts, such as edge attenuation, repeated small objects, and semantic misalignment. To improve HR image quality, PixArt-sigma [3] and ResAdapter [6] fine-tune the base T2I model. However, their results are limited to 2880 x 2880 resolution and exhibit unsatisfied visual quality. Our method leverages the extremely compact latent space of StableCascade and introduces low-resolution (LR) semantic guidance for enhanced structure planning and detail synthesis. Consequently, our approach can generate images up to 6K resolution with high visual quality, overcoming the limitations of previous methods.
3

---

Continuous Upscaling
Guidance Extraction
Generative Diffussion
z
z
Model
Prompt: A portrait of a dog wearing a red polka dot shirt.
Decoder
Decoder
zt
XT2
XT
HR Latent Generation
Cascade Decoding
Figure 3: Method Overview. Initially, we extract guidance from the low-resolution (LR) image synthesis process and upscale by learning an implicit neural representation. This upscaled guidance is then integrated into the high-resolution (HR) generation branch. The generated HR latent undergoes a cascade decoding process, ultimately producing a high-resolution image.
3 Method
Generating ultra-high-resolution images necessitates complex semantic planning and detail synthesis. We leverage the cascade architecture (36] for its highly compact latent space to streamline this process, as illustrated in Figure B Initially, we generate low-resolution (LR) image and extract its inner features during synthesis as semantic and structural guidance for high-
(HR) generation. To enable our model to produce images various resolutions, we learn implicit neural representations (INR) of LR and adapt them to different sizes continuously. With this guidance, the HR branch, aided by scale-aware normalization layers, generates multi-resolution latents. These latents then undergo cascade diffusion and VAE decoding process, resulting in the final images. In Section 3.1 we detail the extraction and INR upscaling of LR guidance. Section 3.2 outlines strategies for fusing LR guidance and adapting our model to various resolutions.
3.1 Low-Resolution Guidance Generation
To address the challenges of high-resolution im- age synthesis, Previous studies 442 [77 have often employed progressive strategy, initially generating low-resolution image and then ap- plying diffusion-basse super-resolution tech- niques. Although this method improves image quality, the diffusion process in the pixel space remains essource-intenivve The cascade archi- tecture (36], achieving 42:1 compression ratio, offers a more efficient approach to this problem.
Self- Anaanaai
Litnrr
Figure 4: Illustraiion of continuous upscaling by implicit neural representation.
Guidance extraction. Instead of relying solely on the final low-resolution output, introduce multi-level internal model representations of the low-resolution process to provide guidance. This strategy inspired by evidence suggesting that representations within diffusion generative models
encapsulate extensive semantic information 49 2 To optimize training efficiency and stability, we leverage features in the later stage, which delineate clearer structures compared to earlier stages. This approach ensures that the high-resolution branch is enriched with detailed and coherent semantic guidance, thereby enhancing visual quality and consistency. During training, the high-resolution image (e.g., 4096 4096) is first down-sampled to the base resolution (1024 1024), then encoded

---

Guidance Fusion
Scale-Aware Norm.
Guidance Fusion
Scale Emb.
Time Block,
Conv.
Attention
Scale Scale-Awar --ware
fh
Time Emb.
Norm. Linear
Time Block
Linear
2(-1
Figure 5: Architecture details of generative diffusion model.
latent 2 (24 x 24) and corrupted with Gaussian noise as
q(z m == (N(t; va.zt I))
(1)
where : 1- -
and 3, is the pre-defined variance schedule for the diffusion process. We then feed of to the denoising network and obtain multi-level features after the attention blocks, denoted as the guidance features
Continuous upsampling. Note that the guidance features g are at the base resolution (24 x 24), while the HR features vary size. To enhance our network's ability utilize the guidance, employ implicit neural representations [33 []] which allow us to upsample the guidance features arbitrary resolutions. This approach also mitigates noise disturbance the guidance features, ensuring effective utilization of their semantic content. As shown in Figure 4 we initially perform dimensionality reduction on the LR guidance tokens via linear operartions for improved efficiency and concatenate them with a set of learnable tokens. These tokens undergo multiple self-attention layers, integrating information from the guidance features. Subsequently. the updated learnable tokens are processed through multiple linear layers to generate the implicit function weights. By inputting target position values into the implicit function, we obtain guidance features g' that matches the resolution of the HR features.
3.2 High- Reshluteso Latent Generation
The nigh-resolution latent generation is also conducted in the compact space (i.e., 96 96 latent for a 4096 4096 image with a ratio of 1:42). significantly enhancing computational efficiency. Additionally, the high-resolution branch shares most of its parameters with the low-resolution branch, resulting in only minimal increase in additional parameters. In detail, to incorporate LR guidance, we integrate several fusion modules. Furthermore, we implement resolution-aus aware normalization layers to adapt our model to varying resolutions.
Guidance fusion. After obtaining the guidance feature g', fuse it with the HR feature f follows:
f' = Linear(Concal) (f,g') g')) f
The fused HR feature further modulated by the time embedding e, to determine the extent of LR guidance influence on the current synthesis step:
f" Norm(f (f') @ Linear, + Lineary(e.) + ff.
(3)
With such semantic guidance, our model gains an early understanding of the overall semantic structure, allowing it to fuse text information accordingly and generate finer details beyond the LR guidance, as illustrated Figure
Scale-aware normalization. As illustrated in Figure 2 changes in feature resolution result in corresponding variations in model representations. Normalization layers trained at base resolution struggle to adapt to higher resolutions, such as 4096 4096. To address this challenge, propose resolution-awan normalization layers to enhance model adaptability. Specifically, we derive the scale embedding e, by calculating l NL, where N denotes the number of pixels in the HR
5

---

features (e.g., 96 x 96) and N corresponds to the base resolution (24 x 24). This embedding is then subjected to a muutti-dimessionll sinusoidal transformation, akin to the transform process used for time embedding. Finally, we modulate the HR feature f as follows:
Norm (f) O Linear (e,) + Lineary(e.) +
(4)
The training objective of the generation process is defined as:
== (000 Illco (Zr, 8, g) el12],
(5)
where and g denote scale and LR guidance, respectively. The parameters 0 of the main generation network are fixed, while newly added parameters including INR, guidance fusion, and scale-aware normalization are trainable.
4 Experiments
4.1
Implementation Details
We train models on 1M images of varying resolutions and aspect ratios, ranging from 1024 to 4608, sourced from LAION-Aestheties (44), SAM (24), and self-collected high-quality dataset. The training is conducted on 8 A100 GPUs with batch size of 64. Using model weight initialization from 1024 x 1024 StableCascade [36., our model requires only 15,000 iterations to achieve high-quality results. We employ the Adam W optimizer [30] with a learning rate of 0.0001. During training, we use continuous timesteps in [0, 1] as 36 while LR guidance is consistently corrupted with noise at timestep 0.05. During inference, the generative model uses 20 sampling steps, and the diffusion decoding model uses 10 steps. We adopt DDIM 45] with classifier-fre guidance [19] weight of 4 for latent generation and 1.1 for diffusion decoding. Inference time is evaluated with batch size 1.
4.2
Comparison to State-of-the-An Methods
100%
1022 1742
2048 x 2048
2160x 1940
4098 x 2040
4 4016
077
85.9%
50%
7ILOS9
75.8%
68.7%
0%
Dul-1
Sexlethater
Ozou DrouPusius
Baser BERRGAK
Prizz
Piaari
StaleChafte
Deass DeasPosico
FanniScal
Stalethafter
DeaseF DenuFosso
Foun Founiscale
SublidR
SUPIR
a UltraPixel
D Methods
Figure 6: Win rate of our UltraPixel against competing methods in terms of PickScore 25
Compared methods. We compare our method with competitive high-resolution image gener- ation methods, categorized into training-free methods (ElasticDiffussio [14] ScaleCrafter [13]. Fouriscale 22 Demofusion [2] and training- based methods (Pixart-o B] DALL-E 3 B4,, and
32 For models that can only generate 1024 x 1024 images, we use representative image super-resolution method (51) for upsampling. We comprehensively evaluate the performance of our model at resolutions of 1024 x 1792, 2048 x 2048, 2160 3840, 4096 x 2048, and 4096 x 4096. For fair comparison, we use the official implementations and parameter settings for all methods. Considering the slow inference time (tens of minutes to generate altra-high-resoluse the heavy computation of training-free methods, we compute all metrics using IK images.
Midjourney V6
image) and
Benchmark and evaluation. We collect 1,000 high-quality images with resolutions ranging from 1024 4096 for evaluation. We focus primarily on the perceptual-orientt PickScore (25), which is trained on a large-scale user preference dataset to determine which image is better given an image pair with text prompt, showing impressive alignment with human preference. Although FID [16]
and Inception Score [43 (IS) may not fully assess the quality of generated images 25 3
we report these metrics following common practice. It is important to note that both FID and are calculated on down-sample images with resolution of 299 x 299, making them unsuitable for evaluating

---

Table 1: Quantitative comparison with other methods. Our UltraPixel achieves state-of oftte-ff-theart
performance on all metrics across different resolutions.
Resolution(H x W)
Method
FID,
FID,
ISp
ISt
CLIPt
Latency(sec)
D
88.44
86.16
16.43
18.30
29.66
1024 x 1792
Ours
60.55
63.53
17.84
26.89
35.34
8
ScaleCrafter 13
64.75
73.79
15.41
22.53
31.79
45
ElasticDiffusion 14
77.19
65.37
11.12
21.97
32.95
295
DemoFusion 12
54.86
63.97
13.38
28.07
32.98
2048 2048
FouriScale 21
68.79
86.71
7.70
18.08
30.70
74
Base BSRGAN
ST
48.52
64.00
13.67
29.87
33.53
11+6
Pixart-E
3
54.35
63,96
14.87
27.13
31.18
57
Ours
44.74
62.50
14.95
30,52
35,43
15
Pixart-E 3
49.86
63.87
10.89
25.35
30.86
2160 x 3840
Ours
46.06
62.41
11.91
25.65
34.98
31
ScaleCrafter 15
101.58
120.71
9.04
12.15
23.71
190
DemoFusion
12
51.16
75.28
10.81
21.83
29.95
325
4096 x 2048
FouriScale
21
128.03
137.16
3.82
10.41
21.98
197
Ours
42.60
64.69
11.76
25.36
34.59
ScaleCrafter
15
74.02
98.11
9.07
14,53
31.79
580
DemoFusion
12
47.40
61.11
9.99
26.40
33.14
728
4096 x 4096
FouriScale
21
72.23
105.12
8.12
14.81
27.73
573
StableSR
48
48.18
65.27
9.25
27.55
32.49
728
SUPIR
50
46.98
64.13
9.83
26.16
31.28
682
Ours
44.59
62.12
10.27
27.69
35.18
78
high-resolution image quality. Therefore, we adopt FIDD-patch and IS-patch for more reasonable
measure. Finally, we evaluate image-text consistency using the CLIP score 7
Quantitative Comparison. As mentioned, PickScore aligns closely with human perception, so we it as our primary metric. FigureBjshows the win rate of UltraPixel compared other methods. Our approach consistently delivers superior results across all resolutions. Notably, UltraPixel is preferred in 85.2% and 84.0% of cases compared the training-based Pixart-E [3] despite Pixart- using separate parameters for different resolutions and training on 33M images, whereas our model uses the same parameters for varying resolutions and is trained on just 1M images. UltraPixel also shows competitive performance compared to advanced T2I commercial product DALL-E 3 [34] yielding win rate of 70.0%. Continuous LR guidance enables our resolution-awee on detail synthesis, resulting in higher visual quality. Furthermore, as shown in Table [ our method performs competitively on FID, IS, and CLIP scores across different resolutions. Training-free HR generation methods 12 15 21 [4] struggle to produce high-quality 4096 x 2048 images, showing limited generalization ability. Our UltraPixel also excels in inference efficiency, generating a 2160 x 3840 image in 31 seconds, which is nearly 3.6x faster than Pixart- (111 seconds). Compared to training-fret -ree methods that take tens of minutes to generate a 4096 x 4096 image, our model is significantly more efficient, being 9.3x faster than DemoFusion [12]. These results highlight the effectiveness of our method in generating hiitr-iihh-rssotttionn images with excellent efficiency.
model focus
Qualitative comparison. Figure 7 illustrates a visual comparison between our UltraPixel and other high-resolution image synthesis methods at various resolutions. Training-fre free methods like ScaleCrafter [15] and FouriScale [21] often produce visually unpleasant structures and large areas of irregular textures, significantly degrading visual quality. DemoFusion [12] suffers from severe small object repetition due to its patch-by-pace generation approach. Compared to Pixart- Σ 3 our method excels in generating superior semantic coherence and fine- fine-grained details. For instance, in the 2160 3840 resolution case, our generated camel and human faces exhibit richer details. Despite using a single model to generate images at different resolutions, our method consistently produces visually pleasing and semantically coherent results. Besides, as illustrated in Figure B.5 B.6 and B.7 of appendix, our method produces images quality comparable to those generated by DALLE3 and Midjoumey V6.
4.3 Ablation Study
In this section, for computational efficiency, we train all models with 5K iterations. Unless otherwise stated, the results reported at resolution of 2560 x 2560.
7

---

ElastiDDiflusion
ScaleCrafter
DemoFusion
FouriScale
Pixart-sigma
BARe-BSRRGNN
Ours
198002211 21600x3891
Pixart-sigma
Ours
ScaleCrafter
DemoFusion
FouriScale
Ours
19996 Peoo0due
FouriScale
SealeCrafter
Demoiusion
Ours
Visual Comparison of our UltraPixel and other methods. Our method produces images of ultra-high resolution with enhanced details and superior structures. More visual examples provided in the appendix.
Figure 7:
LR guidance. Figure 8 visually demonstrates the effectiveness of LR guidance. The synthesized HR result without LR guidance exhibits noticeable artifacts, with a messy overall structure and darker color tone. In contrast, the HR image generated with LR guidance is of higher quality, for instance, the characters "accepted" on the sweater and the details the fluffy head are more distinct. Visualization of attention maps reveals that the HR image generation process with LR guidance shows clearer structures earlier. This indicates that LR guidance provides strong semantic priors for HR generation, allowing the model to focus more detail refinement while maintaining better semantic coherence. Additionally, Figure 9 compares our method to the post-processing super-resolution strategy, demonstrating that UltraPixel can generate more visually pleasing details.

---

Resats
ACCETED
Accepted
Aaccceppte
Atcatiin
LR
Guidance
I.R.
Figure 8: Ablation study on LR guidance. Leveraging the semantic guidance from LR features allows the HR generation process to focus on detail refinement, improving visual quality. Text prompt: In the forest, a British shorthair cute cat wearing a yellow sweater with "Accepted" written on it. small cottage in the background, high quality, photorealistic, 4k.
LR
BSRGAN
Ours
Figure 9: Visual comparison with super-resolution method BSRGAN [5 at resolution of 4096 4096. Super-resolution has limited ability to refine the details of the low-resolution image, while our method is capable of generating attractive details.
Timesteps of LR guidance extraction. We analyze the effect timesteps used to extract LR guidance in Table 2 and Figure 10 We consider three cases: [ t= ( where LR guidance is synchronized with the HR timesteps; / 0.5, representing a fixed guidance at the middle timestep: and 0.05, near the end. The results show that t " produces poor CLIP score. This can be attributed to the necessity of providing semantic structure guidance early on, but the LR guidance too noisy at this stage to be useful. Similarly, f 0.5 also results in noisy LR guidance, in Figure g Conversely, = 0.05 provides the best performance since features in the later stage of generation exhibit much clearer structural information. With semantics-rie guidance, HR image generation can produce coherent structures and fine- grained details, yielding higher scores in Table 2
Implicit neural representation (INR). To incorporate multi-resolution capability into our model, we adopt an INR design to continuously provide informative semantic guidance. In Table we compare continuous INR upsampling (dubbed "INR") with directly upsampling LR guidance using bilinear interpolation followed by convolutions (denoted as "B" + Conv"). The results show that INR yields better semantic alignment and image quality, it provides consistent guidance of LR features across varying resolutions. Figure IT further illustrates that directly upsampling LR guidance introduces significant noise into the HR generation process, resulting in degraded visual quality.
Scale-aware normalization. As illustrated in Figure 2 features across different resolutions vary significantly. To generate higher- quaberyy results, we propose scale-aware normalization (SAN). Table compares the performance of models with ("INR SAN") and without ("INR") this design. When scaling the resolution from 2560 x 2560 to 4096 x 4096, the CLIP score gap noticeably enlarges, indicating better textual alignment with SAN. Additionally, the Inception Score shows significant improvement when adopting SAN, validating the effectiveness of our design.

---

Table Ablation study Table 3: Ablation on INR and SAN.
Table 4:
Ablation on the number of trainable parameters.
on timesteps of LR guid- ance extraction.
BI + Conv
INR INR SAN
2560² CLIPT
32.41
32.72
33.09
Base LoRA Ours-512 Ours-1024
t 0.5t
0.05
ISt
26.81
27.62
29.14
Param (M)
106
65
CLIP 31.14
32.75
33.09
4096
CLIPT
31.90
31.93
32.87
CLIP
30.39
31.20
32.78
33.09
IST 25.37
28.15
29.14
1St
22.22
25.22
27.15
ISt
20.89 22.73
27.43
29.14
HHR (t= D00
Figure 10: Visual comparisons of guidance across different timesteps. "LR" depicts the low-resolution generation process, whereas the other "HR" cases illustrate the high-resolution process under varying guidance. When employing synchronized structure information provided is messy, while t = 0.05 offers semantics-rich and clear directives.
t") or middle timestep (t 0.5) guidance, the
1024x10224
2048 Without
4096XA096 Without INR
20-BK2D4B With
6050x M99S
INR
Figure 11: Illustration of Implicit neural representation (INR) to provide consistent guidance.
Number of trainable parameters. Our model benefits from high training efficiency, partly because use limited number of trainable parameters based StableCascade [36). Table iillustrates the impact of the number of trainable parameters. Since most new parameters are in the INR module, we can reduce the channel dimension of LR features from 2048 to a lower number. We explore models with LR dimensions of 512 and 1024 and also include a LoRA [20] version with rank of 48. Compared to the "LoRA" model, "Ours-512" produces better results with fewer parameters. Increasing the channel number from 512 to 1024 "'Ours-1044" achieves higher visual quality and better text-image alignment. To balance efficiency and performance, we choose 1024 as the default.
5 Conclusion
We present UltraPixel, an efficient framework for generating high-quality images varying resolu- tions. Utilizing an extremely compact latent space, we introduce low-resolution (LR) guidance to simplify the complexity of semantic plamning and detail synthesis. Specifically, semantics-rich LR features provide structural guidance for high-resolution image generation. To enable our model to handle varying resolutions, we learn an implicit function to consistently upsample LR features and insert scale-aware normalization layers to adapt feature distribution. UltraPixel efficiently generates stunning, heera-tigg-resoluuton images of varying sizes, elevating image synthesis new heights.
Broader Impacts and Limitation
Despite the advancements in UltraPixel. the limited quantity and quality training datasets constrain the realism and quality of our generated images, especially in complex scenes. This issue underscores the ongoing challenges in achieving true photorealism, and we are committed to further exploring this area in future research.
10

---

7 Acknowledgments
This work is supported by the HHangghhou-HKSSTCZZZ Joint Funding Program (No. 2023A03J0671) the Guangzhou Municipal Science and Technology Project (Grant No. 2024A04J4230), Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things(No.2023 T0TBB1112000110 1212010077, and the National Natural Science Foundation of China (Project No. 61902275).
References
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[2] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126 2021.
[3] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak strong training of diffusion transformer for 4k -imtage generation. arXiv preprint arXiv:2403.0m 04692, 2024.
[4] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-imaae synthesis, arXiv preprint arXiv:2310.00426, 2023.
[5] Yinbo Chen and Xiaolong Wang. Transformers neta-learmers for implicit neural representations. In European Conference on Computer Vision, pages 170-187. Springer, 2022.
[6] Jiaxiang Cheng, Pan Xie, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, and Lean Fu. Resadapter: Domain consistent resolution adapter for diffusion models. arXiv preprint arXiv:2403.0208: 2024.
[7] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[8] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang, Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1065-11074, 2019.
[9] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, Cogview: Mastering text-to-image generation via transformers. Advances Neural Information Processing Systems, 34:19822 1985952999935, 2021.
[10] Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295-307, 2015.
[11] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1192 2020.
[12] Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with SSS. arXiv preprint arXiv:2311.1697. 2023.
[13] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[14] Moayed Haji-Ali, Guha Balakrishnan, and Vicente Ordonez. Elasticdiffusion: Training-free arbitrary size image generation. arXiv preprint arXiv:2311. 2023.
[15] Yingqing He, Shaoshu Yang. Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang. Ran Hc, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two scals update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint urXi::2100.02003, 2022.
[18] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
[19] Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. arXiv preprint arXiv:220..12598 2022.
[20] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. arXiv preprint arXiv:2 09685, 2021.
[21] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song. Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403. 12963, 2024.
[22] Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Training-fres diffusion model adaptation for variable- sized text-to-image synthesis. Advances in Neural Information Processing Systems, 36, 2024.
[23] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61144 2013.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[25] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[26] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. Advances in Neural Information Processing Systems, 36:50648-50660. 2023.
[27] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing acsthetic quality text-to-image generation. arXiv preprint arXiv:2402. 17245, 2024.
[28] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. Proceedings of the IEEE/CVF international conference on computer vision, pages 1833-1844, 2021.
[29] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 120.233 2023.
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10X10 2017.
[31] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfea- tures: Searching through time and space for semantic correspondence, Advances in Neural Information Processing Systems, 36, 2024.
[32] Midjourney. Midjourney v6, 2023. https//www..iilouureyy.coott
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
[34] OpenAL. Dall-e 3, 2023. https//opeenai.comddllleaa
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[36] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. Würstchen: An efficient architecture for scale text-to-image image diffusion models. In The Twelfth International Conference Learning Representations, 2023.
12

---

[37] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0190 2023.
[38] Sucheng Ren, Zeyu Wang. Hongru Zhu, Junfei Xiao, Alan Yuille, and Cihang Xie. Rejuvenating image-gpi gpt as strong visual representation learners. Forty-first International Conference on Machine Learning. 2023.
[39] Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 3687-3697, 2023.
[40] Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[41] Robin Rombach, Andreas Blattmann. Dominik Lorenz. Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[43] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[44] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 20255278-22994, 2022.
[45] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02.502, 2020.
[46] Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 15:10011-1003, 2022.
[47] Jianyi Wang. Zongsheng Yue, Shangehen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolee resolution.... International Journal of Computer Vision, pages 1-21, 2024.
[48] Xintao Wang. Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision. pages
2021.
[49] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEL/CVF 2023.
Conference on Computer Vision and Pattern Recognition, pages
[50] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong. Scaling up excellence: Practicing model scaling for photo-realistic image restoration in the wild. In Proceedings of the IEEL/CVF Conference on Computer Vision and Pattern Recognition, pages 25669-25680, 2024.
[51] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791-4800, 2021.
[52] Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3847, 2023.
[53] Shen Zhang, Zhaowei Chen, Zhenyu Zhao, Zhenyuan Chen, Yao Tang, Yuhao Chen, Wengang Cao, and Jiajun Liang. Hidiffusion: Unlocking high-resolution crcativity and efficiency in low-rcsolution trained diffusion models. arXiv preprint arXiv:2311.17528 2023.
13

---

Appendix
In Section A we first demonstrate that the latent space of StableCascade [36] can accommodate images with various resolutions and compare the reconstruction quality with SDXL [37]. Subse- quently, in Section] we provide additional visual comparisons with the super-resolution method, cutting-edge high-resolution generation techniques, and leading closed-source T2I products. We also present more high-resolution results of our method in Section C Next, we illustrate how our model can be customized for controllable generation and personalization in Section D Finally, include text prompts for the images generated, presented in both the main document and the appendix in Section H
A Latent Space of StableCascade
As illustrated in Figure A.1
StableCascade [36] achieves high compression ratio of 42:1 while
capably reconstructing images of varying sizes with promising quality, Although there is some loss of detail, this is considered acceptable given the significant efficiency gains in both training and inference that the high compression ratio facilitates. In contrast, as shown in Table A.I SDXL [37] has lower compression ratio 8:1 and obtains higher PSNR scores, indicating superior fidelity between the reconstructed images and the original high-resolution inputs. Considering the trade-off between efficiency and accuracy, emphasize the value of StableCascade's compact representation and its suitability for ultra-high-resolutinn generation applications.
6656X4352
5120x7936 7936
Input
StableCascade
SDXL
Figure A.I: Visual comparison of reconstruction quality between VAEs of StableCascade
36 and
SDXL. 37 on high-resolution images.
Table A.1: Quantatitive comparison of reconstruction quality and complexity between StableCas-
cade 36 and SDXL [37]
StableCascade
SDXL
PSNRT
Compress Ratio
# of Params (M)
PSNRT
Compress Ratio
of Params (M)
30.87dB
1520
33.08dB
8:1
14

---

B Additional Comparison Results
Comparison with SR method. A common method to obtain high-resolution images involves initially generating a low-resolution image and then upsampling with an off-the-shelf super- resolution (SR) model. In Figure [B.2 and B.3] compare the results produced by our UltraPixel method and the advanced super-resolution techniques, BSRGAN [11] StableSR [47] and SUPIR (50). It is evident that the SR method often fails introduce adequate details; although the resolution increases, the image quality does not improve proportionately In contrast, our UltraPixel method excels by incorporating an abundance of intricate details, significantly enhancing the visual quality of the images.
IRR
BSRGAN
Qurs
Figure B.2: Visual comparison with BSRGAN
51 at 4096 4096 resolution.
T0
AOHASKIING SUPIRA
40%6x40% SulesR
1024K 1024
4016X4401 SUPER
4096X41198 NableS
4016X4196 Ouzx
Figure B.3: Visual comparison with generative iiffusion-bbsed super-resolution method StableSR 47 and SUPIR [50] at 4096 x 4096 resolution.
Comparison with high-resolution image generation methods. We present additional visual comparisons with state-of-the-as high-resolution image generation methods in Figure B.4 The results generated by our UltraPixel method consistently outperform others across various resolutions, highlighting its superior capability.
Comparison with closed-source T2I products. We offer further visual comparisons between our Ultraixel and closed-source commercial text-to-image (T2I) products: DALL-E 3 34 in Figure B.5
and Midjourney V6 [32] Figures B.6 and B.7 Our method showeases the ability to generate high-quality images that par with these leading commercial products.
15

---

2048x2018
ScaleCrafter
DemoFusion
FouriScale
Ours
180008840
2160
Pixart-sigma
Ours
4096x4096
ScalcCrafter
Fouriscale
DemoFusion
Ours
Figure B.4: Visual comparison with high-resolution image generation methods.
C
Additional Visual Results
We present more visual results of UltraPixel in Figure C.8 C.16 Our method produces images of diverse resolutions with excellent quality, excelling in a range of scenarios from close-up portraits and imaginative content to photo-realistic scenes.
C.9 C.10
C.I3 C.14 C.I5
D Controllable High- High-Resolution Image Synthesis
Spatial control. We present high-resolution (HR) results controlled by edge maps. Notably, we do not train our models directly; rather, utilize the officially released control weights from StableCascade [36]. These control features are integrated during the low-resolution (LR) guidance extraction process. The results are demonstrated in Figures D.17 (D18 Currently, the maximum supported resolution 3K. Further fine-tuning of the control weights will enable support for higher resolutions.
Personalization. Figure D.19 demonstrates high-resolution personalized results based on a user- provided instance. Specifically, optimize the model parameters of the attention layers using
16

---

Ours
Dall-E 3
Figure B.5: Visual comparison with Dall-E 33 at 1024 X 1792 resolution.
17

---

Ours
Midjourney V6
Figure B.6: Visual comparison with Midjourney V6
at 2048 2048 resolution.
18

---

Ours
Midjourncy V6
Figure B.7: Visual comparison with Midjourney V6
2048 2048 resolution.
19

---

Figure C.8: Visual results of UltraPixel at 5120 x 2560 resolution.
20

---

Figure C.9: Visual results of UltraPixel at 5120 x 3840 resolution.
21

---

with rank of 4. The training involves an initial phase at base resolution for 5,000 iterations, followed by fine-tuning at higher resolution for additional 5,000 iterations. Figure[.119 showcases our method's capability to incorporate personalized techniques for achieving personalized high-resolution image generation.
LoRA 20
E
Text Prompts
Text prompts are provided in Table E.2
E.3
22

---

Figure 10 Visual results of UltraPixel at 3072 x 6144 resolution.
23

---

Figure C.I1: Visual results of UltraPixel at 3072 x 6144 resolution.
24

---

Figure 12 Visual results of UltraPixel at 5120 x 2560 resolution.
25

---

Figure C3: Visual results of UltraPixel at 3840 x 2160 resolution.
26

---

Figure C. Visual results of UltraPixel at 2880 x 5760 resolution.
27

---

Figure C.15: Visual results of UltraPixel at 4096 x 4096 resolution.
28

---

2160x3840
4096x 4096
4096x 4096
$120x38-0 3840 O.
5120x3840
4096X4096
5120x3072
2160x3840
Figure C.16: Visual results of UltraPixel.
29

---

Figure D.17: Edge-controlled results of UltraPixel at 3072 x 3072 resolution.
30

---

Figure D.18: Edge-controlled results of UltraPixel at 2160 3840 resolution.
31

---

User Input
2048x2048
4096x4096
3840x2160
4096x3840
Personalized Results
4096x2560
2048x4096
Figure D.19: Personalization results of UltraPixel.
32

---

Table E.2: Text prompts of figures in this manuscript. For each figure, prompts are provided in order from left to right, top to bottom. (to be continued)
Figure
Text Prompt
A close-up of blooming peony, with layers of soft. pink petals, delicate frugrance, and dewdrops glistening in the carly morning light.
A serene mountain landscape with towering snow-capped peaks, a crystal-clear blue lake reflecting the mountains, dense pine forests, and a vibrant orange sunrise illuminating the sky.
An expressionist landscape with twisted trees, a turbulent sky, and a path leading to unknown destination,
painted in vivid, unsettling colors and expressive brushstrokes that convey sense of anxicty and movement. A close-up portruit a young woman with flawless skin, vibrant red lipstick, and wavy brown hair, wearing vintage floral dress and standing in front of blooming garden.
Figure 1
The imuge features snow-covered mountain range with large, snow-covered mountain in the buckground.
The mountain is surrounded by a forest of trees, and the sky is filled with clouds. The scene is set during the
winter scason, with snow covering the ground and the trees.
A statue of person holding torch A pair of cuddly rabbits, one white with floppy ears and the other brown with a twitching nose, snuggling lugether in a vozy hutch filled with straw.
Figure 2
Ford anglia van
Idaho wedding party
A man standing next to a camel
Figure
A wooden wagon yard
Crocodile in a sweater
A vibrant anime scene of young girl with long. flowing pink hair, big sparkling blue eyes, and school uniform, standing under cherry blossom tree with petals falling around her. The background shows traditional Japanese school with cherry blossoms in full hloom.
Figure
A playful Labrador retriever puppy with a shiny, golden coal, chasing a red ball in spacious backyard, with green grass and a wooden fence.
Figure 10
Dogs sitting around poker table
Figure
Bluirgowrie Holiday Park, Bluirgowrie, Perthshire Head Outside
A cozy. rustic log cabin nestled in snow-covered forest, with smoke rising from the stone chimney, warm lights glowing from the windows, and path of footprints leading the front door.
Figure B.2
A striking close-up of a young boy with curly blonde hair and bright green eyes, his face sprinkled with freckles. He is smiling widely, showcasing a gap-toothed grin, and the background a sunny. out-of-focus playground.
Campsite with picnic table surrounded by boulders and green plants. Jocy Fatone Hosts The Price Is Right- Live Show At Bally's Las Vegas steam rises from a geyser in mountainous area
Figure B,5
FOUNTAINE PAJOT Greenland 34
A person with backpack and skis in the snow
A charning depiction a koala resting in eucalypius tree, with the soft gray fur and the lush green leaves creating peaceful seene.
Figure B.6
brown wooden house in the middle of snow covered trees
Smiling woman in white shirt
A highly detailed, high-quality image of the Banff National Park in Canada. The turqunise waters Lake Louise are surrounded by snow-cupped mountains and dense pine forests. wooden cance docked the edge of the lake. The sky is a clear, bright blue, and the air crisp and fresh.
Figure B.7
A highly detailed, high-quality image of a Shih Tzu receiving hath in home bathroo.. The dog standing in tub, covered in suds, with a slightly wel and adoruble look. The background includes buthroom fixtures, towels, and clean, tiled floor.
SSt. Basil's Cathedral
2014 brabus 663s 700 6x6 mercedes benz g class hd pictures. Black Bedroom Furniture Sets. Home Design Ideas
Figure R.4
Exi for in lig and se gallinburg cabin wahoo sale night cubins rentals of american homes in log city
Figure C,8
A tiger is playing football.
A detailed view of a blooming magnolia tree, with large, white flowers and dark green lcaves, set against a clear blue sky.
Figure C.9
A highly detailed, high-quality image of the Patagonia region in Argentina. Towering mountains with snow- covered peaks rise ahove pristine lakes and dense forests. Glaciers he seen in the distance, reflecting the bright sunlight. The sky is deep blue with scattered white clouds.
Figure C.10
A highly detailed, high-quality image of Shih Tzu receiving bath in home bathroom. The dog standing in tub, covered in suds, with a slightly wet and adorable look. The backgruund includes hathroom fixtures, towels, and a clean, tiled floor.
Figure C.11
Figure C2
Flowrider taking his chow chow for a walk.
Figure C3
A laughing woman
33

---

Table E.3: Text prompts of figures in this manuscript. For each figure, prompts are provided in order from left to right, top to bottom. (continued)
Figure
Prompt
Adventure Romance Trip Ideas tree outdoor sky grass retlection Nature wilderness tarn mountain mountain- ous landforms water leaf nature reserve Lake highland pond mount scenery loch national park fell landscupe bank biome cloud wetland auturnn hill tundra mountain runge River valley larch meudow Forest surrounded lush hillside
Figure C.14
A detailed and realistic painting of a vintage steam train, with intricate details in the muchinery and sense motion and power, capturing the nostalgia and romance of era, highly detailed, high quality
Figure
Ice Kingdom; A stunning ioc kingdom with crystalline castles and frozen landscapes. The castles are made entirely of ice, with spires that sparkle the sunlight. Snow-covered trees and icy pathways lead the grand palace at the heart of the kingdom. where the icc queen resides. The sky above is brilliant bluc. with snowflakes gently falling.
A detailed macro shot of a daisy, showcasing its white petals and bright yellow center, with tiny insects like bees and butterflics hovering nearby.
A high-fashion runway show featuring models in avant- wand-ggadde clothing, dramatic makcup, and elaborate hairstyles, with bright lights and a stylish, modern backdrop.
Anthropomorphic profile the white snow owl Crystal pricstess, art deco painting. pretty and expressive eyes, ornate costume, mythical, ethereal, intricate, claborate, hyperreralism, hyper detailed, 3D, 8K. Ultra Realistic, high octane, ultra resolution, amazing detail. perfection. In frame, photorealistic, cinematic lighting. visual clarity, shading. lumen reflections, super-resolution gigapixel, color grading, retouch, enhanced, PBR, Blender, V-ray, procreate, zBrush.
Art collection style and fashion shoot, the style of made of glass, dark blue and light pink. paul rand. solarpunk, camille vivier, beth didonato hair, barbiecure, hyper-realistic
Figure C.16
A highly detailed, high-quality image of a Scottish Fold cat sitting on bookshelf. The cat's distinctive folded cars and round face give it a unique appearance as sits among books and decorative items. The background feutures well-lit
with wooden shelves and reading nook
Traditional Breukfast: A hearty traditional breakfast with fluffy scrambled eggs, crispy bacon, golden hash browns, and buttered toast. The plate is garnished with fresh parsley and accompanied by a glass of freshly squeezed orange juice and a steaming cup of coffee, The background features a cozy kitchen setting with morning sunbeam streaming through the window.
Space Adventure: A thrilling scene of spaceship navigating through asteroid field. The ship sleek and futuristic, with glowing thrusters and advanced weaponry. The astervids are large and rugged. illuminated by the light distant stars, the kackground, nebula vibrant hues of blue and purple adds touch of cusmic beauty the scene.
A medieval knight shining armor. standing proudly in a lush, green field dotted with wildflowers, with grand stone castle and rolling hills in the background.
Several brightly colored rocks on colorful beach, in the style luminous spheres, emek golan, transluceni color, 32k uhd, toyen, cuptivating
An East Asian girl with simple wreath
Figure D.17
A Pacific Islander girl with tropical flower crown, against backdrop of a pristine beach at sunset, in vibrant, colorful painting style.
Small cottage near the lake, snow, winter. Small cottage near the lake, summer.
Figure D.18
A cinematic photo cat [roubao] in space suit. [roubao] with flower.
Figure D.19
cinematic photo
A cinematic photo of cat [roubao] in white chef suit.
A cinematic photo of cat [roubao] in red outdoor jacket, Pixar anime style.
A cinematic photo cat [roubao] playing the piano, oil painitng style.
cinematic photo [roubao] with black suit and sunglasses, the beach.
34

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4.2 of main paper and Section B of appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Section
35

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Some key implementation details are provided in Section and have shared code link
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section and the ul ultrapixel//e to the project homepage below the author information.
.it.ubitttuu..... io/
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code repository link is provided in the home page https: // j bangiiingeeabb.githhue 10/ io/ultrapixel/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c http://dddee/oeeuummieeeeeee ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
37

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
38

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://euurpps.c http://aaarra..ccoobb/ol./ccccle
Answer: [Yes]
Justification: The research has conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [Yes]
Guidelines: Section 6
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Section 6
Guidelines:
39

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
40

---

Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
41


Reviews:
[{"review_id": "4axynMPINH", "reviewer": "Reviewer_rwnV", "summary": "This paper presents a method for Ultra-High-Resolution image generation from text prompts. The method is based on StableCascade. The original StableCascade can generate 1024x1024 images. This paper proposes another HR latent diffusion model that can utilize the guidance from 1024 x 1024 images and generate 4096 x 4096 images. Unlike previous methods that directly use the low-resolution output, the method chooses to use the features of the base model as guidance and proposes an implicit-based method to upsample the low-res guidance features.", "questions": "- Is the base model frozen from StableCascade?\n- Is the implicit model jointly trained with the HR latent model?", "limitations": "It will be good to show some visual failure cases.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of guidance feature and implicit-based upsampling is simple but effective.\n- The paper reads well, and the presentation is clear.\n- The results are very impressive. \n- The proposed method only needs light-weight finetuning from StableCascade.", "weaknesses": "- More validation and analysis are needed. In the comparison, a traditional image upsampler is used, but the traditional image upsampler is often smaller and also trained on much smaller datasets. For a fair comparison, it will be good to compare with the state-of-the-art generative image upsampler such as StableSR and Stable Diffusion Upscaler.\n- A comparison of this baseline is missing: instead of using guidance features, the HR latent model can directly use the LR images / latents from the base model.\n- It would be good to have visual results of the ablation on LR guidance timesteps.\n- Ablation on scale-aware normalization is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1` **Comparison with the SOTA generative upsampler**\n\nThank you for the valuable suggestion. We compare our method with SOTA diffusion-based SR methods, namely **SUPIR** and **StableSR**. The visual results in **Figure 1** of the rebuttal PDF demonstrate that our method produces more reasonable structures and richer details. As a T2I task, our objective is to generate a high-quality, high-resolution image aligned with the text input, rather than merely enlarging the synthesized low-resolution (LR) image. Notably, our method excels in refining images that contain artifacts in the LR version, enhancing visual quality, e.g., facial and hand details in 2nd example. Besides, processing a 4096x4096 image takes **12 minutes** for SUPIR and **11 minutes** for StableSR on an A100 GPU, while our model requires only approximately **78 seconds**. We also evaluate the PickScore, FID, patch-based FID,Inception-score (IS), patch-based IS,  CLIP-score on resolution of 4096x4096, as shown below, where our method achieves superior results. These findings collectively underscore the effectiveness of our approach. We have added these comparisons in the revised manuscript.\n|       method   | PickScore (vs StableSR) $\\uparrow$ |PickScore (vs SUPIR)$\\uparrow$| FID$\\downarrow$| FID_p$\\downarrow$| IS$\\uparrow$| IS_p$\\uparrow$| CLIP$\\uparrow$| Latency(sec.)$\\downarrow$|\n|---|---|---|---|---|---|---|---|---|\n| StableSR           | 31.3%    | - | 65.27| 48.18|27.55|9.25|32.49|728\n| SUPIR            | -    | 34.7%   |64.13|46.98|26.16|9.83|31.28|682\n| Ours | **68.7%**  | **65.3%**|**63.80**|**44.32**|**27.65**|**10.23**|**33.10**|**78**\n\n`Q2` **Comparison with the baseline using latent of base model**\n\nAs suggested, we quantitatively and qualitatively compare our UltraPixel method with the baseline that **directly uses the low-resolution (LR) latent generated by the base model (an SR pipeline)**. We evaluate the PickScore, FID, patch-based FID,Inception-score (IS), patch-based IS,  CLIP-score on resolution of 2048x2048 in the table below. As shown in the table below, our method consistently outperforms the baseline across all metrics, achieving a **71.1%** win-rate on PickScore . **Figure 3** in the rebuttal PDF illustrates that using LR latents causes the model to overly rely on the LR input, often failing to refine details further. In contrast, our approach, which utilizes multi-level semantics-rich features, provides  semantic-rich guidance and allows for additional refinement. Consequently, our method produces higher-quality images with more reasonable structures and richer details. We have included these details in the revised manuscript.\n|       method   | PickScore $\\uparrow$| FID$\\downarrow$| FID_p$\\downarrow$| IS$\\uparrow$| IS_p$\\uparrow$| CLIP$\\uparrow$\n|---|---|---|---|---|---|---\n| Baseline | 28.9%   |63.32|48.19|26.73|11.88|30.38\n| Ours | **71.1%**  |**62.82**|**43.97**|**29.67**|**14.15**|**33.18**\n\n`Q3` **Visual results of the ablation on LR guidance timesteps**\n\n Thank you for the constructive advice. We include visual results of the ablation study on LR guidance timesteps in **Figure 2** of the rebuttal PDF. The figure illustrates the LR generation process and HR generation processes with different guidances, considering three cases: **(1) ‘t=t^H’** where the timestep of LR guidance is synchronized with the timestep of HR generation, **(2) ‘t=0.5T’** means LR features at the middle timestep, and **(3) ‘t=0.05T’** near the end. The HR generation process establishes the overall semantic plan in the early stage of the sampling process. However, synchronized or middle timestep guidance is too noisy to provide sufficient direction at this stage. In contrast, the final step of LR guidance delivers clear semantic information. With semantics-rich guidance, the HR image generation can produce coherent structures and fine-grained details. We have included this visualization in the revised manuscript.\n\n`Q4` **Ablation on scale-aware normalization**\n\nWe have quantitatively evaluated the performance of our model with and without scale-aware normalization (SAN) in **Table 3 of the main paper**. The method labeled ‘INR’ refers to the model without SAN, while ‘INR+SAN’ incorporates SAN. The results indicate that SAN consistently enhances performance across different resolutions. Additionally, we compute feature statistics (mean and variance) for varying resolutions, with and without SAN, as shown in **Figure 5** of the rebuttal PDF. With SAN, the features of different resolutions exhibit similar distributions, while the mean values vary significantly across resolutions without SAN. This demonstrates that SAN enables our model to handle different resolutions stably, resulting in better visual results.\n\n`Q5` **Is the base model frozen from StableCascade**\n\nYes. As described in lines **170-172**, the base model is frozen\n\n`Q6` **Is the base model frozen from StableCascade**\n\nYes. As described in lines **170-172**, the implicit model is jointly trained with the HR branch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VP7U3aFaqE", "reviewer": "Reviewer_z6xV", "summary": "This paper introduces UltraPixel, a method for generating high-quality ultra-high-resolution images. It utilizes the semantics-rich representations of lower-resolution images in a later denoising stage to guide the overall generation of highly detailed high-resolution images. The method incorporates implicit neural representations for continuous up-sampling and scale-aware normalization layers that are adaptable to various resolutions. The experimental results show that it has excellent ability in generating high-resolution images of different sizes.", "questions": "1. Why is the perturbed version $𝑧_{1}$ preferred over the $𝑧_{0}$ from the low-resolution synthesis for guidance purposes?\n\n2. Regarding Figure 4, could you clarify why additional learnable tokens are integrated with the guidance tokens for the self-attention mechanism, instead of solely relying on the guidance tokens? What unique function do these learnable tokens serve?\n\n3. Can you outline the computational steps involved in the implicit neural representation? Is there a need for manually specifying positions?\n\n4. What justifies the forms of Equations (3) and (4), which amalgamate terms with distinct physical interpretations? Is there an underlying principle that supports their direct summation, as it seems to go against intuitive reasoning?\n\n5. In the context of line 255, the use of 𝑡=0.5 and 𝑡=0.05 is ambiguous. Are these intended to denote specific sampling stages within the low-resolution synthesis—fixed and terminal steps, respectively? Consequently, is 𝑡=1 encompassed within the scenario where 𝑡=0.5?", "limitations": "1. The paper may not sufficiently address how well the model generalizes to datasets beyond the training distribution. It is crucial to understand if the model's performance degrades with different or less common image content.\n\n2. There is a need for more rigorous testing of the model's robustness to various corruptions and perturbations that could be encountered in real-world applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of implicit neural representations for continuous up-sampling and scale-aware normalization layers adaptable to various resolutions is a creative solution that addresses a challenge in the scalability of image generation models.\n\n2. The methodology is well-articulated, with a clear explanation of how the model manages to generate high-quality images while maintaining computational efficiency.\n\n3. The ablation experiments are thoroughly conducted, systematically revealing the contribution of each component to the overall performance.\n\n4. The paper proposes an innovative method for generating high-quality, ultra-high-resolution images efficiently, tackling a major challenge in image synthesis.", "weaknesses": "1. The explanation of the implicit neural representation (INR) requires further clarity regarding its ability to enable continuous upscaling. Moreover, an in-depth analysis and dedicated ablation study of the Scale-Aware Normalization (SAN) feature would provide insights into its role in resolution adaptability.\n\n2. To underscore the advantages of the proposed framework, the experiments should be expanded to include comparative analyses with Latent Diffusion Model (LDM)-based and pixel-based image synthesis methods, showcasing the superior performance of the framework in high-resolution image generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1` **Clarification on INR and analysis of SAN feature**\n\nCompared to discrete grid pixels, INR represents data as a neural function, mapping continuous coordinates to signals. Its representation capacity depends not on grid resolution but on the neural network's ability to capture underlying data structures, reducing redundancy and providing a compact yet powerful consistent representation. INR has proven useful in various 3D/2D tasks, such as NeRF[1] and LIIF[2]. To illustrate the effectiveness of INR , we provide visual examples in Fig 6 of the rebuttal PDF. With INR guidance, our model consistently generates high-quality images across different resolutions. In contrast, using simple bilinear upsampling followed by several conv. fails to provide clear guidance for higher resolution images like 4K, resulting in noisy artifacts. The quantitative comparisons of ‘BI+Conv’ and ‘INR’ in Table 3 of the main paper also support this observation.\n\nRegarding SAN, we compute feature statistics for varying resolutions, with and without SAN, as shown in Fig. 5 of the rebuttal PDF. With SAN, different resolutions have similar feature distributions, while without SAN, mean values vary significantly. This shows SAN enables stable handling of different resolutions, resulting in better visuals.\n\n[1] Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV 2020 \n\n[2] Learning continuous image representation with local implicit image function. CVPR 2021 \n\n`Q2` **Compare with LDM-based and pixel-based methods**\n\nOur method is fundamentally a LDM model, performing the generative diffusion process in a latent space. The methods we compared, including Demofusion, ScaleCrafter, FouriScale, and Pixart-Sigma, are all LDM adapted with various techniques for HR image generation. To our best knowledge, no open-source pixel-based synthesis method can directly generate HR images due to the extreme memory and computational demands of doing so in pixel space.\n\nTo demonstrate our method's superiority, we compare it with leading T2I products, DALLE-3 and MJ V6. Quantitative evaluation shows our method is favored by PickScore in 70% of cases against DALLE-3. Visual comparisons in Fig 4 of the rebuttal PDF show our images are of comparable quality to both products.\n\n`Q3` **Why z1 over z0**\n\nWe do not use z1 or z0 latents. Instead, we utilize intermediate multi-level model features by forwarding z1 to the base model.\n\n`Q4` **Why additional learnable tokens**\n\nUnlike LR guidance tokens with a strong local bias, the learnable tokens can globally query and aggregate compact useful information from guidance tokens, enhancing the model’s comprehension  capabilities. Additionally, learnable tokens have proven effective in adapting to various vision tasks, as evidenced by  works [3-5]. While the LR guidance tokens primarily encode local patterns, the learnable tokens acquire auxiliary information from large datasets, improving the adaptability and performance of models.\n\n[3] Vision transformers need registers. ICLR 2024 \n\n[4] End-to-end object detection with transformers. ECCV 2020\n\n[5] Generalized decoding for pixel, image, and language. CVPR 2023\n\n`Q5` **Steps involved in the INR**\n\nAfter obtaining the LR feature, INR is involved in all HR generation sampling steps to forward the LR guidance to the HR branch.\n\n`Q6` **What justify eq 3 4**\n\nEq (3) (4) inject time and image scale information into the model using AdaIN, which predicts scale and shift from the input information to modulate model features. This technique is widely employed and proven effective in injecting style or time information into generative models [6-8].\n\n[6] High-resolution image synthesis with latent diffusion models. CVPR 2022\n\n[7] Sdxl: Improving latent diffusion models for high-resolution image synthesis. ICLR 2024 \n\n[8] A style-based generator architecture for generative adversarial networks. CVPR 2019\n\n`Q7` **Clarification of t**\n\nWe have revised the notation to t=0.5T and t=0.05T. For example, when the number of sampling steps is 20 (T=20), t=0.5T refers to using the LR features at the 10th step to guide all HR sampling steps. As shown in Table 4 of the main paper and Fig 3 of the rebuttal PDF, t=0.05T is preferred, as the LR features at this time point provide clear semantic guidance for HR generation. We do not consider t=T, as the LR features at the first step of the LR sampling stage are too noisy to provide useful information.\n\n`Q8` **Generalization concern**\n\nAs mentioned in lines 177-178, we build our model upon the well-trained 1024^2 StableCascade, which is trained on huge datasets and generalizes well across various scenarios. As described in lines 10-13 and 170-172, we train an additional 3% of the parameters specifically for high-resolution image generation, while keeping the other parameters frozen. This method preserves the model's generative power. As shown in Fig 1 of the main paper, and Fig 1 (“A photo of an astronaut riding a horse in the forest. There is a river in front of them with water lilies.”) and 2 (“Dogs sitting around a poker table”) in the rebuttal PDF, our method can generate less common content of high quality. Fig 8 in the rebuttal PDF also shows that our method can generate pleasing images of various styles (photo-realistic or oil-painting) and content (real or imaginary).\n\n`Q9` **Model robustness**\n\n Our model performs well in situations involving corruptions and perturbations. Fig 7 of the rebuttal PDF shows that even when the prompts are miswritten as \"2014 brabus b63s 700 6x6 mercedes benz g class hd pictures\" and \" Ext for in ldg and sc gatlinburg cabin wahoo sale cabins rentals of american homes tn log city\" our method accurately generates high-quality 4K images. Even if the prompt is incomplete, jumbled, or contains spelling errors, our method still can generate high-quality images. This demonstrates the robustness of our method in understanding and interpreting diverse and imperfect input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5xRQxntKXW", "reviewer": "Reviewer_UnLF", "summary": "The paper presents UltraPixel, an innovative architecture for ultra-high-resolution image generation that tackles semantic planning, detail synthesis, and high resource demands. UltraPixel uses cascade diffusion models to generate images at multiple resolutions within a single model, efficiently guiding high-resolution generation with lower-resolution semantics. It features implicit neural representations for continuous upsampling and scale-aware normalization layers. Moreover, it requires less than a 3% increase for high-resolution outputs, boosting efficiency.", "questions": "1. The discussion in Section 4.3 regarding the timesteps of LR guidance extraction is intriguing. It would be valuable to see final generated images using different timesteps for guidance, rather than just attention visualizations.\n2. The authors' use of LR guidance bears similarities to recent diffusion-based image super-resolution methods. A comparative discussion of these approaches could provide valuable context.\n3. Given the method's design, it should theoretically support even higher resolutions (e.g., 5K, 6K). Have the authors explored this possibility?\n4. The visual results demonstrating the limitations mentioned in the paper could be included in the supplementary materials to provide a more comprehensive understanding of the method's constraints.\n5. Will the code and pre-trained models be made publicly available to facilitate reproducibility and further research in this area？", "limitations": "See Weaknesses.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、The paper demonstrates impressive results, with generated high-resolution images exhibiting remarkable detail. The proposed method outperforms existing approaches in terms of speed and flexibility, supporting arbitrary resolution image generation with a single model. This represents a significant advancement in the field.\n\n2、The authors present a clear and well-motivated approach. They provide compelling evidence (Figures 2 and 6) to support their argument that the absence of low-resolution (LR) guidance can lead to suboptimal generation results.", "weaknesses": "1、 The manuscript's layout requires some refinement. For instance, Figure 4 extends beyond the page margins, and the text adjacent to Figure 9 appears overly condensed.\n\n2、 Given that this is a text-to-image generation work, the paper would benefit from a more comprehensive set of visual results, including additional comparisons with state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1`: **The manuscript's layout requires some refinement.**\n\nThank you for your constructive suggestion. We have carefully revised the format accordingly.\n\n`Q2`: **Paper would benefit from a more comprehensive set of visual results, including additional comparisons with state-of-the-art methods.**\n\nThank you for the valuable advice. In the rebuttal PDF, we provide visual comparisons with diffusion-based SR methods, including SUPIR and StableSR in **Figure 1**, and leading T2I commercial products, DALLE 3 and MidJourney V6 in **Figure 4**. Our results are visually pleasing and richer in detail. We have included more comparisons in the revised supplementary materials.\n\n`Q3`: **Final generated images using different timesteps for guidance**\n\nThank you for the constructive advice. We include visual results of the ablation study on LR guidance timesteps in **Figure 2** of the rebuttal PDF. The figure illustrates the LR generation process and HR generation processes with different guidances, considering three cases: **(1) ‘t=t^H’** where the timestep of LR guidance is synchronized with the timestep of HR generation, **(2) ‘t=0.5T’** means LR features at the middle timestep, and **(3) ‘t=0.05T’** near the end. The HR generation process establishes the overall semantic plan in the early stage. However, synchronized or middle timestep guidance is too noisy to provide sufficient direction at this stage. In contrast, the final step of LR guidance delivers clear semantic information. With semantics-rich guidance, the HR image generation can produce coherent structures and fine-grained details. We have included this visualization in the revised manuscript.\n\n`Q4`: **Comparative discussion of diffusion-based image SR methods**\n\nThank you for the valuable suggestion. As a T2I task, our objective is to generate a high-quality, high-resolution image aligned with the text input, rather than merely enlarging the synthesized low-resolution (LR) image. Unlike SR methods that typically use LR images as input, we adopt multi-level intermediate model features as LR guidance for high-resolution generation. This approach allows for refinement when the LR guidance is unreliable.\n\nWe compare our method with state-of-the-art diffusion-based SR methods, namely **SUPIR** and **StableSR**. The visual results in **Figure 1** of the rebuttal PDF demonstrate that our method produces more reasonable structures and richer details. Notably, our method excels in refining images that contain artifacts in the LR version, enhancing visual quality, particularly in facial and hand details, as shown in the second example. Additionally, processing a 4096x4096 image takes **12 minutes** for SUPIR and **11 minutes** for StableSR on an A100 GPU, while our model requires only approximately **78 seconds**.\n\nWe also evaluate the FID, patch-based FID, Inception, Inception-score, PickScore, clip-score at resolution of 4096x4096 , as shown below, where our method achieves superior results. These findings collectively underscore the effectiveness of our approach. We have included these comparisons in the revised manuscript.     \n\n  method   | PickScore (vs StableSR) $\\uparrow$ |PickScore (vs SUPIR)$\\uparrow$| FID$\\downarrow$| FID_p$\\downarrow$| IS$\\uparrow$| IS_p$\\uparrow$| CLIP$\\uparrow$| Latency(sec.)$\\downarrow$|\n|---|---|---|---|---|---|---|---|---|\n| StableSR           | 31.3%    | - | 65.27| 48.18|27.55|9.25|32.49|728\n| SUPIR            | -    | 34.7%   |64.13|46.98|26.16|9.83|31.28|682\n| Ours | **68.7%**  | **65.3%**|**63.80**|**44.32**|**27.65**|**10.23**|**33.10**|**78**\n\n`Q5`: **Support even high resolution image generation**\n\nYes, we have further trained a model that supports up to 6K generation. Some visual results at resolutions of **5760x3840** and **3072x6144** are provided in **Figure 8** of the rebuttal PDF. We have included more visual results in the revised manuscript and will make the model publicly available.\n\n`Q6`: **The visual results demonstrating the limitations mentioned in the paper could be included in the supplementary materials to provide a more comprehensive understanding of the method's constraints**\n\nThank you for the constructive advice. We found that our model may occasionally fail in generating accurate structures in people's hands. Introducing more human hand data should help address this issue. We will include a more detailed analysis of both the strengths and weaknesses of our model in the revised paper.\n\n`Q7`: **Will the code and pre-trained models be made publicly available to facilitate reproducibility and further research in this area**\n\nYes, all models and code will be released to promote the development of the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to AC and reviewers (with PDF)**


We sincerely appreciate your time and efforts in reviewing our paper. We are glad to find that reviewers recognized the following merits of our work:
- **Innovative and effective solution [rwnV, z6xV, UnLF]**: The proposed UltraPixel introduces a Low-Resolution (LR) guidance feature to reduce the complexity of High-Resolution (HR) image generation. Additionally, it employs an implicit function and scale-aware normalization to assist the network in generating images of varying resolutions. This approach is both innovative and novel.
- **Impressive results [rwnV, z6xV, UnLF]**: UltraPixel generates ultra-high-resolution images with impressive quality and rich details, effectively addressing a major challenge in image synthesis.
- **Clarity and Readability [rwnV, z6xV, UnLF]**: Our paper is well-motivated, clearly articulated, and easy to read.

We also thank all reviewers for their insightful and constructive suggestions, which help further improve our paper. In addition to the pointwise responses below, we summarize the major revision in the rebuttal according to the reviewers’ suggestions

- **Comparative study on advanced T2I and diffusion-based super-resolution methods**: We have incorporated extensive quantitative and qualitative comparisons with leading commercial T2I products, DALLE 3 and MidJourney V6, as well as state-of-the-art SR methods, SUPIR and StableSR. These results further demonstrate UltraPixel's impressive ability to generate ultra-high-resolution images and its superior efficiency.
- **Enhanced visual analysis and method clarification**: We have added visual comparisons of different timesteps for extracting LR guidance, feature distribution across different resolutions, and an in-depth analysis of the proposed method.

Best,

Authors

Meta-review: The paper received mixed ratings of Strong accept, Weak accept, and a Weak reject. The major strength of the paper lies in its effective approach to high-resolution photorealistic image synthesis, impressive results as shown extensively in the paper, and a well-written paper. 

The negative reviewer has several concerns regarding the experimental validation. More specifically, missing comparisons with the state-of-the-art upsampler, baseline architecture (e.g., directly using LR image/latent), and ablation on LR guidance timesteps. 

The authors provide a rebuttal, including additional experimental results that address these concerns. However, the reviewer did not engage with the rebuttal. Two other reviewers are satisfied with the rebuttal and agree that the paper has sufficient merits. 

There is also an ethic review raising issues with self-collected image datasets. The authors' rebuttal clarified the source of the dataset (Unsplash Full Dataset), potential fairness issues and mitigation strategies, and potential negative societal impacts. 

Based on the reviews and the author's rebuttal, the AC believes that the concerns have been clarified and recommends to accept.
Decision: Accept (poster)