UltraPixel: Advancing Ultra High-Resolution Image Synthesis to New Peaks

Authors: jingjing  ren
wenbo  li
haoyu  chen
renjing  pei
bin  shao
yong  guo
long  peng
fenglong  song
lei  zhu
huawei  noah's ark lab
  mpi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 16:37:28

Abstract: Ultra-high-resolution image generation poses great challenges, such as increased semantic planning complexity and detail synthesis difficulties, alongside substantial training resource demands. We present UltraPixel, a novel architecture utilizing cascade diffusion models to generate high-quality images at multiple resolutions (\textit{e.g.}, 1K, 2K, and 4K) within a single model, while maintaining computational efficiency. UltraPixel leverages semantics-rich representations of lower-resolution images in a later denoising stage to guide the whole generation of highly detailed high-resolution images, significantly reducing complexity. Specifically, we introduce implicit neural representations for continuous upsampling and scale-aware normalization layers adaptable to various resolutions. Notably, both low- and high-resolution processes are performed in the most compact space, sharing the majority of parameters with less than 3$\%$ additional parameters for high-resolution outputs, largely enhancing training and inference efficiency. Our model achieves fast training with reduced data requirements, producing photo-realistic high-resolution images and demonstrating state-of-the-art performance in extensive experiments.

Main paper: 
 introduction: 
 Recent advancements in text-to-image (T2I) models, e.g., Imagen [42] , SDXL [37] , PixArt-α [4] , and Würstchen [36] , have demonstrated impressive capabilities in producing high-quality images, enriching a broad spectrum of applications. Concurrently, the demand for high-resolution images has surged due to advanced display technologies and the necessity for detailed visuals in professional fields like digital art. There is a great need for generating aesthetically pleasing images in ultra-high resolutions, such as 4K or 8K, in this domain.While popular T2I models [37, 4, 36] excel in generating images up to 1024 × 1024 resolution, they encounter great difficulties in scaling to higher resolutions. To address this, training-free methods have been proposed that modify the network structure [15, 22] or adjust the inference strategy [1, 12, 21] to produce higher-resolution images. However, these methods often suffer from instability, resulting in artifacts such as small object repetition, overly smooth content, or unreasonable details. Additionally, they frequently require long inference time [12, 14, 21] and manual parameter adjustments [15, 14, 12] for different resolutions, hindering their practical applications. Recent efforts have focused on training models specifically for high resolutions, such as ResAdapter [6] for 2048 × 2048 pixels and PixArt-Σ [4] for 2880 × 2880. Despite these improvements, the resolution and quality of generated images remain limited, with models optimized for specific resolutions only.Training models for ultra-high-resolution image generation presents significant challenges. These models must manage complex semantic planning and detail synthesis while handling increased computational loads and memory demands. Existing techniques, such as key-value compression [3] in attention [40, 11, 38, 39, 35] and fine-tuning a small number of parameters [6] , often yield suboptimal results and hinder scalability to higher resolutions. Thus, a computationally efficient method supporting high-quality detail generation is necessary. We meticulously review current T2I models and identify the cascade model [36] as particularly suitable for ultra-high-resolution image generation. Utilizing a cascaded decoding strategy that combines diffusion and variational autoencoder (VAE), this approach achieves a 42:1 compression ratio, enabling a more compact feature representation. Additionally, the cascade decoder can process features at various resolutions, as illustrated in Section A in the appendix. This capability inspires us to generate higher-resolution representations within its most compact space, thereby enhancing both training and inference efficiency. However, directly performing semantic planning and detail synthesis at larger scales remains challenging. Due to the distribution gap across different resolutions (i.e., scattered clusters in the t-SNE visualization in Figure 2 ), existing models struggle to produce visually pleasing and semantically coherent results. For example, they often result in overly dark images with unpleasant artifacts.In this paper, we introduce UltraPixel, a high-quality ultra-high-resolution image generation method. By incorporating semantics-rich representations of low-resolution images in the later stage as guidance, our model comprehends the global semantic layout from the beginning, effectively fusing text information and focusing on detail refinement. The process operates in a compact space, with low-and high-resolution generation sharing the majority of parameters and requiring less than 3% additional parameters for the high-resolution branch, ensuring high efficiency. Unlike conventional methods that necessitate separate parameters for different resolutions, our network accommodates varying resolutions and is highly resource-friendly. We achieve this by learning implicit neural representations to upscale low-resolution features, ensuring continuous guidance, and by developing scale-aware, learnable normalization layers to adapt to numerical differences across resolutions. Our model, trained on 1 million high-quality images of diverse sizes, demonstrates the capability to produce photo-realistic images at multiple resolutions (e.g., from 1K to 6K with varying aspect ratios) efficiently in both training and inference phases. The image quality of our method is comparable to leading closed-source T2I commercial products, such as Midjourney V6 [32] and DALL•E 3 [34] . Moreover, we demonstrate the application of ControlNet [52] and personalization techniques [20] built upon our model, showcasing substantial advancements in this field. 
related work: 
 Text-guided image synthesis. Recently, denoising diffusion probabilistic models [45, 18] have refreshed image synthesis. Prominent text-guided generation models [37, 4, 3, 36, 9, 35, 29, 46, 42, 27] have demonstrated a remarkable ability to generate high-quality images. A common approach is to map raw image pixels into a more compact latent space, in which a denoising network is trained to learn the inverse diffusion process [4, 3, 37] . The use of variational autoencoders [23] has proven to be highly efficient and is crucial for high-resolution image synthesis [13, 41] . StableCascade [36] advances this approach by learning a more compact latent space, achieving a compression ratio of 42:1 and significantly enhancing training and inference efficiency. We build our method on StableCascade primarily due to its extremely compact latent space, which allows for the efficient generation of high-resolution images.High-resolution image synthesis. Generating high-resolution images has become increasingly popular, yet most existing text-to-image (T2I) models struggle to generalize beyond their trained resolution. A straightforward approach is to generate an image at a base resolution and then upscale it using super-resolution methods [51, 10, 28, 48, 8] . However, this approach heavily depends on the quality of the initial low-resolution image and often fails to add sufficient details to produce high-quality high-resolution (HR) images. Researchers have proposed direct HR image generation as an alternative. Some training-free approaches [15, 12, 21, 1, 22, 53, 26] adjust inference strategies or network architectures for HR generation. For instance, patch-based diffusion [1, 26] employ a patchwise inference and fusion strategy, while ScaleCrafter [15] modifies the dilation rate of convolutional blocks in the diffusion UNet [37, 41] based on the target resolution. Another method [22] adapts attention entropy in the attention layer of the denoising network according to feature resolutions. Approaches like Demofusion [12] and FouriScale [21] design progressive generation strategies, with FouriScale further introducing a patch fusion strategy from a frequency perspective.Despite being training-free, these methods often produce higher-resolution images with noticeable artifacts, such as edge attenuation, repeated small objects, and semantic misalignment. To improve HR image quality, PixArt-sigma [3] and ResAdapter [6] fine-tune the base T2I model. However, their results are limited to 2880 × 2880 resolution and exhibit unsatisfied visual quality. Our method leverages the extremely compact latent space of StableCascade and introduces low-resolution (LR) semantic guidance for enhanced structure planning and detail synthesis. Consequently, our approach can generate images up to 6K resolution with high visual quality, overcoming the limitations of previous methods.  
method: 
 Generating ultra-high-resolution images necessitates complex semantic planning and detail synthesis. We leverage the cascade architecture [36] for its highly compact latent space to streamline this process, as illustrated in Figure 3 . Initially, we generate a low-resolution (LR) image and extract its inner features during synthesis as semantic and structural guidance for high-resolution (HR) generation. To enable our model to produce images at various resolutions, we learn implicit neural representations (INR) of LR and adapt them to different sizes continuously. With this guidance, the HR branch, aided by scale-aware normalization layers, generates multi-resolution latents. These latents then undergo a cascade diffusion and VAE decoding process, resulting in the final images. In Section 3.1, we detail the extraction and INR upscaling of LR guidance. Section 3.2 outlines strategies for fusing LR guidance and adapting our model to various resolutions. To address the challenges of high-resolution image synthesis, Previous studies [42, 17] have often employed a progressive strategy, initially generating a low-resolution image and then applying diffusion-based super-resolution techniques. Although this method improves image quality, the diffusion process in the pixel space remains resource-intensive. The cascade architecture [36] , achieving a 42:1 compression ratio, offers a more efficient approach to this problem.Guidance extraction. Instead of relying solely on the final low-resolution output, we introduce multi-level internal model representations of the low-resolution process to provide guidance. This strategy is inspired by evidence suggesting that representations within diffusion generative models encapsulate extensive semantic information [49, 2, 31] . EQUATIONwhere α t := 1 -β t , √ α t := t s=0 α s , and β t is the pre-defined variance schedule for the diffusion process. We then feed z L t to the denoising network and obtain multi-level features after the attention blocks, denoted as the guidance features g. Continuous upsampling. Note that the guidance features g are at the base resolution (24 × 24), while the HR features vary in size. To enhance our network's ability to utilize the guidance, we employ implicit neural representations [33, 5] , which allow us to upsample the guidance features to arbitrary resolutions. This approach also mitigates noise disturbance in the guidance features, ensuring effective utilization of their semantic content. As shown in Figure 4 , we initially perform dimensionality reduction on the LR guidance tokens via linear operartions for improved efficiency and concatenate them with a set of learnable tokens. These tokens undergo multiple self-attention layers, integrating information from the guidance features. Subsequently, the updated learnable tokens are processed through multiple linear layers to generate the implicit function weights. By inputting target position values into the implicit function, we obtain guidance features g ′ that matches the resolution of the HR features.The high-resolution latent generation is also conducted in the compact space (i.e., 96 × 96 latent for a 4096 × 4096 image with a ratio of 1:42), significantly enhancing computational efficiency. Additionally, the high-resolution branch shares most of its parameters with the low-resolution branch, resulting in only a minimal increase in additional parameters. In detail, to incorporate LR guidance, we integrate several fusion modules. Furthermore, we implement resolution-aware normalization layers to adapt our model to varying resolutions.Guidance fusion. After obtaining the guidance feature g ′ , we fuse it with the HR feature f as follows:EQUATIONThe fused HR feature f ′ is further modulated by the time embedding e t to determine the extent of LR guidance influence on the current synthesis step:EQUATIONWith such semantic guidance, our model gains an early understanding of the overall semantic structure, allowing it to fuse text information accordingly and generate finer details beyond the LR guidance, as illustrated in Figure 8 .Scale-aware normalization. As illustrated in Figure 2 , changes in feature resolution result in corresponding variations in model representations. Normalization layers trained at a base resolution struggle to adapt to higher resolutions, such as 4096 × 4096. To address this challenge, we propose resolution-aware normalization layers to enhance model adaptability. Specifically, we derive the scale embedding e s by calculating log N H N L , where N H denotes the number of pixels in the HR features (e.g., 96 × 96) and N L corresponds to the base resolution (24 × 24) . This embedding is then subjected to a multi-dimensional sinusoidal transformation, akin to the transform process used for time embedding. Finally, we modulate the HR feature f as follows:EQUATIONThe training objective of the generation process is defined as:EQUATIONwhere s and g denote scale and LR guidance, respectively. The parameters θ of the main generation network are fixed, while newly added parameters θ ′ including INR, guidance fusion, and scale-aware normalization are trainable. 
implementation details: 
 We train models on 1M images of varying resolutions and aspect ratios, ranging from 1024 to 4608, sourced from LAION-Aesthetics [44] , SAM [24] , and self-collected high-quality dataset. The training is conducted on 8 A100 GPUs with a batch size of 64. Using model weight initialization from 1024 × 1024 StableCascade [36] , our model requires only 15,000 iterations to achieve high-quality results. We employ the AdamW optimizer [30] with a learning rate of 0.0001. During training, we use continuous timesteps in [0, 1] as [36] , while LR guidance is consistently corrupted with noise at timestep t = 0.05. During inference, the generative model uses 20 sampling steps, and the diffusion decoding model uses 10 steps. We adopt DDIM [45] with a classifier-free guidance [19] weight of 4 for latent generation and 1.1 for diffusion decoding. Inference time is evaluated with a batch size of 1. Compared methods. We compare our method with competitive high-resolution image generation methods, categorized into training-free methods (ElasticDiffusion [14] , ScaleCrafter [15] , Fouriscale [21] , Demofusion [12] ) and training-based methods (Pixart-σ [3] , DALL•E 3 [34] , and Midjourney V6 [32] ). For models that can only generate 1024 × 1024 images, we use a representative image super-resolution method [51] for upsampling. We comprehensively evaluate the performance of our model at resolutions of 1024×1792, 2048×2048, 2160×3840, 4096×2048, and 4096×4096. For a fair comparison, we use the official implementations and parameter settings for all methods. Considering the slow inference time (tens of minutes to generate an ultra-high-resolution image) and the heavy computation of training-free methods, we compute all metrics using 1K images.Benchmark and evaluation. We collect 1,000 high-quality images with resolutions ranging from 1024 to 4096 for evaluation. We focus primarily on the perceptual-oriented PickScore [25] , which is trained on a large-scale user preference dataset to determine which image is better given an image pair with a text prompt, showing impressive alignment with human preference. Although FID [16] and Inception Score [43] (IS) may not fully assess the quality of generated images [25, 3] , we report these metrics following common practice. It is important to note that both FID and IS are calculated on down-sampled images with a resolution of 299 × 299, making them unsuitable for evaluating high-resolution image quality. Therefore, we adopt FID-patch and IS-patch for a more reasonable measure. Finally, we evaluate image-text consistency using the CLIP score [7] .Quantitative Comparison. As mentioned, PickScore aligns closely with human perception, so we use it as our primary metric. Figure 6 shows the win rate of our UltraPixel compared to other methods. Our approach consistently delivers superior results across all resolutions. Notably, UltraPixel is preferred in 85.2% and 84.0% of cases compared to the training-based Pixart-Σ [3] , despite Pixart-Σ using separate parameters for different resolutions and training on 33M images, whereas our model uses the same parameters for varying resolutions and is trained on just 1M images. UltraPixel also shows competitive performance compared to advanced T2I commercial product DALL•E 3 [34] , yielding a win rate of 70.0%. Continuous LR guidance enables our resolution-aware model to focus on detail synthesis, resulting in higher visual quality. Furthermore, as shown in Table 1 , our method performs competitively on FID, IS, and CLIP scores across different resolutions. Training-free HR generation methods [12, 15, 21, 14] struggle to produce high-quality 4096 × 2048 images, showing limited generalization ability. Our UltraPixel also excels in inference efficiency, generating a 2160×3840 image in 31 seconds, which is nearly 3.6× faster than Pixart-Σ (111 seconds). Compared to training-free methods that take tens of minutes to generate a 4096 × 4096 image, our model is significantly more efficient, being 9.3× faster than DemoFusion [12] . These results highlight the effectiveness of our method in generating ultra-high-resolution images with excellent efficiency.Qualitative comparison. Figure 7 illustrates a visual comparison between our UltraPixel and other high-resolution image synthesis methods at various resolutions. Training-free methods like ScaleCrafter [15] and FouriScale [21] often produce visually unpleasant structures and large areas of irregular textures, significantly degrading visual quality. DemoFusion [12] suffers from severe small object repetition due to its patch-by-patch generation approach. Compared to Pixart-Σ [3] , our method excels in generating superior semantic coherence and fine-grained details. For instance, in the 2160 × 3840 resolution case, our generated camel and human faces exhibit richer details. Despite using a single model to generate images at different resolutions, our method consistently produces visually pleasing and semantically coherent results. Besides, as illustrated in In this section, for computational efficiency, we train all models with 5K iterations. Unless otherwise stated, the results are reported at a resolution of 2560 × 2560. LR guidance. Figure 8 visually demonstrates the effectiveness of LR guidance. The synthesized HR result without LR guidance exhibits noticeable artifacts, with a messy overall structure and darker color tone. In contrast, the HR image generated with LR guidance is of higher quality, for instance, the characters "accepted" on the sweater and the details of the fluffy head are more distinct. Visualization of attention maps reveals that the HR image generation process with LR guidance shows clearer structures earlier. This indicates that LR guidance provides strong semantic priors for HR generation, allowing the model to focus more on detail refinement while maintaining better semantic coherence. Additionally, Figure 9 compares our method to the post-processing super-resolution strategy, demonstrating that UltraPixel can generate more visually pleasing details. Super-resolution has limited ability to refine the details of the low-resolution image, while our method is capable of generating attractive details.We analyze the effect of timesteps used to extract LR guidance in Table 2 and Figure 10 . We consider three cases: t = t H , where LR guidance is synchronized with the HR timesteps; t = 0.5, representing a fixed guidance at the middle timestep; and t = 0.05, near the end. The results show that t = t H produces a poor CLIP score. This can be attributed to the necessity of providing semantic structure guidance early on, but the LR guidance is too noisy at this stage to be useful. Similarly, t = 0.5 also results in noisy LR guidance, as seen in Figure 8 . Conversely, t = 0.05 provides the best performance since features in the later stage of generation exhibit much clearer structural information. With semantics-rich guidance, HR image generation can produce coherent structures and fine-grained details, yielding higher scores in Table 2 .To incorporate multi-resolution capability into our model, we adopt an INR design to continuously provide informative semantic guidance. In Table 3 , we compare continuous INR upsampling (dubbed "INR") with directly upsampling LR guidance using bilinear interpolation followed by convolutions (denoted as"BI + Conv"). The results show that INR yields better semantic alignment and image quality, as it provides consistent guidance of LR features across varying resolutions. Figure 11 further illustrates that directly upsampling LR guidance introduces significant noise into the HR generation process, resulting in degraded visual quality.Scale-aware normalization. As illustrated in Figure 2 , features across different resolutions vary significantly. To generate higher-quality results, we propose scale-aware normalization (SAN). Table 3 compares the performance of models with ("INR + SAN") and without ("INR") this design.When scaling the resolution from 2560 × 2560 to 4096 × 4096, the CLIP score gap noticeably enlarges, indicating better textual alignment with SAN. Additionally, the Inception Score shows significant improvement when adopting SAN, validating the effectiveness of our design. Number of trainable parameters. Our model benefits from high training efficiency, partly because we use a limited number of trainable parameters based on StableCascade [36] . Table 4 illustrates the impact of the number of trainable parameters. Since most new parameters are in the INR module, we can reduce the channel dimension of LR features from 2048 to a lower number. We explore models with LR dimensions of 512 and 1024 and also include a LoRA [20] version with a rank of 48. Compared to the "LoRA" model, "Ours-512" produces better results with fewer parameters.Increasing the channel number from 512 to 1024 ("Ours-1024") achieves higher visual quality and better text-image alignment. To balance efficiency and performance, we choose 1024 as the default. 
conclusion: 
 We present UltraPixel, an efficient framework for generating high-quality images at varying resolutions. Utilizing an extremely compact latent space, we introduce low-resolution (LR) guidance to simplify the complexity of semantic planning and detail synthesis. Specifically, semantics-rich LR features provide structural guidance for high-resolution image generation. To enable our model to handle varying resolutions, we learn an implicit function to consistently upsample LR features and insert scale-aware normalization layers to adapt feature distribution. UltraPixel efficiently generates stunning, ultra-high-resolution images of varying sizes, elevating image synthesis to new heights.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code repository link is provided in the home page https:// jingjingrenabc.github.io/ultrapixel/. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
broader impacts and limitation: 
 Despite the advancements in UltraPixel, the limited quantity and quality of training datasets constrain the realism and quality of our generated images, especially in complex scenes. This issue underscores the ongoing challenges in achieving true photorealism, and we are committed to further exploring this area in future research.Question: Does the paper specify all the training and details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Section 4.1Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Section 4Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines: Section 6 • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Section 6 Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: https://github.com/Stability-AI/StableCascadeGuidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: https://jingjingrenabc.github.io/ultrapixel/ Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]  
 
 Appendix: Comparison with an SR method. A common method to obtain high-resolution images involves initially generating a low-resolution image and then upsampling it with an off-the-shelf superresolution (SR) model. In Figure B .2 and B.3, we compare the results produced by our UltraPixel method and the advanced super-resolution techniques, BSRGAN [51] , StableSR [47] and SUPIR [50] .It is evident that the SR method often fails to introduce adequate details; although the resolution increases, the image quality does not improve proportionately. In contrast, our UltraPixel method excels by incorporating an abundance of intricate details, significantly enhancing the visual quality of the images. We present more visual results of UltraPixel in Spatial control. We present high-resolution (HR) results controlled by edge maps. Notably, we do not train our models directly; rather, we utilize the officially released control weights from StableCascade [36] . These control features are integrated during the low-resolution (LR) guidance extraction process. The results are demonstrated in Figures D.17 LoRA [20] with a rank of 4. The training involves an initial phase at a base resolution for 5,000 iterations, followed by fine-tuning at a higher resolution for an additional 5,000 iterations. Figure D.19 showcases our method's capability to incorporate personalized techniques for achieving personalized high-resolution image generation.Text prompts are provided in Table E .2, E.3. Figure Text Prompt Figure 1A close-up of a peony, with layers of soft, pink petals, a delicate fragrance, and dewdrops glistening in the early morning light.A serene mountain landscape with towering snow-capped peaks, a crystal-clear blue lake reflecting the mountains, dense pine forests, and a vibrant orange sunrise illuminating the sky. An expressionist landscape with twisted trees, a turbulent sky, and a path leading to an unknown destination, painted in vivid, unsettling colors and expressive brushstrokes that convey a sense of anxiety and movement.A close-up portrait of a young woman with flawless skin, vibrant red lipstick, and wavy brown hair, wearing a vintage floral dress and standing in front of a blooming garden.The image features a snow-covered mountain range with a large, snow-covered mountain in the background. The mountain is surrounded by a forest of trees, and the sky is filled with clouds. The scene is set during the winter season, with snow covering the ground and the trees.A statue of a person holding a torch A pair of cuddly rabbits, one white with floppy ears and the other brown with a twitching nose, snuggling together in a cozy hutch filled with straw. Idaho wedding party A man standing next to a camel A wooden wagon in a yard Crocodile in a sweater Figure 9 A vibrant anime scene of a young girl with long, flowing pink hair, big sparkling blue eyes, and a school uniform, standing under a cherry blossom tree with petals falling around her. The background shows a traditional Japanese school with cherry blossoms in full bloom.A playful Labrador retriever puppy with a shiny, golden coat, chasing a red ball in a spacious backyard, with green grass and a wooden fence. A cozy, rustic log cabin nestled in a snow-covered forest, with smoke rising from the stone chimney, warm lights glowing from the windows, and a path of footprints leading to the front door.A striking close-up of a young boy with curly blonde hair and bright green eyes, his face sprinkled with freckles. He is smiling widely, showcasing a gap-toothed grin, and the background is a sunny, out-of-focus playground. Smiling woman in white shirt A highly detailed, high-quality image of the Banff National Park in Canada. The turquoise waters of Lake Louise are surrounded by snow-capped mountains and dense pine forests. A wooden canoe is docked at the edge of the lake. The sky is a clear, bright blue, and the air is crisp and fresh.A highly detailed, high-quality image of a Shih Tzu receiving a bath in a home bathroom. The dog is standing in a tub, covered in suds, with a slightly wet and adorable look. The background includes bathroom fixtures, towels, and a clean, tiled floor. Ice Kingdom: A stunning ice kingdom with crystalline castles and frozen landscapes. The castles are made entirely of ice, with spires that sparkle in the sunlight. Snow-covered trees and icy pathways lead to the grand palace at the heart of the kingdom, where the ice queen resides. The sky above is a brilliant blue, with snowflakes gently falling.A detailed macro of a daisy, showcasing its white petals and bright yellow center, with tiny insects like bees and butterflies hovering nearby.A high-fashion runway show featuring models in avant-garde clothing, dramatic makeup, and elaborate hairstyles, with bright lights and a stylish, modern backdrop. Anthropomorphic profile of the white snow owl Crystal priestess, art deco painting, pretty and expressive eyes, ornate costume, mythical, ethereal, intricate, elaborate, hyperreralism, hyper detailed, 3D, 8K, Ultra Realistic, high octane, ultra resolution, amazing detail, perfection, In frame, photorealistic, cinematic lighting, visual clarity, shading, lumen reflections, super-resolution, gigapixel, color grading, retouch, enhanced, PBR, Blender, V-ray, procreate, zBrush. Art collection style and fashion shoot, in the style of made of glass, dark blue and light pink, paul rand, solarpunk, camille vivier, beth didonato hair, barbiecore, hyper-realistic A highly detailed, high-quality image of a Scottish Fold cat sitting on a bookshelf. The cat's distinctive folded ears and round face give it a unique appearance as it sits among books and decorative items. The background features a well-lit room with wooden shelves and a reading nook Traditional Breakfast: A hearty traditional breakfast with fluffy scrambled eggs, crispy bacon, golden hash browns, and buttered toast. The plate is garnished with fresh parsley and accompanied by a glass of freshly squeezed orange juice and a steaming cup of coffee. The background features a cozy kitchen setting with a morning sunbeam streaming through the window. Space Adventure: A thrilling scene of a spaceship navigating through an asteroid field. The ship is sleek and futuristic, with glowing thrusters and advanced weaponry. The asteroids are large and rugged, illuminated by the light of distant stars. In the background, a nebula in vibrant hues of blue and purple adds a touch of cosmic beauty to the scene. A medieval knight in shining armor, standing proudly in a lush, green field dotted with wildflowers, with a grand stone castle and rolling hills in the background. Several brightly colored rocks on a colorful beach, in the style of luminous spheres, emek golan, translucent color, 32k uhd, toyen, captivating • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper received mixed ratings of Strong accept, Weak accept, and a Weak reject. The major strength of the paper lies in its effective approach to high-resolution photorealistic image synthesis, impressive results as shown extensively in the paper, and a well-written paper. 

The negative reviewer has several concerns regarding the experimental validation. More specifically, missing comparisons with the state-of-the-art upsampler, baseline architecture (e.g., directly using LR image/latent), and ablation on LR guidance timesteps. 

The authors provide a rebuttal, including additional experimental results that address these concerns. However, the reviewer did not engage with the rebuttal. Two other reviewers are satisfied with the rebuttal and agree that the paper has sufficient merits. 

There is also an ethic review raising issues with self-collected image datasets. The authors' rebuttal clarified the source of the dataset (Unsplash Full Dataset), potential fairness issues and mitigation strategies, and potential negative societal impacts. 

Based on the reviews and the author's rebuttal, the AC believes that the concerns have been clarified and recommends to accept.

Reviews: review: paper_summary: paper_summary: This paper presents a method for Ultra-High-Resolution image generation from text prompts. The method is based on StableCascade. The original StableCascade can generate 1024x1024 images. This paper proposes another HR latent diffusion model that can utilize the guidance from 1024 x 1024 images and generate 4096 x 4096 images. Unlike previous methods that directly use the low-resolution output, the method chooses to use the features of the base model as guidance and proposes an implicit-based method to upsample the low-res guidance features.
strength_weakness: strength_weakness: - More validation and analysis are needed. In the comparison, a traditional image upsampler is used, but the traditional image upsampler is often smaller and also trained on much smaller datasets. For a fair comparison, it will be good to compare with the state-of-the-art generative image upsampler such as StableSR and Stable Diffusion Upscaler.
- A comparison of this baseline is missing: instead of using guidance features, the HR latent model can directly use the LR images / latents from the base model.
- It would be good to have visual results of the ablation on LR guidance timesteps.
- Ablation on scale-aware normalization is missing.
questions: questions: - Is the base model frozen from StableCascade?
- Is the implicit model jointly trained with the HR latent model?
limitations: limitations: It will be good to show some visual failure cases.
paper_summary: paper_summary: This paper introduces UltraPixel, a method for generating high-quality ultra-high-resolution images. It utilizes the semantics-rich representations of lower-resolution images in a later denoising stage to guide the overall generation of highly detailed high-resolution images. The method incorporates implicit neural representations for continuous up-sampling and scale-aware normalization layers that are adaptable to various resolutions. The experimental results show that it has excellent ability in generating high-resolution images of different sizes.
strength_weakness: strength_weakness: 1. The explanation of the implicit neural representation (INR) requires further clarity regarding its ability to enable continuous upscaling. Moreover, an in-depth analysis and dedicated ablation study of the Scale-Aware Normalization (SAN) feature would provide insights into its role in resolution adaptability.

2. To underscore the advantages of the proposed framework, the experiments should be expanded to include comparative analyses with Latent Diffusion Model (LDM)-based and pixel-based image synthesis methods, showcasing the superior performance of the framework in high-resolution image generation tasks.
questions: questions: 1. Why is the perturbed version $𝑧_{1}$ preferred over the $𝑧_{0}$ from the low-resolution synthesis for guidance purposes?

2. Regarding Figure 4, could you clarify why additional learnable tokens are integrated with the guidance tokens for the self-attention mechanism, instead of solely relying on the guidance tokens? What unique function do these learnable tokens serve?

3. Can you outline the computational steps involved in the implicit neural representation? Is there a need for manually specifying positions?

4. What justifies the forms of Equations (3) and (4), which amalgamate terms with distinct physical interpretations? Is there an underlying principle that supports their direct summation, as it seems to go against intuitive reasoning?

5. In the context of line 255, the use of 𝑡=0.5 and 𝑡=0.05 is ambiguous. Are these intended to denote specific sampling stages within the low-resolution synthesis—fixed and terminal steps, respectively? Consequently, is 𝑡=1 encompassed within the scenario where 𝑡=0.5?
limitations: limitations: 1. The paper may not sufficiently address how well the model generalizes to datasets beyond the training distribution. It is crucial to understand if the model's performance degrades with different or less common image content.

2. There is a need for more rigorous testing of the model's robustness to various corruptions and perturbations that could be encountered in real-world applications.
paper_summary: paper_summary: The paper presents UltraPixel, an innovative architecture for ultra-high-resolution image generation that tackles semantic planning, detail synthesis, and high resource demands. UltraPixel uses cascade diffusion models to generate images at multiple resolutions within a single model, efficiently guiding high-resolution generation with lower-resolution semantics. It features implicit neural representations for continuous upsampling and scale-aware normalization layers. Moreover, it requires less than a 3% increase for high-resolution outputs, boosting efficiency.
strength_weakness: strength_weakness: 1、 The manuscript's layout requires some refinement. For instance, Figure 4 extends beyond the page margins, and the text adjacent to Figure 9 appears overly condensed.

2、 Given that this is a text-to-image generation work, the paper would benefit from a more comprehensive set of visual results, including additional comparisons with state-of-the-art methods.
questions: questions: 1. The discussion in Section 4.3 regarding the timesteps of LR guidance extraction is intriguing. It would be valuable to see final generated images using different timesteps for guidance, rather than just attention visualizations.
2. The authors' use of LR guidance bears similarities to recent diffusion-based image super-resolution methods. A comparative discussion of these approaches could provide valuable context.
3. Given the method's design, it should theoretically support even higher resolutions (e.g., 5K, 6K). Have the authors explored this possibility?
4. The visual results demonstrating the limitations mentioned in the paper could be included in the supplementary materials to provide a more comprehensive understanding of the method's constraints.
5. Will the code and pre-trained models be made publicly available to facilitate reproducibility and further research in this area？
limitations: limitations: See Weaknesses.

Comments: title: Response to z6xV by Authors
Response to rwnV by Authors
Response to ethic reviewer
comment: We would like to thank you again for the valuable time you devoted to reviewing our paper. Since the end of discussion period is getting close and we **have not heard back from you yet**, we would appreciate it if you kindly let us know of any other concerns you may have, and if we can be of any further assistance in clarifying them.
Thank you once again for your contribution to our paper's refinement.
We would like to thank you again for the valuable time you devoted to reviewing our paper. Since the end of discussion period is getting close and we **have not heard back from you yet**, we would appreciate it if you kindly let us know of any other concerns you may have, and if we can be of any further assistance in clarifying them.
Thank you once again for your contribution to our paper's refinement.
**Q1 In line 176, the training data includes self-collected data. Clarifications are recommended regarding the copyright, privacy, and potential violent content within these datasets. Ensuring that all data collection adheres to relevant laws and ethical standards.**

We utilized data from the Unsplash Full Dataset 1.2.2, which is publicly available at https://unsplash.com/data. This dataset is provided by Unsplash with the intent of enabling researchers and developers to explore and uncover meaningful insights from the data. It is important to highlight that the creators of the Unsplash dataset have explicitly stated that no user-level data is included.

We formally requested and received approval from Unsplash to access their dataset for our research. To ensure quality, we filtered the data to include only images with a resolution of 1536x1536 pixels or higher. Unsplash's privacy policy ensures no personal information is collected or disclosed, and their robust licensing framework, including the Unsplash+ License, allows us to use the images legally and ethically without infringing on third-party rights. Given these comprehensive privacy practices and the legal protections afforded by the Unsplash+ License, we are confident that our use of the Unsplash dataset adheres to the highest ethical standards. We have taken all necessary precautions to protect privacy, comply with data protection regulations, and ensure that our research is conducted responsibly.

**Q2 When the self-collected image dataset includes images of humans, it is recommended to discuss the potential fairness issues. Specifically, the paper should address whether there are significant fairness concerns and if any mitigation strategies have been applied.**

Concerning fairness issues about human, we recognize that the inclusion of images of humans may raises potential biases, particularly related to gender, age, and cultural representation. For example, prompts like "policemen" often yield images of male officers, "a stylish woman with curly hair" typically generates images of very young women. 

To address these concerns, we plan to implement several mitigation strategies. Firstly, we will ensure that the dataset is diverse and representative across key demographic dimensions to reduce the risk of over- or under-representation of any group. Additionally, we will employ bias detection and evaluation techniques [1-5] during the model training process to identify and mitigate any emerging biases. These efforts are part of our ongoing commitment to developing a model that is as fair and inclusive as possible. 

[1] Unified detoxifying and debiasing in language generation via inference-time adaptive optimization

[2] Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you.

[3] The male ceo and the female assistant: Probing gender biases in text-to-image models through paired stereotype test

[4] Imagereward: Learning and evaluating human preferences for text-to-image generation.

[5] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models

**Q3 Potential Negative Social Impact: From the application perspective, there are concerns about biased outcomes resulting from text-to-image models due to imbalanced training data that reflects historical biases. The reduced availability of high-resolution images can exacerbate this issue, potentially leading to even more imbalanced training sets. It is recommended to include a discussion on the potentially more biased learning outcomes when adopting high-resolution text-to-image models and the potential mitigation methods.**

Thank you for raising this important concern. We acknowledge that the limited availability of high-resolution images might contribute to a more imbalanced training set. Such imbalances could lead models to perpetuate or even amplify existing biases, particularly regarding gender, race, and cultural representation. For instance, prompts about horses often generate images of brown-haired horses, and prompts about astronauts frequently depict space suits with a USA flag. 

To address these issues, we plan to implement several mitigation strategies. First, we will curate a more balanced and representative dataset by actively sourcing high-resolution images from diverse backgrounds, ensuring a wide range of demographic and cultural contexts. Second, we will incorporate bias detection and correction techniques [6-8] during the model training process to identify and address any biases that may arise from the use of high-resolution data.

[6] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models

[7] Diverse diffusion: Enhancing image diversity in text-to-image generation.

[8] Mitigating stereotypical biases in text to image generative systems
The author's rebuttal addresses my concerns. The explanation using a figure is clear. After reading the rebuttal and other reviewers' comments, I think this work is worth being accepted for subsequent researchers to follow up on their studies.