Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT
Authors: Le Zhuo
Ruoyi Du
Han Xiao
Yangguang Li
Dongyang Liu
Rongjie Huang
Wenze Liu
Xiangyang Zhu
Fu-Yun Wang
Zhanyu Ma
Xu Luo
Zehan Wang
Kaipeng Zhang
Lirui Zhao
Si Liu
Xiangyu Yue
Wanli Ouyang
Yu Qiao
Hongsheng Li
Peng Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative Models
Text-to-Image Generation
Diffusion Models
Flow Matching
Primary area: generative_models
TL;DR: Lumina-Next, an improved version of flow-based large diffusion transformers, enable better resolution extrapolation and multilingual text-to-image generation, while demonstrating strong performance across various modalities.

Abstract: Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers (Flag-DiT) that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions. Despite its promising capabilities, Lumina-T2X still encounters challenges including training instability, slow inference, and extrapolation artifacts. In this paper, we present Lumina-Next, an improved version of Lumina-T2X, showcasing stronger generation performance with increased training and inference efficiency. We begin with a comprehensive analysis of the Flag-DiT architecture and identify several suboptimal components, which we address by introducing the Next-DiT architecture with 3D RoPE and sandwich normalizations. To enable better resolution extrapolation, we thoroughly compare different context extrapolation methods applied to text-to-image generation with 3D RoPE, and propose Frequency- and Time-Aware Scaled RoPE tailored for diffusion transformers. Additionally, we introduce a sigmoid time discretization schedule for diffusion sampling, which achieves high-quality generation in 5-10 steps combined with higher-order ODE solvers. Thanks to these improvements, Lumina-Next not only improves the basic text-to-image generation but also demonstrates superior resolution extrapolation capabilities as well as multilingual generation using decoder-based LLMs as the text encoder, all in a zero-shot manner. To further validate Lumina-Next as a versatile generative framework, we instantiate it on diverse tasks including visual recognition, multi-views, audio, music, and point cloud generation, showcasing strong performance across these domains. By releasing all codes and model weights at https://github.com/Alpha-VLLM/Lumina-T2X, we aim to advance the development of next-generation generative AI capable of universal modeling.

Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
Le Z2
Ruoyi Du
Han Xiao
Yangguang Lil Dongvang Liu
Rongjie Huang"
Wenze Liu
Xiangyang Zhu'
Fu-Yun Wang*
Zhanyu Ma²
Xu Luo
Zehan Wang! Kaipeng Zhang' Lirui Zhao! Liu
Xiangyu Yue Wanli Ouyangl2 Yu Qiao't Hongsheng Li23t Peng Gao liv
"Shanghai AI Laboratory "The Chinese University of Hong Kong
*HKGAI under InnoHK "Beihang University
Beijing University of Posts and Telecommunications
Abstract
Lumina-12X nascent family of Flow-based Large Diffusion Transformers (Flag-DiT) that establishes unified framework for transforming noise into var- ious modalities, such as images and videos, conditioned on text instructions. De- spite its promising capabilities, Lumina-T2X still encounters challenges including training instability, slow inference, and extrapolation artifacts. In this paper, we present Lumina-Next Next, an improved version of Lumina- showcasing stronger generation performance with increased training and inference efficiency. We begin with comprehensive analysis of the Flag-Dil architecture and identify several suboptimal components, which we address by introducing the Next-DiT archi- tecture with 3D RoPE and sandwich normalizations. To enable better resolution extrapolation, we thoroughly compare different context extrapolation methods ap- plied to text-to-image generation with 3D RoPE, and propose Frequency- and Twme-Aware Scaled RoPE tailored for diffusion transformers. Additionally, we in- troduce a sigmoid time discretization schedule to reduce sampling steps solving the Flow ODE and the Context Drop method to merge redundant visual tokens for faster network evaluation, effectively boosting the overall sampling speed. Thanks to these improvements, Lumina-Next not only improves the quality and efficiency of basic text-to-ine text-to-image generation but also demonstrates superior resolution extrap- olation capabilities and multilingual generation using decoder-basee LLMs the text encoder, all in a zero-shot manner. To further validate Lumina-Ness as a versa- instantiate it diverse tasks including visual recog-
tile generative framework, nition, multi-view, audio, music, and point cloud generation, showcasing strong performance across these domains. By releasing all codes and model weights https: //gi thub. Aha- com/AlpphaVLLLMLLmmm VLLM-LLLLLLLLLLa T2X, we aim to advance the de- velopment of next-generation generative AI capable of universal modeling.
Introduction
Scaling diffusion transformers has unveiled significant improvements in text-conditional image and video generation [15, 14, 31, 35, 62]. Notably, Lumina-T2X [35] introduces flow-based large dif- fusion transformer (Flag-DiT), which has proven be stable, scalable, flexible, and training- efficient architecture for generative modeling across various data domains. For instance, the use
"Equal Contribution
"orresponning Authors
t Project Lead
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2048 2048
2048x208 2048
Resolution Extrapolation (2K); : A A swan glides gracefully across the surface of tranquil lake, its snowy white feathers ruffled by the gentle breeze.
Resolution Extrapelation (2K): Inka warrior with make up, medium shot, natural light, Award winning wildlife.
1024x4096
Resolution Extrapolation (Panorama): The majestic Eiffel Tower standing tall against the Parisian skyline dusk.
Multi-view: Eleggnt figurine a girl hair, wearing golden headpiece. Audio: dreany trippy instrumental jam with casygoing and mellow single electric guitar playing simple tune, accompunied guitur solo, ocho, and effect peda. Point Cloud; blue chair with long red chuir legs.
As foundational generative framework, we demonstrate Lumina-Next's capabilities to generate high-resolution images, multi-view images, general audio and music, and 16K point clouds.
Figure 1:
of advanced normalization techniques like KQ-Norm and RMSNorm enable stable mrixed-reeission training when scaling Lumina-T2X to 5B Flag-Di DiT with 7B LLaMA [80]. Besides, the flow matching framework and Rotary Position Embedding (RoPE) unlock the potential of Lumina-T2X to generate data with arbitrary resolutions, aspect ratios, and durations during sampling. While Lumina-T2X achieves superior visual aesthetics quality with remarkably low training resources, suffers from weak image-text alignment, slow inference, and extrapolation artifacts due inade- quate training, insufficient training data, and inappropriate context extension strategy.
To fully unleash the potential of scaling diffusion transformers for generative modeling, present next generation of Lumina-T2X, Lumina-Next with improved architecture, scaled dataset, op- timized sampling techniques, and better context extrapolation strategy, all of which make Lumina- Next stronger and faster. Specifically, the key improvements of Lumina-Nexs are listed as follows:
Architecture of Dext-DiT We revisit the architecture design of Flag-DiT and find suboptimal com- ponents for scalable generative modeling the visual domain. First, we replace the ID RoPE with 3D RoPE to eliminate the inappropriate positional priors to model images and videas using the at- tention mechanism. We further remove all learnable identifiers in Flag-DiT, such as [nextline] and [nextframe]] since our 3D RoPE already provides sufficient 3D positional information. Next, we task an in-depth analysis of the instability during both training and sampling and find it stems from the uncontrollable growth of network activations across layers. Therefore, we introduce the sandwich normalization block [25] in attention modules, which is proven to effectively control the activation magnitudes. Additionally, employ the Grouped-Qurey Attention [4] to reduce

---

putational demand especially when generating high-resolution images. The improved architecture of Next-DiT is validated on the ImageNet-255 benchmark, demonstrating a faster convergence rate compared to both Flag-DiTT and SiT [59].
Frequency- and Time-Aware Scaled RoPE Inspired by the recent progress of context extrapola- tion in LLMs [18, 1, 64], Lumina-T2X directly applies NTK Scaled RoPE to generate higher resolution images thanks to the design of ID RoPE. However, a comprehensive analysis of using 3D RoPE for length extrapolation in the vision domain is still lacking, especially tailored for diffusion and flow models. To bridge this gap, we first conduct a holistic comparison between existing extrap- olation methods applied to text-to-image Position Interpolation [18], and NTK- -ware Scaled RoPE [1]. We rethink the relationship between the encoding frequency in RoPE and the generated content and propose Frequency-Awar Scaled RoPE, significantly reducing content repetition during extrapolation. Furthermore, considering the difference between the auto-regressive generation of L.MSs and the time-aware generation process with fixed sequence length diffusion models, we propose a novel Time-Aware Scaled RoPE for diffusion transformers to generate high-resolution images with global consistency and local details.
diffusion transformers, including Position Extrapolation,
Optimized Time Schedule with Higher-Order Solvers In contrast various time sched- ules [61, 74, 42] and advanced SDE/ODE solvers [53, 54, 26, 42, 81] in diffusion models, most flow models [45, 50, 31] still adopts Euler's method with uniform time discretization during sampling. SiT [59] conducts a preliminary exploration of using different diffusion schedules and righer-order ODE solvers on the ImageNet benchmark. In this work, we demonstrate that existing diffusion schedules are not suitable for flow models and propose novel time schedules tailored for flow mod- els to minimize discretization errors. We further combine the optimized schedules with higher-order ODE solvers, achieving high-quality text-to-image generation samples within only 5 to 10 steps.
Time-Aware Context Drop To reduce the network evaluation time of each single step, we propose dynamically merging latent tokens to reduce redundancy in attention blocks. Different from the complex merge-and-unmerge algorithm in Token Merging [9], we employ a simple average pooling for keys and values to aggregate similar context tokens in spatial space. Additionally, we augment this Context Drop with time awareness to align with the dynamic sampling process of diffusion models. The resulting Time- Awne-Aware Context Drop is validated to effectively enhance inference speed while maintaining visual quality, achieving 2x inference-time speed up in 1K resolution image generation. When integrated with advanced attention inference techniques like Flash Attention [22], our method further improves inference speed of generating ultra-high resolution images.
By composing all improvements over Lumina-T2X. we reach Lumina-Next. Our Lumina-Next, featuring a 2B Next-DiT and Gemma-2B text encoder, achieves better text-to-image generation compared to Lumina-T22 MTX with a 5B Flag-DiT and LLaMA-7B [80] as text encoder, while signif- icantly reducing training and inference costs. Unlike previous models [67, 65, 31, 15. 14] that rely on CLIP [66] or T5 [20]. Lumina-Next demonstrates strong zero-shot multilingual ability with LLMs as text encoder. Moreover, equipped with improved architecture and the proposed inference techniques, Lumina-Nes
can achieve tuning-free 2K and few-step generation. We also show that Luxina-Next's framework can be easily extended to other modalities with excellent performance, demonstrating that Lumina-Next is unified, versatile, powerful, and efficient framework for genera- tive modeling. All codes and checkpoints of Lumina-Next are released. We hope the reproducibility of Lumina-Next can foster transparency and innovations in the generative AI community.
2 Improving Lumina-T2X
2.1 Architecture of Next-DiT
In Lumina-T2X [35], Flag-DiT serve as the core architecture with flow matching formulation, en- suring training stability and scalability for larger models. In this section, we propose Next-DiT, an improved version of Flag-DiT with the following key modifications. The comparison of detailed architectures of Dex and Fiag-DiT can be found in the Appendix B.I.
Replacing 1D RoPE and Identifiers with 3D RoPE Flag-DiT replaces absolute positional embed- ding (APE) with RoPE to enable flexibility of extrapolating to unseen resolution during training.
3

---

It further introduces learnable tokens includ- ing [nextline] and [nextframe] as identi- fiers to model images and videos with differ- ent aspect ratios and durations. We discov- ered that ID RoPE is a lossy representation to encode accurate the learnable identifiers introduce additional de- sign complexity, all of which can be simplified to 3D RoPE without losing any information.
positions and
(a)
(b)
Specifally, Lumina- T2X encode input from dif- ferent modalities into latent frames of shape for images, T =
Figure 2: Visualization of attention score using (a)
ID RoPE and (b) 2D RoPE on images. We set the central point in the image the anchor query.
[H. W, C1, where T numframes for videos, and T = numviews for multi-view images. Then, 1D RoPE is applied
to the flattened ID token sequences, which is formulated as follows given the m-th query and n-th key qm: hin E IR dress;
4m = f(m m) = 9m im
kn = f(in, = k e f
(I)
where θ Diagg(0 (01 ..., 0 is the frequency matrix. However, ID RoPE directly treats flattened frame tokens as ID language sequences, discarding the spatial and temporal relations between dif- ferent positions. As shown in Figure 2(a). when applying ID RoPE to the 2D images, the resulting long-term decaying property is obviously incorrect for the 2D scenario. Inspired by recent works that introduce 2D RoPE in the image domain [55, 32], it is natural extend the original ID RoPE to 3D RoPE for dent parts and calculate positional embedding for the z-axis, y-axis, and z-axis separately. Given then 3D coordinates of the m-th query and n-th key, the 3D RoPE is defined as:
modeling. We divide the embedding dimensions into three indepen-
im f(qms Im lem, Wm) = 9m [e Gr e en Or || @..
k, S'kin Ins bn; wn) ki, (e eh On cdw,,Ow,
where θ; = θ = o = Diag(f1..... ..., 0dywa/6) is the divided frequency matrix for each axis and denotes concatenating complex vectors of different axes at the last dimension. The attention scores with 3D RoPE are calculated by taking the real part of the standard Hermitian inner product:
Re(s(Im, li Nm, Wf) (km: bu hn, wa)]
= =Re(gm k (Be(mttt) |l e -hn) 9.
where Re(-] denotes the real part of complex numbers and indicates complex conjugates.
With the introduction of 3D RoPE. we provide a unified and accurate spatial-temporal representa- tion of positional encoding for different modalities, without the need to add extra Icarnable tokens such as [nextline] and [nextframe]
indicate the row, column, and frame index. Our 3D RoPE also incorporates the 2D RoPE formulation since it naturally becomes 2D RoPE when ap- plied to images with only 1 frame. Figure 2 shows the 2D RoPE assigns high attention scores to positions within the same row and column, which is a natural prior for images. Furthermore, our decoupled implementation of 3D RoPE on spatial and temporal axes better unlocks the potential for length extrapolation when combined with inference-time techniques mentioned in Section 2.2. We demonstrate the superior zero-shot extrapolation results with arbitrary resolutions Section 3.
Controling Magnititude of Network Activations To stabilize the training process when scaling up model size and token length, Lumin-T2X replaces all LayerNorm with RMSNorm and add QK- Norm before key-query dot product attention computation. Though effective to some extent, this recipe insufficient for stabilizing the training of extremely large diffusion transformers on long sequences. A closer examination of the Flag-DiT architecture reveals the presence of long signal paths without any normalization due to the residue structure of attention and mlp layers. We argue that these unnormalized paths appeared in many diffusion transformer architectures [63, 15, 59] can accumulate signal values across layers, causing uncontrollable growth of network activations, especially in deeper layers. We validate this hypothesis by visualizing the evolution of activation magnitudes over different depths of the text-to-image model using 500 random samples at different timesteps, as shown in Figure 3. What's 's worse, the uncontrollable network activations not only lead

---

DECOROAEE Carverr
Mear wo sordecct poom
20
Canmum
noJ
190
2080
Figure 3: Sandwich normaliza- tion effectively controls activa- tion magniuudes over layers.
Figure 4: Discretization errors and local curvatures grow at the start and end of sampling.
Figure 5: Next-DiT converges faster than SiT and Flag-DiT in terms of FII and IS.
to unstable training process but also deteriorate the sampling process. Particularly for resolution extrapolation, small errors in the early layers or sampling steps are amplified in subsequent layers steps. These compounding errors gradually cause extremely iitoof-distribution inputs for the network, leading to failure in generated samples.
Therefore, we propose simple yet effective approach that adds RMSNorm both before and after each attention and mlp layer. The modified architecture is equivalent to using the sandwich nor- malization [25] in transformer blocks. Importantly, both pre-norm and post-norm are placed before the scale operation in AdaLN-Zero to prevent normalizing an all-zero tensor at the start of training, which is caused by the zero-initialized scale parameter in AdaLN-Zero. However, after the scale operation, the activation magnitudes still cannot preserved. We further add a tanh gating to the second scale prediction after post-norm prevent extremely large modulation values from contribut- ing to the residual branch. With all these efforts including the original QK- QK-Norm in Flag-DiT, we effectively control both then input and output activation magnitudes of cach attention and mlp layer, as validated in Figure 3.
Experiments on ImageNet To quantitatively assess the effects of Next-DiT with the above im- provements, we conduct experiments on the label-conditional ImageNet-256 benchmark. We fol- low the training setups and evaluation protocols of SiT [59] and Flag-DiT [35]. As depicted in Figure 5, Next-DiT converges significantly faster than both Flag-DiT and SiT evaluated by FID and Inception Score (IS). This observation confirms that the improved design of Next-DiT can en- hance its fundamental generative capabilities. Additionally, we ablate the effectiveness of long-skip connections[6] designed to provide shorteuts for low-level features. Unfortunately, long-skip con- nections lead to training instability and significantly worsen the performance of Next-DiT. Therefore, Next-DiT chooses to not incorporate long-skip connection.
Improving NTK-Aware Scaled RoPE with Frequency- and Time-Awareness
In the large language models (LLMs) community, tuning-free or few-shot tuning length extrapola- tion is a popular focus currently and has been thoroughly analyzed, e.g., position interpolation [18]. NTK-Aware Scaled RoPE [1]. YaRN [64], etc. In Lumina-T2X, due to the adoption of ID RoPE same LLMs, NWK-Aware Scaled RoPE has been directly applied to achieve certain degree of resolution extrapolation. In Lumina-Next, we further investigate the effects different length ex- trapolation methods 3D RoPE for the first time. Based and time-awarene awarenesss grounded in NTK-Aware Scaled RoPE, resulting in novel Frequency- Scaled RoPE and Time-Awart Scaled RoPE targeted the visual generation task. In Figure 6 (a), we demonstrate the wavelength of each dimension of the RoPE embedding under different extrapo- lation techniques with toy setting of /head 24, and T H W 16.
findings. introduce frequency-
Revisiting NTK-Aware Scaled RoPE 3D RoPE encodes position information of each axis using A with 0g b 6-6/// where b is the rotary base. When we want to perform a times resolution extrapolation, the most straightforward way is encoding unseen positions with no change of RoPE, known as Position Extrapolation. How- ever, these unseen positions will directly confuse the model's spatial understanding, leading to the generation of unreasonable repetitive content (refer to Figure 6 (c)).
frequency matrix θ Diag(#; ........ Bu....
5

---

Extocittoon Emquene
wrop
Devecrio haiee (o Wesokogth of the eabeddiigs
du Origggaal Recattion (i Posliiss Exrapelato (4) Proiiia Ixcerplation
Priqanscy saar
1g Tiam ARaR
Figure 6: (a) Toy illustration of RoPE embeddings' wavelength with different extrapolation strate- gies. (b)-(g) Results of different resolution extrapolation strategies of 2K generation.
In contrast, another straightforward approach, Position Interpolation [18], involves linearly rescal- ing the frequency matrix as Of = 02 8, thereby bringing all position embeddings back into the training value domain. However, noted by [1], this linear interpolation makes it difficult for RoPE to distinguish between positions of tokens that are very close to each other. The same conclusion exists in the diffusion transformer maintained at the cost of losing all local details.
shown in Figure 6 (d), the global structure of the image is
To address these issues, NTK-Aware Scaled RoPE [1] is proposed to achieve training-free length extrapolation for LLMs. Particularly, scales the rotary base 1/ b 8 and then calculates Bmn 6d such that the lowest frequency term of RoPE is equivalent performing position interpolation, allowing gradual transition from position extrapolation of high-frequency terms to position interpolation low-frequency ones. In Lumina- T2X [35]. NTK-Aware Scaled RoPE has been shown to seamlessly integrate with diffusion transformers achieve resolution extrapolation to a certain extent. However, failure cases still occur, such the repetition issue shown in Figure 6 (e), indicating that the model still struggles with spatial awareness during resolution extrapolation.
Frequency-Aware Scaled RoPE Rethinking the limitations of NTK-Aware Scaled RoPE, we be- lieve that performing interpolation on the lowest frequency component is a suboptimal design. Con- sidering that RoPE uses periodic functions with frequencies @ to encode relative positions, many dimensions encounter unseen repeated cycles during resolution extrapolation, which in turn causes the model to output repetitive content. This echoes the statement of YaRN [64] that position embed- ding should be divided into parts using wavelength as the boundary and treated differently. To solve this problem, we first identify the dimension with the wavelength equivalent to the training sequence length as durrer = dhead log, 2=) where L is the maximum training sequence length. After that, a
ducal
which enable the ponent at duarget equivalent to position interpolation. Note that dearget may not be an integer, but we do need use as an index so that we can use its exact value. Additionally, to prevent the part where d > durrgtt from being excessively interpolated, we take the larger value comparing with the position interpolation result as ofieg max(b'-Ga 6d/dass b-6d/dacs s). This Frequency-Awar Scaled RoPE further addresses the issue of content repetition, shown in Figure 6 (f).
scaled base with frequeney-awareenss can be formulated by b' = b Fouget
Time-aware Scaled RoPE Observing Figure 6, we can conclude that: (1) while position inter- polation confuses local details, it effectively ensures reasonable global structure, and (2) while NTK-Aware Scaled RoPE has content repetition, it retains good local details. By combining these observations with the time-conditioned feature of diffusion models, we can achieve a more targeted resolution extrapolation approach. A common understanding of diffusion models is that they first restore global concepts before local details [19]. This motivates
design time-aware strategy to benefit from different approaches at different denoising steps using position interpolation in the early stages of denoising to ensure the overall structure and gradually shifting to NTK-Aware Scaled RoPE to preserve local details. This echoes DemoFusion's 's [29] strategy of gradually re- ducing global guidance over the denoising process, allowing the model to focus on adding local details. Specifically, the previously proposed Frequency-Awan Scaled RoPE is naturally adapt- able to this progressive shift. When durget = 1, equivalent to position interpolation; when draget = dhead: it is equivalent to NTK-Aware Scaled RoPE. Therefore, we design coefficient de

---

Steps
Stops
Stops
Stepa
Steps
30 Stops
Stego
Steps
Sqnid
Rational
Uniffrr
Figure 7: Comparison of ftw-step generation using different time schedules with Euler's method.
that varies with time in the most direct way, making the frequency base time-conditioned, i.e. b', = b where d, = (dhead - 1) + 1. Then the time-aware frequency base can expressed as Odme max((/ 6d/ddas bd s). This Time-Aware Scaled RoPE makes globally consistent high-detail resolution extrapolation possible, as shown in Figure 6 (g). In the following experiments on resolution extrapolation, we use Time-Aware Scaled RoPE for the best ouning-free performance.
2.3 Improving Sampling Efficiency with Fewer and Faster Steps
Different from one-step generative models, the sampling of diffusion and flow models involves an iterative denoising process. The overall time complexity of sampling can be written as N xT, which controlled by the number of function evaluations N and the inference time of single function evaluation T. Therefore, to improve the efficiency during inference, we propose corresponding strategies for each factor to reduce N and T respectively.
Fewer Sampling Steps with Optimized Time Schedule and Higher-Order Solvers Both dif- fusion models and flow-based models numerically solve SDE ODE within the interval ['miny bmaz] during sampling. A common approach is to discretize the continuous time interval into multiple sub-intervals and Buler's method with O(h²) discretization error with respect to step size ht or other advanced solvers. Given a constant budget of N sampling steps, the time dis- cretization strategy {to, tto.tt..... tx.. or time schedule, determines the grid of timesteps on which the network is evaluated. The resulting accumulated truncation eтrоrs can greatly impact the quality of generated samples, especially when N is small. Various time schedules have been explored for dif- fusion models, such as Cosine [61], Linear [74], and EDM [42]. However, most flow- how-based models still adopt the simplest uniform time intervals and first-order Euler's method for sampling, resulting in poor-quality images in low- scenarios, as shown in Figure 7.
Therefore, we provide detailed analysis of truncation errors different timesteps using Euler's method to solve the Flow ODE for text-to-image generation. We begin by setting the anchor sam- pling steps, N = 50, with uniform intervals for evaluation. For each timestep 4i the sample pro- duced by one-step Fuler's method is defined as x,, = X + + 4)) (X,,, 4). To approximate the local truncation error Tr, at time &,, we estimate the ground truth sample X by running large number of Euler steps from t; to t++1 with uniform intervals and compute the discretization error as Tr, Xt,41 _ In addition, since the flow model builds linear interpolation between noise and data and predicts the constant velocity field, it is natural to use the curvature to describe local errors on the trajectory, i.e., the straighter the trajectory, the smaller the discretization error. Fol- lowing [60], we compute the second-order derivative to estimate the local curvature of each step as Ki; Xt; (xt + Xt; .) x 0.5. The average 1.2-norm of discretization error тı, and curvature Ki; are visualized in Figure 4. Our analysis reveals that discretization errors are much larger when I near pure noise (1 = 0) and are relatively larger near clean data (L = 1), compared to intermediate timesteps. This observation contradicts the findings in diffusion models, where EDM [42] discover
7

---

the discretization errors monotonically increase as samples get closer to clean data. Hence, diffusion schedules like EDM are not ideal for flow-based models.
To optimize the time schedules for sampling, we propose two parametric functions to map & E [0, 1]:
1 SIGMOID t' 1+exp(-a((11)
ift < P if 2
RATIONAL
t'= 0 a-t+ot
(4)
[texp(B(1-H)
These time schedules are tailored for flow-based models. The first, a rational function parametrized by o, maps all timesteps to smaller values, minimizing the discretization emrors near pure noise. This schedule reduces to the uniform time schedule when o and is also used in prior works [35, 31, 38] for high-resolution image generation. The second schedule, a piecewise sigmoid-like function, ensures that the step sizes at the start and end of sampling are larger than those at intermediate steps. The central point is controlled by 14, while the weights on lower and higher timesteps are adjusted by a and 3, respectively. As shown in Figure 7, both rational and sigmoid schedules improve few-step sampling performance compared to all-black or all-white images generated by the uniform schedule. This highlights the importance of the early stage of sampling in creating the main subject in the image. The sigmoid schedule generates images with better details and is adopted for the remainder of this paper with μ = 0.6, a = 6, and / = 20.
Notably, this optimized schedule requires no extra computation but free launch for fast sampling using our flow model. It also compatible with advanced solvers applicable to flow models, which can unleash its full potential. However, limited work has explored the use of higher-order solvers for solving the Flow ODE. Unlike the Probability Flow ODE [75] in diffusion models, which has a semi-linear structure, the Flow ODE derived from the continuity equation is in a simpler form of x = vo(x.,), t). This simplicity allows for the direct use of existing higher-order ODE solvers, such as the traditional explicit Runge- Kutta (RK) methods in the form of x, = x. + S va (x,, T)dT, where the n-th order solver ensembles n intermediate steps between [s, t] to reduce the approximation errors. During sampling, explicit RK family solvers also need to specify the timesteps {4a, 41,..,,/x, } in advance and adopt the uniform intervals as default. Thus, our improved schedules, such as the rational sigmoid schedule, can be directly applied to these higher-order solvers. Algorithm B.1 illustrates the pseudocode for combining the midpoint method and sigmoid schedule for sampling.
Faster Network Evaluation with Time-Aware Context Drop After reducing the denoising steps during sampling, our next goal is to optimize the network evaluation time T for each single step, which constrained by the quadratic complexity of attention blocks. Among various inference- time acceleration methods, Token Merging (ToMe) [9] is a simple yet effective approach that merges similarly visual tokens to reduce redundancy in attention blocks, thereby speeding up inference for image classification. Although ToMe has been successfully applied to diffusion U-Net [10, 73], we find that a straightforward application of ToMe Next-DiT, purely transformer- basedd diffusion model, completely fails to produce satisfactory images. We hypothesize that performing the original token merging in all transformer layers across all diffusion steps is clearly unsuitable for the task of generative modeling with diffusion transformers.
To bridge this gap. we propose Time-Aware Context Drop, an optimized method to drop redun- dant visual tokens during inference for Next-DiT. First, the original bipartite soft matching in ToMe not only introduces additional complexity but also contradicts the design of 3D RoPE in Next-DiT. Therefore, we replace this complex token partitioning strategy in ToMe with simple average pool- ing for downsampling. aligning with the spatial prior introduced in 3D RoPE. Similar to [73], we find that applying the merge-and-ug und-uerr-- operation to all queries, keys, and values greatly under- mines the quality of generated images as this strategy designed for visual recognition. To preserve the complete visual content, we only perform average pooling on keys and values for downsampling. Finally, considering the time-awareness in diffusion models, we perform token dropping for all lay- ers at / 0 to enhance efficiency and gradually transform to no token dropping at i. = 1 to maintain visual quality. The Context Drop without time-awareness is similar to the key-value (KV) token compression in PixArt-- [14], which merges KV tokens in spatial space using 2 x 2 convolution to reduce complexity. However, unlike their fine-tuning methods, Context Drop is a training-free technique with adaptive compression rate that aligns with the diffusion time schedule. In Ap- pendix C. We demonstrate samples generated by our methods, which reduce inference time while achieving comparable visual quality.
8

---

Lumina-Next (IK)
SDXL
PixArt-a
MuldiDiftusion
DemoF usion
ScaleCrafter
Lumina-121
Lumina-Next
Figure 8: Comparison of 4x resolution extrapolation.
Lumina-Next by Composing Everything
By composing all improvements, we finally reached Lumina-Next, a powerful generative framework for various modalities with improved efficiency. We first instantiate Lumina-Next for text-to-image generation, which stronger and faster than Lumina-T2X. Besides, compared with SDXL and PixArt-a, it demonstrates preliminary supports of multilingual prompts (refer to Appendix C.2), better visual quality with smaller sampling steps, and more flexibility at synthesizing images with any resolution. After this, we further verified that the Lumina-Next framework also exhibits excel- lent performance in any resolution recognition (refer to Appendix D) and multi-vien images, music, audio, point clouds generation (refer to Appendix E).
3.1 Flexible Resolution Extrapolation
Failure of SDXL and PixArt-a SDXL [65] and PixArt-a [15] are representative methods of U-Net and transformer-based diffusion models, respectively. Despite their ability to generate im- pressive images, they fail to produce images beyond the training resolution, resulting in repetitive and unreasonable content higher resolutions, shown in Figure 8. This is because the receptive field of U-Net and the absolute position embedding of transformers are significantly biased towards the training resolution or sequence length.
Comparison with Tuning-free Resolution Extrapolation Methods Due to the enormous cost of directly training high-resolution (greater than 2K) generative models, researchers have sought to enable pre-trained diffusion models to infer beyond the training resolution in tuning-free manner. MultiDiffusion [7] the first attempt to generate high-resolution panoramas via sampling over- lapped sliding windows the original resolution. DemoFusion [29] further utilizes the generated

---

Steps
Steps
Steps
Stepa
Steps
Steps
Steps
Steps
SONL +DPM-Swher
PIAAro -DPhi-Suker
Lumine Nest +hiddpois Saip Siler
Figure 9: Comparison of few-step generation using second-orde ODE solvers.
results at training resolution as global guidance, achieving object-oriented high-resolution image generation. Meanwhile, ScaleCrafter [37] adopts dilated convolution to achieve resolution extrapo- lation tiled for U-Net based diffusion models (e.g., SDXL [65]).
In Figure 8, compare Lumina-Nex with the methods above on the 2K generation task. We can conclude that MultiDiffusion, proposed for panorama generation, lacks global awareness, leading to the issue of repetitive content still being present. DemoFusion and ScaleCrafter work well, but DemoFusion requires multiple runs, resulting in a non-negligible inference budget, and ScaleCrafter built upon dilated convolution, which is unsuitable for modern transformer- els. In contrast, Lumina-Next inherently supports a direct resolution extrapolation with the carefully designed Next-DiT architecture and frequency- and time-aware scaled RoPE inference time, of- fering more fundamental solution in the era of diffusion transformer.
diffusion mod-
On the Importance of Next-DiT for Resolution Extrapolation Regarding Figure 8, we partic- ularly emphasize that Lumina-Ner Next's s 2K extrapolation performance significantly surpasses that of Lumina-T22 In addition to the newly proposed time-aware scaled RoPE, we attribute this to the ad- vanced design of Next-DiT: (1) 3D RoPE enables independent modelling of the f, h, dimensions, decoupling extrapolation three dimensions so that they do not affect each other, and (2) sandwich norm stabilizes the output magnitude of each transformer block, reducing the model's sensitivity to sequence length. Composing them together, Lumina-Nest possesses excellent resolution extrapola- tion capabilities without the need for any special inference algorithms like DemoFusion [29].
3.2 Few-Step Text-to-Image Generation
We demonstrate how our improved sampling schedules together with higher-order ODE solvers can significantly boost the sampling efficiency for flow models. Specifically, we employ the midpoint solver, second-order Runge-Kutt Kutta method, combined with our sigmoid schedule following Algo- rithm B.1. For comparison, we choose open-source text-to-image models, including SDXL [65] and PixArt-o [15] equipped with DPM-Solver [53, 54]. Figure 9 shows Lumina-Next using the midpoint solver significantly improves the sample quality in 10-20 number of function evaluations (NFEs), consistently achieving better conditional sampling performance compared to Pix Art-cand SDXL using DPM-Solver. Switching to higher-order solvers can further enhance few-step sampling performance with the cost of linearly increasing NFEs, presenting speed-quality trade-off.
Conclusion
In this paper, we introduce Lumina-Next, which successfully addressed the limitations of its pre- decessor, Lumina-T2X. With the improved Next-DiT architecture, context extrapolation together, and fast sampling techniques tailored for our flow-based diffusion transformers, Lumina-Nes show- cases strong generation capabilities, such as generating high-quality images significantly larger than its training resolution and multilingual text-to-image generation. Remarkably, these results are all produced in a training-free free manner and outperform previous methods including SDXL and Pix Art-a... We extend I aumin--Nex'ss 's versatility to other modalities, such as multiviews, audio, music, and point clouds, with minimal modifications, achieving superior results across these diverse applications.
10

---

Acknowledgements
This project funded in part by the National Key R&D Program of China Project 2022ZD0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Tech- nology Commission (ITC)s InnoHK, by the General Research Fund of Hong Kong RGC Project 14204021, by the National Natural Science Foundation of China (Grant No. 62306261), and by Shun Hing Institute of Advanced Engineering (SHIAE) No. 15044 Hongsheng Li a of HKGAI under the InnoHK.

---

References
[1] Ntk-aware Scaled Rope Allows Llama Models to Have Extended (8k+) Con-
text Size Without Any Fine-tuning and Minimal Perplexity Degradation. https:
//www.rewe reddit. wtt.c////////aaaaeee allows 11amaaaaaaaa
comments/ /aattt//1177///tkkaaa scwlsddaaorrpeeee lww.aaameealaaaaaaaeeaaaaaaaaaaa models to have/, 2024. Accessed: 2024-4-10.
[2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas, Learning repre- sentations and generative models for 3d point clouds. In International Conference on Machine Learning, pages 40-49. PMLR, 2018.
[3] Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.113 2023.
[4] Joshua Ainslie, James Lee-Thare Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference Empirical Methods in Natural Lan- guage Processing, pages 4895-4901, 2023.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[6] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. Proceedings of the IEEE/CVF Conference on 2023.
Computer Vision and Pattern Recognition, pages
[7] Omer Bar,Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference Machine Learning, pages 1737-1752, 2023.
[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle: pdf. 2(3):8, 2023.
[9] Daniel Bolya, Sheng--Yng Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2022.
[10] Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4598-4602, 2023.
[11] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. arXiv preprint arXiv:2008.0652 2020.
[12] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report, arXiv preprint 2xXiv:2403.1729) 2024.
[13] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information -ichh 3D Model Repository. Technical Report 1rXiv:1512.03012 [cs.GR], 2015.
[14] Junsong Chen, Chongjian Ge, Enze Xie. Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion trans- former for 4k text-to-image generation. arXiv preprint arXiv:2403. 04699, 2024.
[15] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lui, al. Pixart-a: Fast training of diffusion transformer for photorealistic ext-to-image synthesis. arXiv preprint arXiv:2310.00426. 2023.
12

---

[16] Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Musicldm: Enhancing novelty in text-to-music generation using bynh-synnssonous mixup strategies. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 22061210. IEFF, 2024.
[17] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12999 2023.
[18] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context win- dow of large language models via positional interpolation. arXiv preprint MxXv:2306.15599 2023.
[19] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481, 2022.
[20] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and others. Scaling instruction- finetuned language models. arXiv preprint arXiv:2210.1 1416, 2022.
[21] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défosse7. Simple and controllable music generation, 2023.
[22] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691/ 2023.
[23] Mostafa Dehghami, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, al. Patch npack: Navit, vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024.
[24] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Lud- wig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023.
[25] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural information Processing Systems, 34: 144419922219999 19835, 2021.
[26] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150-30166 2022.
[27] SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.1637,, 2023.
[28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter- national Conference on Learning Representations, 2020.
[29] Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofu- sion: Democratising high-resolution image generation with no S$S. In CVPR, 2024.
[30] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learn ing audio concepts from natural language supervision. 2022.
[31] Patrick Esser, Sumith Kulal, Andreas Blattaann Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint ArXiv:2403. 032060 2024.
[32] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02:
A visual representation for neon genesis. arXiv preprint arXiv: 11331, 2023.
13

---

[33] Seth Forsgren and Hayk Martiros. Riffusion-stable diffusion for real-time music generation.
URL httpss//riffusion. com, 2022.
[34] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Sealing data and parameters for family of multi-modal large language models. arXiv preprint 1rXiv:2402.05995 2024.
[35] Peng Gao, Le Zhuo, Ziyi Lin, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint (eXXv:2405.05000 2024.
[36] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to- audio generation using instruction-tune -unte Ilm and latent diffusion model. arXiv preprint arXiv:2304. 13731, 2023.
[37] Yingqing He, Shaoshu Yang. Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang. Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[38] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213- 13232. PMLR, 2023.
[39] Jiawei Huang, Yi Ren, Rongjie Huang. Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio cemporal-enhaneee eet--to-addio generation. arXiv preprint 23X.v:2305..14777 2023.
[40] Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training- free high-resolution image synthesis. arXiv preprint arXiv:2403. 22663, 2024.
[41] Rongjie Huang, Jiawei Huang, Donghaa Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023.
[42] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 2022.
[43] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually Guided Audio Generation.
[44] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
[45] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint
2022.
[46] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mci, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Texiioo-Audio Generation with Latent Diffusion Models.
[47] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with sulf-supe pretraining. arXiv preprint arXiv:2308.055: 057344 2023.
[48] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Ad- vances in neural information processing systems, 36, 2024.
[49] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-t-3: Zero-shot one image to 3d object. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 9298-9309, 2023.
14

---

[50] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 03003, 2022.
[51] Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xi- aoshui Huang. Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for re- lightable text-to-3d generation. arXiv preprint arXiv:2312.08754. 2023.
[52] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingje Liu, Yuexin Ma. Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 5500,, 2023.
[53] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver:
A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[54] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint (tt::2211.011993. 2022.
[55] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402. 12376, 2024.
[56) Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[57] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07277 07279, 2023.
[58] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024.
[59] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion- based generative models with scalable inter- polant transformers. arXiv preprint arXiv:2401.0874 2024.
[60] Bao Nguyen, Binh Nguyen, and Viet Anh Nguyen. Bellman optimal step-size straightening of flow-matching models. arXiv preprint arXiv:2312.1664 1641, 2023.
[61] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[62] OpenAI. https://ppenai.com/srr. In OpenAl blog, 2024.
[63] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4195 2023.
[64] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.000/ 000.900071.2220 2023.
[65] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[67] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photore- alistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[68] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Text-to-music generation with long-context latent diffusion, 2023.
15

---

[69] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25221 2022.
[70] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 1croo23++: single image consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110, 2023.
[71] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308. 165/2, 2023.
[72] Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE International Conference on Computer Vision, pages 3859-388800 2359, 2019.
[73] Ethan Smith, Nayan Saxena, and Aninda Saha. Todo: Token downsampling for efficient gen- eration of high-resolution images. arXiv preprint arXiv:2402.13572 2024.
[74] Jiaming Song, Chenlin Meng, and Stefano Ernon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[75] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[76] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. Advances in Neural Infor- mation Processing Systems, 33:7537-7547, 2020.
[77] Paul Taylor. Text-to-speech synthesis. Cambridge university press, 2009.
[78] Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22721-22731 2023.
[79] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[80] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139711 2023.
[81] Fu-Yun Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased con- sistency model. arXiv preprint arXiv:2405.18407, 2024.
[82] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d genera- tion. arXiv preprini arXiv:23/2.02200 2023.
[83] Zhengyi Wang. Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongx- uan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint 4rXiv:2403.05034, 2024.
[84] Dongchao Yang, Jianwei Yu, Helin Wang. Wen Wang. Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete Diffusion Model for Text-to-sound Generation.
[85] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In IEEE International Conference on Computer Vision, pages 4541 2019.
16

---

Novey input
Novey ipput
hext lnc toler
INI
Loyanlore
AMd Nore
Parch brbwal
(aachh
NX
NX
Erboudding
Erbecdie
Cribiddding
soole & shitt
Lubel Enterdinng
Dog
Upgrade From Flag-bil Next-DiT
Rethooer CC Soj
Fee Cooefforent
Reeeforwerd
Preashertees
Prideted
Fieiicee Novd
Fackk
Read ctei
Figure 10: Architecture details of DiT and Next-DiT.
A Broader Impacts and Limitations
A.1 Broader Impacts
In this work, we propose a fundamental framework for generative modeling. On the positive side, the open-source of our code and checkpoints will foster transparency well as innovations in the generative AI field. On the negative side, our text-to-image generation model, for example, may be used to generate images with potential biases and malicious information, similar to any generative AI tool. However, we have designed and used a filtering pipeline to filter out all malicious content in the training dataset.
A.2 Limiations
In this work, we demonstrate the superior high-resolution and multilingual text-to-image generation capabilities Lumina-Next compared to open-source models like SDXL and PixArt-a. However, our cext-to-image generation model still fall short of ofte-oo-the-an closed-source models like Mid- journey and DALLE 3 in terms of text-image alignment and visual aestheties. notable gap lies in the size of text-image pairs used for multi-stage training. Although we expand the size of our training dataset to 20 million, remains considerably smaller than the datsets used by these closed- source models. In addition, fine-tuning models with human preference data using techniques like Direct Preference Optimization also important to improve image quality. We plan address these challenges in future research.
17

---

Algorithm I Few-step sampling using midpoint method and optimized schedule procedure MIDPOINTSANPLERRV/XX// sample Xo ~ (),
(N,
for {0,-.., 13 do
At 4,+1 -t;
Adaptive step size
d, t vo((;,t))
> Evaluate dx/dt at (x, 14) b Additional evaluation point
(x4+1 t- (xt +
At d, + At
d,41 t vo (0
Evaluate dx/dt at (X++1, t++1) Second order step from t, to f/+1
x;+1 t- x; + Atd end for return XN end procedure
B Additional Details Lumina-Next B.1
Illustration of Next-DiT's Architecture Improvements
In Figure 10, we illustrate the structural difference between Flag- Fia and Next-DiT. The main im- provements include 3D RoPE, sandwich normalization, group-query attention, etc. Specifically, We leverage grouped-quen, attention [4] in transformer blocks, which is an interpolation between multi- head and qulti attention to improve inference speed as well as reduce memory consumption. For instance, our 2B model divides 32 query heads into 8 groups, resulting in 8 shared key-value heads. This modification reduces the parameter size while achieving similar performance compared to the original multi-head attention.
B.2 Sampling with Optimized Schedule and Higher-Order ODE Solver
In the main text, propose two optimized time schedules and combine them with higher-order ODE solvers for few-step sampling. Algorithm B.I illustrates the pseudocode for using the midpoint method with the optimized schedule.
B.3 Improved Image-Text Pairs
Lumina-T21 constructs 14M high-aesthetic synthetic text-image pairs as the training dataset. While the limited data size leads to fast convergence, these data only cover a small proportion of real image distribution, which contains diverse contents and styles. Besides, the quality of text captions in this dataset is imbalanced, consisting of incomplete or inaccurate image descriptions. As pointed out in recent works [8, 14], improving the diversity and overall quality of text prompts can greatly boost text-image alignment. Therefore, it is important to re-examine the construction of training datasets.
LAION Augmented High-Quality Images We first expand the size of our training dataset from 14M 20M using an additional 6M text-image pairs from the LAION [69] dataset. To efficiently filter high-quality images from large-scale internet datasets, we employ two-stage data filtering pipeline:
In the first stage, we aim to obtain a coarse-grained candidate pool from massive images. Specif- ically, we first filter out all images with resolutions lower than 512 x 512. Then, we further filter out images with low aesthetic scores (< 5.5), high watermark probabilities (> 0.5), and high unsafe probabilities (> 0.5). With these filtering thresholds, we construct image pool with approximately 30 million items. In the second stage, we introduce the overall image quality rank as a fine- fine-grained filtering metric to further obtain subset with superior overall quality. Concretely, we rank the 30 million coarse-filtes filtereed images based on aesthetic score, watermark probability, and safety probabil- ity, respectively. Then we sum the three types of ranks into an overall image quality rank, and the top-ranking 6 million images are selected to form our final high-quality image dataset. Upon manual evaluation, the 6 million image data filtered by our two-stage pipeline exhibit high resolution, high aesthetic appeal, and minimal probabilities of watermarking and unsafe content.
18

---

Mixture of Captioners
BLIP2
GPT-4V
6.00 coffee ando of
can dil regeentgg groine pn fromt rain bend cate
Sharegp14y
to lnnpull uelling
wo onfletennt meper wor click the
tabk ofouu deciesie
une allie picts
vagee grovntt
muaiyy
The undin
malle
slite ue
decas sie wnita ca crodes secerige the
SPHINX-X
10% sitigres thu odder aifiical
surher lec the colll fygi
The aag hare rale
Adald
faclurng
conterts be ueheoee bur semes oilling to
once byl coffee lette
and
coee moied pooe clde ses daseel
wnicit Boow
ruble beckdnp greer
sterals
cunilng softy levects seth masidrr
tabls necerrigg serring rndeect contertabe
coffer peodlyy neen darr sige cpper poder reperrt warerrade begg loom
lenarily enjoy
table ougportng
bemene
LLaVA-next
Cogvim
vocalche co apain clinamin ng aiciddr sost
offiee
megiciiig ig licerdly natiker sefa.
Qveat lugg pulll piclem poccald aut darig indern opent unjert chae upp uella sampuay
sahann
Figure 11: Illustration of mixture-of-captionees.
melo Drop
Ourind Di
Flask vdantion Content
Inference Time
Drup
332
Resolution
(a) Generated Samples
(b) Inference Time
Figure 12: (a) Qualitative results of 2K images generated by Lumina-Nexy with and without context
drop. (b) Comparison of inference speed between different settings using 50 Euler steps.
CCatureeof-Capptoners Although PixArt-a [14] and DALLE-3 [8] highlight the importance of improving the quality of text descriptions, they only leverage single captioner enhance caption accuracy. As illustrated in Figure 11, we compare captions generated from open-sourced Vision- Language Models (VLMs) such BLIP2 [44], L.LaVa [48], SPHINX [34], and Share-GPT4V [17]. We observe different VLMS describe different information of the given image in different styles at different visual granularities. Moreover, these generated captions are complementary to each other since none of them achieves 100% captioning accuracy without any hallucination. Motivated by this observation, we propose a novel Eixture-of-Caatiooee pretrained VLMs to generate mutual complementary captions. Besides, we collect a high-quality and high-resolution dataset consisting of 100K images and leverage GPT-4V to perform multi-faces holistic descriptions over these high-resolution images, depicted in Figure 11. After re-captioning using our proposed MoC, each image is paired with a caption pool with multiple descriptions from various models. During the training stage, we randomly sample one caption from collected MoC caption pools. This also resembles a kind of data augmentation that enhances model robustness during inference with all types of user inputs.
(MoC) approach which employs multiple
19

---

SDXL
PisArt-a
Lamina- Neat
Reglik: Tokyo fown.
Jewenn 9 1IP
Okione 5.PG. SUSOATT 217
Toudinar Cauas, Virdas Oapeas
Maden Noadard www.ddadiiiieeo. bodde 120
Korra (44) deEal 4ES A.A
Vimurase Thip Tokon (iily Rib
SDXL
PixArt-
Luminna- Next
Ueaaaia: Tadkuasco
Tha:: www.e. Hiaaa At
Peniu dranial:
Mandi: 2641 Worer
Rundhh: Du lyd Tays, Dana
Tadka: Taboo Kalok Edociooci
Figure 13: Results of multilingual text-to-image generation by Lumina-Next, SDXL, and PixArt-a.
C More Text to Image Generation Results
C.1 Results of Time-Aware Context Drop
As shown in Figure 12 (a), Next-DiT using the proposed Time-Aware Context Drop with a 75% downsampling ratio generates comparable and even better 2K images compared the baseline. The inference time comparison in Figure 12 (b) highlights that our Context Drop can further enhance inference speed when combined with advanced inference techniques for transformers, such as Flash Attention [22], which is more significant for higher-resolution image generation. Notably, our Con- text Drop is similar masked attention or dilated convolution, which has been proven to be the key of resolution extrapolation in diffusion U-Net [37, 40]. Hence, this high-level resemblance in our diffusion transformer architecture may explain why Context Drop can reduce repetitive and un- reasonable artifacts in high-resolution image generation, presenting interesting avenue for future research.
C.2 Results of Zero-shot Multilingual Generation
Previous T2I models, such as Imagen [67], PixArt-ax [15] and Stable Diffusion [31, 65), utilized CLIP and T5 as text encoder for generating high aesthetics images. Such choices took advantage of pretrained text encoders or multimodal aligned text encoders as good representations for text- conditional image synthesis. However, employing T5 and CLIP as text embedding shows limited
20

---

SDXL
PixArt-a
Luunian--NN Next
Acall with axaj
graatt @ Syku
Figure 14: Results of text-to-image generation with emojis by Lumina-Nees -ext, SDXL, and PixArt-a.
effectiveness in handling multilingual prompts. Different from those approaches, the Lumina series adopts decoder-only LLMs text encoders. For instance, Lumina-T2X uses 7 billion LLaMa as text encoder, while Lumina-Ne employs the smaller Gemma-2B to decrease GPU memory costs and increase training throughput. Remarkably, both LLaMa-7B and Gemma-22 endow Lumina T2I models with the ability to understand multilingual prompts in zero-shot manner, despite being primarily pretrained on English-only corpora. As shown Figure 13, our T2I models not only accurately interpret multilingual prompts but also generate images with cultural nuances and show preliminary ability to understand emojis, even though was mainly trained with English-only image-text pairs. Compared to SDXL and PixArt-a, which use CLIP and T5, there is significant improvement the understanding of multilingual prompts. We argue that decoder-only LLMs with strong language understanding abilities better align multilingual semantics than the encoder-decoder architecture of TS or the multimodal alignment CLIP text embeddings. To further explore the re- lationships between multilingual generation abilities and text encoders, we train Lumina-Next with different open-source LLMs, including Gemma-2B, Qwen- 1.8B [5], and InternLM-7B [12]. As demonstrated in Figure 15, Lumina-Ness with Qwen-1. w..B- and InternLM-7B showcase significantly better text- text-image alignment compared to Gemma-2B when using diverse Chinese prompts as inputs. Remarkably, Lumina-Next with Qwen- and InternLM-78 accurately captured the meanings and emotions classic Chinese poems, reflecting these nuances in the generated images. This observa- tion corresponds with the stronger multilingual abilities in Qwen-1.88 1.n-1.81 and InternL.-7B.. Therefore, we argue that using more advanced LLMs as text encoders enhances text-to-image generation per- formance.
C.3 Resolution Extrapolation with Any Aspect-ratio
In Figure 16, further compare the panorama (1024 x 4096 images) generation performance of the proposed Lumina-Next with MultiDiffusion [7] and DemoFusion [29] to highlight its any aspect ratio extrapolation ability. We found that although panorama generation is one of the primary goals of MultiDifuusion, its "ropy-paste" inference scheme easily leads to unreasonable content. DemoFusion adopts a progressive upscaling strategy, limiting its ability to generate arbitrary aspect ratios by the base model in extreme aspect ratio scenarios, it also fails to produce satisfactory content. In contrast, Lumina-Next perfectly maintains the ability generate at any aspect ratio under the training resolution, even when performing 4x resolution extrapolation.
C.4 Generation with Long Prompt
I Lumina-Next can handle image generation with long prompts. We give examples in Figure 17. As seen, with long prompts as inputs, Lumina-Next can produce high-quality images that strictly follow the details in the prompt. This ability remains when using other languages such as Chinese.
21

---

Gemma-28
Internl M-M77
Qwen- 1.813
ATA-T II 583988
FL/) 022
18 10
Bt.
01R MEE.N
Figure 15: Results of multilingual text-to-image generation using different L.MSs encoders.
MultDiitasion
Demoo uskon
Lamino-- Next
Figure 16: Results of 4x resolution extrapolation with extreme aspect ratio.
D Any Resolution Recognition with Lumina-Next
Lumina-Next is not only a generative modeling framework for generating images with various res- olutions but also framework for recognizing images any resolution. As illustrated in Figure 18, our Next-DiT can be seamlessly adapted into a robust backbone for image representation learning. By incorporating the sandwich normalization and 2D RoPE design into our architecture, Nixt-DiT achieves superior performance and faster convergence while maintaining computational efficiency compared the original vision transformer in image recognition tasks. Furthermore, Next-DiT overcomes the limitations previous vision transformers in processing images of varying resolu- tions, demonstrating remarkable generalization capabilities to handle input images with arbitrary resolutions and aspect ratios.
D.1 Pipelines
The unified framework of Lumina-Next for image generation and understanding at any resolution depicted Figure 18. Specifically, we introduce a learnable gate factor in the attention module, thereby avoiding the incorporation of time embeddings and label embeddings in image recogni- class token,
tion tasks. We apply global average pooling (GAP) to all image tokens to obtain which is then fed into the classification head implemented by MLP network. By integrating 2D RoPE during attention computation and incorporating RMSNorm before and after each attention and MLP layer, our Next-DiT architecture exhibits enhanced resolution extrapolation capabilities.
22

---

young sirl ploss cyyfllyy with small Duffy diy un sendy bench. Sazl veuring secenazz wists polia piysalc fropparn
diaerly zus site contorsabyy lbrrr,, randing a leades-bozzte Hibs weating sacy caddgan over alst,, sound spectacles perched
fidking villege rocky partyy the calls,, tarquriee
dog
pantly
ovelleed
colerti unloollas aovdl the sky clair with scspills shore.
booch
lamp chosic grastttther clock tole
illriioating sufily the beckgrozzé.
grand, medieval castle perched stog nolyy diff everloking list urdvaating brndecgp of fhoms and monrowe The caild' arposing store lorvers and tundo the majoiically agaioto o beckdry of' builiast Blue dky, drawisidaa spans docn, mis-tiled moal, harnere dlpp in the gentle trrns. diplaying regal cocts
WT I RE: WM NRT R S
AM
A
SA PASS #A2
F.F. BANAAB
BRY
MOM BARRE O J
Figure 17: Generated images of Lumina-Next with long prompts.
We employ progressive training scheme with a fixed-resolution pre-training stage followed by an arbitrary-resolltion fine-tuning stage, enabling Next-DiT to achieve robust image understanding across various resolutions.
Fixed Resolution Pre-training We first pre-train the Next-DiT architecture at fixed resolution by cropping and resizing images to 224 x 224, following the approach of the original VIT. This resolution balances the trade-off between training speed and input resolution, as higher resolutions require more FLOPs and reduced throughput. After pre-training, we directly evaluate our model on unseen resolutions to guide subsequent fine-tuning stage with arbitrary resolutions.
Any Resolution Fine-tuning To enable our model to handle images with any resolution and as- pect ratio, we employ an any-resolution fine-ture tin-ttnnig stage. While previous works [23, 78] also utilize multi-resolution training approach, they rely on a set of predefined input resolutions and randomly sample different resolutions for each input image from this set. These methods often alter the origi- nal resolution of aspect ratio of the input images a lot, potentially causing performance drop, and they incur additional computational costs by resizing low-resolution images to higher resolutions un- necessarily. In contrast, propose dynamic padding scheme that allows efficient model training while preserving the original resolution and aspect ratio of the input images. This approach enables us to fully exploit the multi-resolution understanding capability of Next-DiT in computationally efficient manner.
Dynamic Partitioning and Padding with Masked Attention Our aim is retain the original resolution and aspect ratio of each input image to the fullest extent during the subsequent fine-
23

---

Noley input
Input
RWS
Fatch
NX
NX
Time
Erteedding
RMC
Lobel Erieelding
Any Res, Recognition
Beckoaed CA 10
RNG
Feeffornar
Pradishin
Fred ched Velctty &y
Pradicted Neux
Dog
Figure 18: Comparison between the original architecture of Next-DiT and modified version for visual recognition.
tuning stage. Additionally, seek to ensure that the width and height of the input image divisible by the patch size of the patch embed layer, enabling us to fully utilize the image in- formation during the embedding process. To achieve this, propose dynamic partitioning and padding scheme to optimize the handling of image tokens instead of relying on a fixed res- olution, as illustrated in Figure 19. Specifically, given constraints the maximum number of patches N and the maximum aspect ratio Rmaxx we define a set candidate patch partitions (Hp, Wp) Hp W, S N and max H,, S Rmax). For an input image of size (H,, Wr), we determine its optimal patch partition by calculating the matching ratio between the original input min Hy We Subsequently,
min(H,,WW, W,)
size and the target size, given by (Hş, WP) argmax (,,,W,)et max
Ry W7
resize the input image to (HG. -P, WC-P), where P denotes the patch size. This dynamic partitioning process leads to varying sequence lengths of patch tokens for each input image. To handle batches containing inputs with different token lengths, we pad all image token sequences match the length of the longest sequence using pad token. Additionally, we introduce attention masks within each attention module to prevent unwanted interactions between pad tokens and regular image tokens.
Flexible Resolution Inference By incorporating NTK-Aware Scaled RoPE and sandwich normal- ization, Next-DiT demonstrates exceptional resolution extrapolation during inference, even without fine-tuning on varied resolutions. Furthermore, the proposed dynamic partitioning and padding scheme allows for efficient fine-tuning while preserving image resolutions and aspect ratios. Con- sequently, our model can perform flexible inference on images with arbitrary resolutions and aspect ratios.
24

---

Image
Inage
Image N
Dynamic Partition
Resize
Image Taken
Rope of position (.))
Podding Token
Seq
Seq
Warked Attertinn
Softmax
[I5e /Vd+ M
Seg
Image takens with Dynamic Padding
Attention Masks M
Figure 19: Illustration of our dynamic partitioning and padding scheme with masked attention for handling input images of arbitrary resolutions.
D.2 Setups
We conduct experiments on the ImageNet-IK dataset to validate the effectiveness of Next-DiT for image recognition. We build our Next-DiT model following the architecture hyperparameters of ViT-base [28], stacking 12 transformer layers with hidden size 768 and 12 attention heads. This configuration ensures that architecture has comparable number of parameters to the original ViT. During the fixed-resolution pre-training stage, we train the models from scratch for 300 epochs with input size of 224 224. We use the Adam optimizer with cosine decay learning rate scheduler, setting the initial learning rate, weight decay, and batch size to le-3, 0.05, and 1024, respectively. We follow the same training recipe the DeiT model [79], including its data augmen- tation and regularization strategies. In the subsequent any-resolution fine-tuning stage, we further train the pre-trained model using the proposed dynamic partitioning and padding scheme. This fine- tuning is performed for an additional 30 epochs with a constant learning rate of le-5 and a weight decay of 1e-8.
D.3 Experiments
We report the performance of our Next-DiT in Table 1. We first test our model at a fixed resolution of 224 224. Compared with DeiT-base, which has comparable parameters, Next-DiT exhibits superior performance and faster convergence. Notably, training the model for only 100 epochs achieves a Top-1 accuracy of 81.6%, nearly matching DeiT-base's result achieved after 300 epochs. Under the same training setting with 300 epochs of pre-training, Next-DiT surpasses DeiT-base with accuracy of 82.3% versus 81.6%. respectively, demonstrating its effectiveness image backbone. We also adopt the flexible resolution inference strategy, preserving the original resolution and aspect ratio of the input images validate the model's generalization capability across varied resolutions. After the additional fine-tuning stage, Next-DiT achieves 84.2% accuracy when handling arbitrary resolutions. We report performance of DeiT-base pre-trained at 224 x 224, using the same inference configuration. To align with the input resolution, we interpolate its positional embeddings accordingly. Compared to our fine-tuned Next-DiT, the performance DeiT- base drops significantly, suggesting its limitations in processing images with varied resolutions.
robust
25

---

Table 1: Comparison of Next-DiT with DeiT [79]
on ImageNet classification.
Model
Paramns
Setting
Resolution
Top-1 Acc(%)
DeiT-base [79]
86M
300E
224 x 224
8188
Next-DiT
86M
100E
224 224
81.6
Next-DiT
86M
300E
224 224
82.3
334
2034
DeiT-base [79]
86M
300F
Flexible
67.2
Figure 20: Performance of Next-DiT across different resolutions.
Next-DiT
86M
300E+30E
Flexible
84.2
We also evaluate the performance across different image resolutions, illustrated in Figure 20. Next-DiT demonstrates better generalization to larger image sizes compared to DeiT-base, even without fine-tuning. With additional fine-tuning using the proposed dynamic partitioning scheme, our model significantly enhances its ability to handle varied resolutions, particularly high resolutions such 1024 x 1024, outperforming DeiT-base by significant margin. These experimental results highlight the effectiveness of Next-DiT as powerful image backbone for recognizing images at any resolution, in addition to its capability generate images various resolutions.
E
Generating Multi- Views, Music and Audio with Lumina-Next
Lumina-Next is versatile and expandable framework for generative modeling that goes beyond lext-oonditoonig image generation. In this section, we demonstrate the application of Lumina-Nex to multi-view generation using both image and text conditioning. Additionally, extend Lumina- Next to encompass a wider range of modalities, including text-conditional audio, music, and point cloud generation.
E.1 Image- and Text-conditional Multi-View Generation
Pipelines Having witnessed the success of Next-DiT on text-to-image generation, we use Next- DiT's text-to-image model pre-trained model and extend it to image- and cext-conditional multi- view generation, called MV-NexI-DiT. as shown in Figure 21. Firstly, we introduce relative pose control to Next-DiT to distinguish between views. Specifically, for N multi-view images with the same elevation and random azimuths rendering from 3D object, taking the azimuth of the first image as the reference view, the relative azimuths of the remaining N images to the first image are calculated sequentially. The obtained relative azimuths are encoded into the same dimension as time embedding using a 2-layer MLP network after sin and cos transformation, and are added to the time embedding to inject camera control into the model. Secondly, we expanded the original text condition of Next-DiT into text and image conditions. Specifically, the same encoding and injection method Next-DiT is used for text condition. For the image condition,
VAE to encode the input image, and then concatenate it to the multi-viww noises (N + 1). This image is not used for supervision during training. In addition, when building the training data, this condition image has the same azimuth as the first supervised view (both are 0), but it is a random elevation. It is worth noting that the image condition is dropped based probability during training, and the input of the image condition is optional during inference. Finally, a progressive training strategy of views and resolution is adopted during training, and views flexible inference strategy is used during inference. Specifically, during training, we first performed training at 256 x 256 resolution with N = 4. Then it expanded to the training of 512 x 512 resolution with N 4, and finally it expanded to the training of 512 x 512 resolution with N = 8. In inference, since each object is trained using all random views, any number of view can be inferenced one time. These improvements allow MV-Next-DiT to output multi-view images with high quality and flexible azimuth.
Setups Data preparation: we use the Objaverse [24] list and caption provided by Cap3D [58] as training data. For data rendering, we first use the same elevation and 8 random azimuths for each object, render with the same camera setting, and obtain supervision data. Then the azimuths corresponding to 4 images randomly selected from the 8 rendered views, and cooperate random
26

---

Image Conditional
Velsoty
Veleotty
Velectty
Viow?
Training from Next-biT infidiaasionn
MV-Next-Dii
Text Exbodsing
Inferince
MV-Next-DiT
Teat
(All View Trairing)
(Any View Inference)
Time Enteddingg
Carch (Ootina))
Noteed View
(optisns)
Vina!
View-
Viea-
Seattve
Feee Entoeddagg
Faae EeloolingeN
Pert Erssedding
Figure 21: An illustration of multi-view images generation. Left: All view training paradigm of MV-Next-Dit Right: Any view inference paradigm of MV-Next-DiT.
elevations render input images. Each object then has 4 groups of training samples. Training details: We use Next-DiT with 600M parameters for training of MV-Next-DiT, The specific training settings are shown in Table 2.
Table 2: The detailed training setting of MV-Next-DiT.
Training method
Pre train model
Total batch (Images)
Learning rile
Iteration
A cust
256 x 256, N
Next-DiT, Text-to-Image
256 (N
le-4
100k
16GPUs 45h
512 x 512, =4
MV-Next-DT, 256 x 256, N
128 x (N +
lc-4
100k
IGG x 57h
MV-Nex-DDIT 512
lc-4
16GPUs 72N
Experiments We collected images provided by previous work, such CRM [83], test the per- formance of our MV-Next-DiT. In particular, we provide the generated results of only text condtion, text and image conditions, and the results of sampling 4 views, 6views, and 8views respectively from the setting of N=8, as shown in Figure 22. In addition, we compared the capabilities with existing multi-view methods shown in Table The overall performance demonstrates the advantages of our MV-Next-DiT. The rich higher-resolution flexible view generated results demonstrate MV-Next- DiTs excellent multi-view generation capabilities and the superiority of the Next-DiT architecture.
Table 3: Comparison of capabilities between MV-Next-DD DiT and other mutli-view methods.
Methods
Base model
Resolution
Condition type
Inference views
Zero-123 [49]
Image Variations
256 256
Image
MV [71]
v2.1
256 x 256
Text
[51]
256 256
Text
ImageDream [82]
MVDream v2.
256 256
Image
Wonder3D [52]
Image Variations
256 256
Image
7em-1255 123e+ [70]
SD v2
256 256
Image
CRM [83]
ImageDream (SD v2.1)
256 x 256
Image
MV-Next-DiI
Next-DiT'
Text Image
E.2 Text-Conditional Audio and Music Generation
Pipelines We use a modified audio VAE with a cioonoouuton--aseee model to derive audio latent.
The audio signal is sequence of mel-spectrogram sample E [0, 1JC.TT, where Ca. T respectively denote the mel channels and the number of frames. Our spectrogram autoencoder composed of 1) encoder network E' which takes samples decoder network G reconstructs the -el-spectrogram signals x' from the compressed representation and 3) multi-windos discriminator leamns to distinguish the generated samples G(z) from real ones in different mueti-re fields of mel-spectrograms.
input and outputs latent representations z; 2)
In text2audio generation, we include the dual text encoder architecture consisting of main text encoder CLAP [30] that takes the original natural language caption y as input and a temporal encoder FLAN-T5 [20] which takes the structured caption y passed by LLM as input. The final conditional
27

---

Input
Text & Image condition
Text condition
Eleguns figurine wia gizl with huir, wcaring a goldan heutpiocz
toddy bear simple fanura, arme uabendichcl
Chostly figure exting burges and flies, wezling white steet.
Blic and white fix chaneter,, featuring lange can and oggaseicc syes
Detrikal
groen
expression
Pigrinn afa cheracier in green hardls ourfs umitng waving
Adonable kitton with bug eyes and plgful exgressor.
Colerii worder oud smiptann featuring fig round eyea and tratired fiathes
Figuiite houded staryy eyes
Stylined with woodon and Amsilldd ongrwings.
Detriled chansoor motel of's musuior man fattritic outil rebtic arms
Figure 22: Results generated by MV-Next-DiT, The upper right is the result of text-to-multivieee and where the first column is the corresponding input
the rest is the result of text- toatt-inaae-tt-mlliiii image with background removed. In addition, every rows from top to bottom are the generated results of 4 views, views, and views respectively. representation is expressed
Linear(Concat((call (y), Sramp (y (y))),
Where Siext is the main text encoder and ftemp is the temporal encoder. For the text encoder in text2music generation, we use the FLAN-T5 [20] text encoder that takes the original natural lan- guage caption y input.
28

---

Setups of Text-to-Music Generation We employ a diverse combination of datasets to facilitate the training process of our model. the context of text-to-music synthesis, we exclusively employ the AP-MusicCaps [27] dataset. The culmination of these efforts yields dataset comprising 0.92
million audio-text pairs, boasting a cumulative duration of approximately 3.7K hours.
We conduct preprocessing on both text and audio data as follows: 1) We convert the sampling rate of audios to 16kHz. Prior works [84, 41, 46] pad truncate the audio fixed length (10s), while we group audio files with similar durations together to form batches to avoid excessive padding which could potentially impair model performance and slow down the training speed. This approach also allows for improved variable-length generation performance. We truncate any audio file that exceeds 20 seconds, speed up the training process. 2) For audios without natural language annotation, apply the pseudo prompt enhancement method from Make- Annee-An-a Audio [41] to construct captions aligned with the audio. We traverse the AudioCaps training set and the L.M augmented data with probability of 50%, while randomly selecting data from all other sources with probability of 50%. For the latter dataset, we use "<text & all>" their structured caption.
Experiments of with other systems, including 1) GT. the ground audio; 2) MusicGen [21]: 3) MusicLM [3]; 4) Mousai [68]; 5) Riffusion [33]; 6) MusicLDM [16]: 7) AudioLDM 2 [47]. The results are presented Table and we have the following observations: Regarding audio quality, our model consistently surpasses diffusion-bases methods and language models across FAD and KI. metrics. In terms of subjective evaluation, our model demonstrates the strong text-music alignment faithfulness.
Generation In this part, we compare the generated audio samples
Table 4:
The comparison with baseline models on the MusicCaps Evaluation set. We borrow the results of Mousai, Melody, and MusicLM from MusicGen [21].
Objective Metrics FAD (4)
Subjective Metries
Model
KL (L)
MOS-Q(1) MOS-F(1)
GroundTruth
/
/
88.42 76.
90.34
Riffusion
13.31
2.10 10
77.35
Mousai
7.50
/
Melody
5.41
MusicLM
4.00
/
MusicGen
4.50
1.41
80.74
83.70
MusicLDM
5.20
1.47
80.51
82.35
AudioLDM 2
3.81
1.22
82.24
84.35
Ours
3.75
1.24
83.56
85.69
Setups of Text-to-Audio Generation Following the benchmark studies [43, 41]. train AudioSet-SL [46] and finetune the model in the Audiocaps dataset. Overall, we have ~33 hours with 1M audio-text pairs for training data. For evaluating text-to-audio models, the AudioCaption validation set is adopted as the standard benchmark, which contains 494 samples with five human- annotated captions in each audio clip.
We conduct preprocessing on the text and audio data: 1) convert the sampling rate of audios to I 16kHz and pad short clips to 10-second long; 2) extract the spectrogram with the FFT size of 1024, hop size of 256 and crop it to a mpe-spcccro of size 80 x 624; 3) non-standard words (c.g.. ab- breviations, numbers, and currency expressions) and semiotic classes [77] (text tokens that represent particular entities that are semantically constrained, such as measure phrases, addresses, and dates) normalized.
Our models conduct a comprehensive evaluation using both objective and subjective metrics mea- sure audio quality, text-audio alignment fidelity, and inference speed. Objective assessment includes Kullback-Leible (KL) divergence, Frechet audio distance (FAD), and CLAP score to quantify audio quality. In terms of subjective evaluation, we conduct crowd-sourced human assessments employing the Mean Opinion Score (MOS) evaluate both audio quality (MOS-Q) and text-audio alignment faithfulness (MOS-F).
29

---

Audio/Music Autoencoding
Bx
Aadio//hancc Encocer
Latent
Aucio//hunc
Decedar
After N stops
Training
Inference
Noised Latent
Noise
Vocoder
Tixs Embeciing
Text
ODE Solver
Lumina-Naes Next
Lumina-
Time Embecding
Time Enbeelding
Inh.
Velocity
Velocity
Generated Audio/Music
Figure 23: An illustration of text-guided music/audio generation. It consists of the following main components: 1) VAE encode spectrogram into latent and convert it back spectrogram; text encoder to derive high-level textual representation; 3) flow transformer to inject condition; and 4) separately-traine neural vocoder to convert mel-spectrograms raw waveforms. In the following sections, we describe these components in detail.
Experiments of Text-to-Audio Generation We conduct a comparative analysis of the quality of generated audio samples and inference latency across various systems, including GT (i.e., ground- truth audio), AudioGin [43], Make-An-Audio [41], AudioLDM-LL [47], TANGO [36], Make-An- Audio [39], and AudioL 2 [47], utilizing the published models per the respective paper and the same inference steps of 100 for a fair comparison. The evaluations are conducted using the AudioCaps test and then calculate the objective and subjective metries. The results are compiled and presented in Table 6. From these findings, we draw the following conclusion:
In terms of audio quality, proposed system demonstrates outstanding performance with the FAD of 1.03, demonstrating minimal spectral and distributional discrepancies between the generated au- dio and ground truth. Human evaluation results further confirm the superiority. with MOS-Q and MOS-F scores of 77.53 and 76.52, respectively. These findings suggest a preference among evalua- tors for the naturalness and faithfulness of audio synthesized by model over baseline approaches.
For ablation, we 1) explore the effectiveness of dual text encoders for a deep understanding of arbitrary natural language input and observe that the drop in CLAP encoders has witnessed the distinct degradation text-audio alignment faithfulness; 2) study the usage of Audioset data, and find that without fine-tuning Audiocaps following previous works [47], significant drop objective evaluation happens.
Table 5: The audio quality comparisons with baselines.
Objective Metrics
Subjective Metries
Model
FAD (L)
KL (
CLAP (t)
MOS-Q(t) MOS-F(t)
GT
/
/
0.670
86.65
84.23
AudioGen-Large
1.74
1.43
0.601
Make-An-Audio
2.45
1.59
0.616
70.32
66.24
AudioL.DM
4.40
2.01
0.610
64.21
60.96
Tango
1.87
1.37
0.650
74.35
72.86
AudioLDM 2
1,90
1.48
0.622
AudioLDM 2-Large
1.75
1.33
0.652
75.86
73.75
Make-An-Audio 2
1.80
1.32
0.645
75.31
73.44
Ours
1.03
1.45
0.630
77.53
76.52
Ours (w/o Dual-encoder)
1.48
1.53
0.601
75.13
72.01
Ours (w/ Audioset)
2.09
2.06
0.562
30

---

Table 6: Ablation studies. We use DiT to denote the model architecture but with DDPM formulation.
Model
FAD (4)
KL (C)
CLAP (1)
Next-DiT
1.03
1.45
0.630
w/o Dual-encoder
1.48
1.53
0.601
w/ Audioset
2.09
2.06
0.562
DiT
1.56
1.91
0.573
Table 7: Quantitative results of point cloud generation. We multiplied the value of CD by 10
Shape
Model
MMD (4)
COV (%,
Shape
Model
MMD (J)
COV (%, 1)
PC-GAN
3.819
42.17
PC-GAN
13,436
46.23
TreeGAN
4.323
39.37
TreeGAN
14.936
38.02
PointFlow
3.688
44.98
PointFlow
13.631
41.86
Airplane
ShapeGF
3.306
50.41
Chair
ShapeGF
13.175
48.53
PDiffusion
3.276
48.71
PDiffusion
12.276
48.94
Ours
3,371
49.21
Ours
12.975
48.33
E.3 Label- and Text-Conditional Point Cloud Generation
Pipelines We propose density-invariant point cloud generator to validate the effectiveness of Lumina-Next in the 3D domain. The density-invariant property allows efficient training on
fixed point number (e.g., ., 256 points) and an inference of any point number (e.g., 1024, 2048, or more points). Different from 2D images, increasing the density of point cloud does not affect the overall shape but rather enhances the details. To achieve any point inference, we combine the Time-awan Scaled RoPE with Fourier features [76] and design Time-aware Scaled Fourier feature.
.... cos (2T0 p). (in(2: To inject the time condition of dif- 0 p), sin p)....
fusion steps into the Fourier feature, we scale the frequency via 0', = 0, s diyal
where d, = (dread 1) t 1. This leads the generator to focus on the global information in the early steps of denoising, and include more shape details in later steps. This Time-aware Scaled Fourier feature effectively distinguish continual positions in 3D space, thus model guarantees any point inference even though the model only trained on a fixed low density.
Setups We train label and textual conditioned 3D generator respectively. For the label condi- tional model, we employ the ShapeNet dataset [13] and follow [56] to preprocess it. We randomly sample 256 points from each point cloud train the model. After training, we synthesize point clouds of densities in (256, 512, 1024, 2048, 4096, 8192]. To evaluate the generation, we adopt the Minimum Matching Distance (MMD) and Coverage score (COV) as quality metrics and use Chamfer Distance (CD) as the distance measure to calculate them. For the textual guided genera- tor, we adopt the Cap3D dataset [57] to train the model, where we use RoPE positional embedding instead of the time-aware Fourier feature.
31

---

Airplane
Bench
Lamp
Chair
256 Points
512 Points
1024 Points
2048 Points
4096 Points
8192 Points
Figure 24: Examples of generated point clouds with different densities, sampled from the generator trained on 256 points.
Experiments We compare the generation performance of two categories: airplane and chair. We fine-tune our pre-trained generator on 2048 points to further improve the generation quality, We mainly compare our performance with PC-GAN [2], TreeGAN [72], PointFlow [85], ShapeGF [11], and PDiffusion [56] in Table 7. We observe that our generator achieves a satisfactory performance compared to existing models. We also visualize some generated point clouds in Figure 24, where we directly apply the generator pre-trained on 256 points to sample higher-density point clouds. We observe the density of generated point clouds decreases when increase the density (e.g., airplane and chair), which could be due to the fact that more points would cause a smoother attention matrix in self-attentin layers.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our paper proposes unified framework for generative modeling, which is corresponded with our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation at the beginning of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
Our paper does not focus on theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We depict all implementing details of our method. We are also releasing the code and checkpoints for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide the training and inference code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
mrips.cccpublic//giiee /ooeee///oodSbbmiic ssicnPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We give implementing details as well as configs for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The evaluation of large generative models is expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
35

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list GPU usage for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:r//s.....// cttppp/pibittttiiiio..dddd..ee
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss potential societal impacts in our paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We explain in our paper about filtering the dataset to remove harmful content.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the original paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com/ com/datasete has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We provide readme for our codebase.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "pdBpvRF0X4", "reviewer": "Reviewer_NAeC", "summary": "Lumina-T2X encounters challenges including training instability, slow inference, and extrapolation artifacts. This paper introduces Lumina-Next, which improves Lumina-T2X with improved architecture, scaled dataset, optimized sampling techniques, and better context extrapolation strategy.\n\nOn the architecture side, they replace 1D RoPE with 3D RoPE to eliminate the inappropriate positional priors to model images and videos using the attention mechanism. They further removed all learnable identifiers in Flag-DiT, and introduced the sandwich normalization block in attention modules to control the activation magnitudes.\n\nOn the context extrapolation side, the authors propose a Frequency-Aware Scaled RoPE, reducing content repetition during extrapolation, and a novel Time-Aware Scaled RoPE for diffusion transformers to generate high-resolution images with global consistency and local details.\n\nOn the sample techniques side, the authors propose time schedules tailored for flow models to minimize discretization errors. They further combine the optimized schedules with higher-order ODE solvers, achieving high-quality text-to-image generation samples within 5 to 10 steps. \n\nWith the improved architecture, context extrapolation, and fast sampling techniques, the authors show that Lumina-Next yields strong generation capabilities, such as generating high-quality images much larger than its training resolution and multilingual text-to-image generation. \n\nFurthermore, they extend Lumina-Next’s versatility to other modalities, such as multiviews, audio, music, and point clouds, with minimal modifications, achieving superior results across these diverse applications.", "questions": "With all those changes on controlling the activation magnitudes, what about simply using grad clip norm to reduce gradient norms?", "limitations": "There are more interesting contents in the appendix, which can be moved to the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written with extensive results, applications and details, which is valuable to the community.\n- The paper provides practical improvements over architecture, dataset, sampling techniques, and context extrapolation strategy. \n- The rethinking of existing model design is appreciated, especially showing a much smaller network can get better performance.", "weaknesses": "- the changes over Lumina-T2X are a bit incremental, considering Lumina-T2X is just released a month before the submission. Some of the designs in Lumina-T2X itself such as learnable tokens are not essential and only exist in Lumina-T2X, so the improvement (removing it) is not applicable to other DiTs, so removing extra learnable tokens itself can hardly be considered innovation. \n- the 3D rope does not show advantage over native designs, such as normal 2D + temporal positional embedding. There is no application showing the usage of 3D RoPE, such as video generation.\n- the ImageNet experiments can hardly justify the improvements. The comparison between Lumina-Next and SiT/Flag-DiT is not clear, such as whether the model sizes are the same.\n- there is a lack of explanation why model size can be reduced from the Flag-DiT 5B/7B model significantly. \n- The adoption of newer and stronger text encoders is also not ablated, so the improvement over Lumia-T2X cannot fully be attributed to the proposed changes.\n- There is a lack of quadratic evaluation metrics such as TIFA and ImageReward scores for fair comparison across models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Incremental Changes.\n\nGood point! We agree that Lumina-Next is largely based on Lumina-T2X with several improvements. However, we would like to highlight that these changes are made after comprehensive examinations and are essential for scaling this flow-based diffusion transformer or applying it in diverse fields. For example, we discover the exploding network activation and then propose to use sandwich normalization, which effectively eliminates the instability during both training and inference. In addition, we would like to highlight the additional contribution presented in our paper, as detailed in the Appendix. These include (1) zero-shot multilingual generation, (2) flexible visual recognition, and (3) text-to-multiview/music/audio/point clouds generation, all of which are yet to be explored for flow-based diffusion transformers.\n\n### Q2: The advantage of 3D rope.\n\nWell said! The current implementation of 3D RoPE is almost equivalent to normal 2D RoPE + temporal positional embedding. We use 3D RoPE to continue one of the excellent features of Lumina-T2X — a unified framework for different modalities. For example, we adopt the 3D RoPE for multiview generation as a natural extension, since it can be treated as a sequence of image frames. The high-quality and consistent multiview results shown in Figure 25 verified the effectiveness of 3D RoPE in modeling 3D signals. Due to current limitations in training resources and data availability, Lumina-Next is still a unified multimodal framework with independently trained models. However, we consider a unified multimodal generative model as a necessary future step, so this unified framework can serve as an essential foundation for the next stage.\n\n### Q3: The ImageNet experiments can hardly justify the improvements.\n\nSorry for the confusion. Our comparison in Figure 5 is completely fair — with the same parameter settings and model sizes. Besides, the results from the ImageNet benchmark are also widely regarded as consistent when model scaling up and are commonly used as a preliminary validation test to reduce trial-and-error costs, e.g., EDM [3] conducts extensive experiments on ImageNet to gain valuable insights, which are further proven to be transferrable to various scales and fields.\n\n[3] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.\n\n### Q4:  Lack of explanation about model size.\n\nThanks for posing this question. Since the scale of data we use is much smaller than some open-source models (e.g., 30M in Lumina-Next V.S. over 1B in SD3), we found that, given the current data scale, the benefits of further scaling up the model to more parameters are not significant. This aligns with the finding in Lumina-T2X, where the 3B and 7B models achieved similar results. To achieve a sweet point between performance and efficiency, we adopted a 2B model, making it more accessible for contributors to run Lumina-Next on their personal devices. We have added more explanation in the revised version of our paper.\n\n### Q5: The adoption of Gemma.\n\nPlease allow us to clarify that we chose Gemma 2B mainly for better efficiency, and it is not considered as one of our contributions. Actually, Gemma 2B is weaker than LLaMA 7B used in Lumina-T2X. The influence of using different text encoders is valuable and worth exploring. However, this is not the major focus of our paper. Besides, the effectiveness of our core contributions has been well validated as follows: (i) the experiments on the ImageNet benchmark demonstrate the improvements of Next-DiT over Flag-DiT, and (2) fair experiments on the Lumina-Next T2I model have confirmed the effectiveness of our resolution extrapolation and inference acceleration techniques. We have further supplemented the quantitative results of ablation studies here.\n\n| Method | CLIP Score | FID |\n| --- | --- | --- |\n| Position Extrapolation | 28.01 | 81.54 |\n| Position Interpolation | 27.36 | 113.24 |\n| NTK-aware Scaled RoPE | 28.21 | 61.89 |\n| Frequency-aware Scaled RoPE | 28.52 | 59.92 |\n| Time-aware Scaled RoPE | 28.93 | 58.66 |\n\n| Method | Steps | CLIP Score | FID |\n| --- | --- | --- | --- |\n| Uniform | 10 | 26.87 | 78.23 |\n| Rational | 10 | 28.57 | 64.12 |\n| Sigmoid | 10 | 28.39 | 63.40 |\n\n### Q6: Lack of quadratic evaluation metrics.\n\nThank you for the suggestion. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our model, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.\n\n| Model | Winrate |\n| --- | --- |\n| SD3 | 69.5% |\n| PixArt | 43.6% |\n\n### Q7: What about simply using grad clip norm.\n\nGood point! We have tried using techniques like gradient clip, but they are not that effective in preventing model training from crashing. This finding has also been mentioned in other papers such as ViT-22B [4]. We have included these empirical findings in the paper, hoping to provide some helpful insights to the community.\n\n[4] Dehghani, Mostafa, et al. \"Scaling vision transformers to 22 billion parameters.\" *International Conference on Machine Learning*. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jZUGmPk2eY", "reviewer": "Reviewer_FoVd", "summary": "This paper introduces the next generation of Lumina-T2X, Lumina-Next, which offers improved architecture, a scaled dataset, optimized sampling techniques, and a more efficient context extrapolation strategy. The improved architecture shows faster convergence rates, while the optimized sampling technique enables high-quality text-to-image generation with fewer steps. Through visual results, the authors validate the improvements provided by refining Flag-DiT in Lumina-T2X.", "questions": "* Text-to-image benchmark result should be evaluated and reported", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Next-DiT brings excellent high-resolution extrapolation generation.\n\n* Compared to previous work, it provides better generation quality under a few-step generation setting.", "weaknesses": "* There is a lack of quantitative experiment comparison results on some Text2Image benchmarks to demonstrate the superiority and effectiveness of the proposed method. Most of the comparisons are shown through visualization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: There is a lack of quantitative experiment comparison results on some Text2Image benchmarks to demonstrate the superiority and effectiveness of the proposed method.\n\nThank you for the suggestion. We have further supplemented the quantitative comparison experiments. \n\nDue to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.\n\n| Model | Winrate |\n| --- | --- |\n| SD3 | 69.5% |\n| PixArt | 43.6% |\n\nIn addition to the ImageNet experiments in our paper, we also conduct various ablation studies to verify the effectiveness of each proposed component.\n\n| Method | Resolution | CLIP Score | FID |\n| --- | --- | --- | --- |\n| Position Extrapolation | 2048✕2048 | 28.01 | 81.54 |\n| Position Interpolation | 2048✕2048 | 27.36 | 113.24 |\n| NTK-aware Scaled RoPE | 2048✕2048 | 28.21 | 61.89 |\n| Frequency-aware Scaled RoPE | 2048✕2048 | 28.52 | 59.92 |\n| Time-aware Scaled RoPE | 2048✕2048 | 28.93 | 58.66 |\n\n| Method | Resolution | Steps | CLIP Score | FID |\n| --- | --- | --- | --- | --- |\n| Uniform | 1024✕1024 | 10 | 26.87 | 78.23 |\n| Rational | 1024✕1024 | 10 | 28.57 | 64.12 |\n| Sigmoid | 1024✕1024 | 10 | 28.39 | 63.40 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6Rq9dlnjm", "reviewer": "Reviewer_ki5b", "summary": "The paper introduces a new multi-modal generation model, Lumina-Next, which extends the previous Lumina-T2X approach by several innovations:  i) 3D rotary position embedding (RoPE) ii) extra normalization to stablize training iii) frequency- and time- aware scaled RoPE for training free resolution extrapolation iv) improved time scheduling and higher-order ODE solvers for reverse sampling. Lumina-Next aims to achieve faster, better, and more efficient generation on multiple text-to-X tasks, e.g. text-to-image/audio/point-cloud, etc.", "questions": "Please refer to the weaknesses", "limitations": "Limitations have been discussed in the Appendix", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Topic: the paper studies multi-modal generation using a unified framework, which would be of great interest to the community. The paper also promises to release the source code to support future research\n\nMethodology: the proposed technical extensions for Lumina-T2X are technically sound. \n\nExperiment: Lumina-Next demonstrates promising qualitative results on text-to-image generation, training-free resolution extrapolation, and text-to-point-cloud generation. The paper also provides quantitative evaluation on text-to-audio/music generation.", "weaknesses": "The reviewer appreciates the authors’ effort on improving text-to-X generation. However, as a technical submission for NeurIPS, the paper is fully convincing as it provides limited quantitative evaluation. \n\nAs a representative reader, the reviewer would expect\n\ni) quantitative evaluations on all generation tasks\n\nii) extra apples-to-apples evaluations on any resolution recognition, as for now, there is only one baseline, DeiT-Base. The comparison is also apples-to-oranges as the models have different architectures and were trained on different data.\n\niii) comprehensive quantitative ablations on the proposed technical extensions, a) 3D RoPE,  b) extra normalization, c) frequency- and time- aware scaled RoPE, d) improved time scheduling and higher-order ODE solvers for reverse sampling, etc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Quantitative evaluations on all generation tasks.\n\nThank you for the suggestion. We have further supplemented the quantitative comparison experiments. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.\n\n| Model | Winrate |\n| --- | --- |\n| SD3 | 69.5% |\n| PixArt | 43.6% |\n\n### Q2: Extra apples-to-apples evaluations on any resolution recognition.\n\nThanks for pointing this out! The lack of any resolution fine-tuning could indeed lead to unfair comparisons. To address this, we have added experiments with and without any resolution fine-tuning for any resolution inference. We also included two additional comparison methods: PVT-Base and Swin-Base. The experimental results show that under completely consistent settings, Next-DiT still achieves better performance over Deit-Base and PVT-Base, demonstrating the versatility of Next-DiT as a representation learning network for arbitrary resolutions. However, compared to Swin-Base, an advanced architecture with complicated architecture design for visual recognition tasks, Next-DiT still has room for improvement, objectively highlighting the superiority of specialized models in dedicated tasks. We have supplemented these results in the revised version of our paper.\n\n| Model | Size | Training Resolution | Inference Resolution | Results |\n| --- | --- | --- | --- | --- |\n| Deit-B | 86M | 224✕224 | Any-resolution  | 67.2 |\n| PVT-B | 61M | 224✕224 | Any-resolution  | 71.6 |\n| Swin-B | 88M | 224✕224 | Any-resolution  | 74.1 |\n| Next-DiT-B | 86M | 224✕224 | Any-resolution  | 72.6 |\n| Deit-B | 86M | 224✕224 + Any-resolution Fine-tuning | Any-resolution  | 82.9 |\n| PVT-B | 61M | 224✕224 + Any-resolution Fine-tuning | Any-resolution  | 83.2 |\n| Swin-B | 88M | 224✕224 + Any-resolution Fine-tuning | Any-resolution  | 85.3 |\n| Next-DiT-B | 86M | 224✕224 + Any-resolution Fine-tuning | Any-resolution  | 84.2 |\n\n### Q3: Comprehensive quantitative ablations on the proposed technical extensions, a) 3D RoPE, b) extra normalization, c) frequency- and time- aware scaled RoPE, d) improved time scheduling and higher-order ODE solvers for reverse sampling, etc.\n\nThanks for the advice! Due to time constraints, we are unable to provide all independent ablation experiments for 3D RoPE and additional normalization here. However, our results in Fig. 6 demonstrate that Next-DiT, the combination of these aforementioned techniques, achieved an overall improvement on the ImageNet benchmark compared to the Flag-DiT used by Lumina-T2X. Additionally, In the table below, we supplement quantitative ablation studies on the Lumina-Next T2I model to verify the effectiveness of our resolution extrapolation and inference acceleration techniques. We have added these results to our main paper.\n\n| Method | Resolution | CLIP Score | FID |\n| --- | --- | --- | --- |\n| Position Extrapolation | 2048✕2048 | 28.01 | 81.54 |\n| Position Interpolation | 2048✕2048 | 27.36 | 113.24 |\n| NTK-aware Scaled RoPE | 2048✕2048 | 28.21 | 61.89 |\n| Frequency-aware Scaled RoPE | 2048✕2048 | 28.52 | 59.92 |\n| Time-aware Scaled RoPE | 2048✕2048 | 28.93 | 58.66 |\n\n| Method | Resolution | Steps | CLIP Score | FID |\n| --- | --- | --- | --- | --- |\n| Uniform | 1024✕1024 | 10 | 26.87 | 78.23 |\n| Rational | 1024✕1024 | 10 | 28.57 | 64.12 |\n| Sigmoid | 1024✕1024 | 10 | 28.39 | 63.40 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PEc9Eei9g4", "reviewer": "Reviewer_XpYM", "summary": "The paper introduces Lumina-Next, an enhanced version of the Lumina-T2X model, which is a Flow-based Large Diffusion Transformer (Flag-DiT) aimed at transforming noise into various modalities like images and videos based on text instructions. Compared with Lumina-T2X, Lumina-Next introduces the following improvements:\n* A redesigned architecture named Next-DiT, featuring 3D Rotary Position Embedding (RoPE) and sandwich normalization to stabilize training and inference.\n* Frequency- and Time-Aware Scaled RoPE for better resolution extrapolation in text-to-image generation.\n* An optimized time discretization schedule combined with higher-order Ordinary Differential Equation (ODE) solvers for faster and high-quality generation.\n\nThe model demonstrates stronger performance in text-to-image generation, resolution extrapolation, and multilingual capabilities in a zero-shot manner. It also shows versatility across different tasks such as visual recognition, multi-views, audio, music, and point cloud generation.", "questions": "na", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper, like SD3 and PixArt, explores image generation within the DiT architecture, and the codes and models have been open-sourced, which is beneficial to the community.\n- Strong functionality. Without additional training, Lumina-Next can easily generate higher-resolution images and produce images of decent quality in fewer inference steps.", "weaknesses": "- The paper mainly demonstrates the advantages over comparative methods in terms of functionality but does not show comparisons with these state-of-the-art methods in terms of text-image alignment and image quality.\n- Although the paper's method is about DIT, it does not demonstrate the manifestation and properties of scaling laws in the generation domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The work does not show comparisons with these state-of-the-art methods in terms of text-image alignment and image quality.\n\nThank you for the suggestion. We have further supplemented the quantitative comparison experiments. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.\n\n| Model | Winrate |\n| --- | --- |\n| SD3 | 69.5% |\n| PixArt | 43.6% |\n\n### Q2: The paper does not demonstrate the manifestation and properties of scaling laws in the generation domain.\n\nGood point! The nature of good scaling up and the consequent emergence of modeling capabilities are essential properties for evaluating GenAI nowadays, which has been explored in Lumina-T2X -- scaling up DiT from 0.6B to 7B. However, in this paper, since the scale of data we use is much smaller than the leading open-source models (e.g., 30M in Lumina-Next V.S. over 1B in SD3), the benefits of further scaling up the model to more parameters are not significant. Therefore, we adopt a 2B model to achieve a sweet point between performance and efficiency. The focus of this paper is more on how to make the DiT-based models achieve more stable training, higher inference quality and speed. In addition, we would like to highlight the additional contribution presented in our paper, as detailed in the Appendix. These include (1) zero-shot multilingual generation, (2) flexible visual recognition, and (3) text-to-multiview/music/audio/point clouds generation, all of which are yet to be explored for flow-based diffusion transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EzCuoeDmFL", "reviewer": "Reviewer_Nfk5", "summary": "Lumina-Next is an improved version of Lumina-T2X, featuring a core architecture  that employs a Flow-based Large Diffusion Transformer (Flag-DiT). Through empirical experiments and analysis, Lumina-Next introduces an enhanced Next-Dit architecture and develops a fast sampling algorithm, which boosts the model's generalization capability and improves both training and inference efficiency.", "questions": "see weaknesses.", "limitations": "Although the paper discusses limitations in the supplementary materials, I believe it could benefit from adding some discussion on whether the training data might introduce biases related to violence, pornography, race, etc.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper focuses on Lumina-T2X and provides empirical analysis and discussion on the training stability, inference efficiency, and extrapolation performance of this type of Flow-based Large Diffusion Transformer. Its advantages are as follows:\n\n- This paper uses 3D RoPE to replace 1D RoPE, enabling the perception of positional information in both spatial and temporal dimensions without the need for learnable [nextline] and [nextframe] identifiers. These improvements help stabilize training and benefit the extrapolation of visual sequence context.\n- Sandwich normalization is applied to the model structure to prevent the accumulation of errors due to increased model depth, thereby preserving model performance.\n- Different extrapolation schemes is ablated, and Frequency- and Time-Awareness Scaled RoPE was proposed to improve resolution extrapolation.\n- A time schedule suitable for flow-based diffusion models was proposed, improving the visual quality of fast sampling produced with 10-20 NFEs.\n\nOverall, this paper addresses some of the shortcomings of Lumina-T2X and provides insights for the development of large-scale flow-based diffusion transformers.", "weaknesses": "- Whether identifiers such as [nextline] and [nextframe] are unnecessary when using 3D RoPE is not sufficiently discussed in the paper.\n- The paper lacks a comparison with stable-diffusion-3, which is also a flow-based diffusion model.\n- Is the proposed Time Schedule universally applicable at low NFEs (<10)? Figure 7 contains too few examples to reflect the actual effect. Are there any numerical comparisons? \n\nOverall, this paper leans towards empirical research and analysis, making it more akin to a technical report.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Whether identifiers such as [nextline] and [nextframe] are unnecessary when using 3D RoPE is not sufficiently discussed in the paper.\n\nGood point! Lumina-T2X adopted learnable [nextline] and [nextframe] tokens to achieve flexible modeling of 2D/3D signals with 1D RoPE. However, we found that modeling any modality as the 1D sequence leads to certain limitations, e.g., misalignment with the image/video's physical properties and limited multi-dimensional resolution extrapolation ability (refer to Q1@Reviewer NAeC for more details). For these reasons, we introduced 2D/3D RoPE in lumina-next to replace 1D RoPE with learnable tokens. In this case, we naturally don't need learnable tokens to assist the model in understanding the 2D / 3D structures. We shall supplement the relevant discussion upon acceptance. \n\n### Q2: The paper lacks a comparison with Stable Diffusion 3, which is also a flow-based diffusion model.\n\nThanks for the suggestion! The SD3 model remains closed-source when submitting this paper. Here we further supplement the quantitative comparison results with SD3. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.\n\n| Model | Winrate |\n| --- | --- |\n| SD3 | 69.5% |\n| PixArt | 43.6% |\n\n### Q3: Is the proposed Time Schedule universally applicable at low NFEs (<10)? Figure 7 contains too few examples to reflect the actual effect. Are there any numerical comparisons?\n\nThanks for posing this question. As a zero-shot method simply changing the time schedule, it is hard to get satisfactory results using extremely low NFEs (<10). Usually, using ~20 NFEs can get decent results. Besides, the design of the temporal condition in Time-aware scaled RoPE makes it intuitively more sensitive to the inference step number. Here, we give the results for NFEs of 40, 20, and 10 with NTK/Time- aware Scaled RoPE, and we can see that resolution extrapolation with Time-aware scaled RoPE is getting worse. However, the overall trend is consistent with the degradation of image quality in the case of using NTK-aware Scaled RoPE, and there is no significant nullification. We will add these experimental results to the main paper along with the visualization results.\n\n| Method | NFE | Time-aware Scaled RoPE | NTK-aware Scaled RoPE |\n| --- | --- | --- | --- |\n| Time-aware Scaled RoPE | 40 | 28.93 | 58.66 |\n| Time-aware Scaled RoPE | 20 | 28.34 | 60.07 |\n| Time-aware Scaled RoPE | 10 | 28.21 | 67.98 |\n| NTK-aware Scaled RoPE | 40 | 28.21 | 61.89 |\n| NTK-aware Scaled RoPE | 20 | 27.82 | 62.19 |\n| NTK-aware Scaled RoPE | 10 | 27.71 | 68.59 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a “bag of tricks” style paper that proposes a set of techniques to improve upon the recent Lumina-T2X family of models. Although one can argue that these improvements are largely incremental on their own, there is still great value to testing and integrating them in a single system. Reviewers are generally positive about the contributions, and the AC believes that the authors did a good job in the rebuttal to address some of the weaknesses on the evaluation front. The AC leans towards accepting this work based on the promise that the code and model weights will be released, which should be of great value to the community.
Decision: Accept (poster)