Title: Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generative Models

Text-to-Image Generation

Diffusion Models

Flow Matching
Abstract: Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers (Flag-DiT) that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions. Despite its promising capabilities, Lumina-T2X still encounters challenges including training instability, slow inference, and extrapolation artifacts. In this paper, we present Lumina-Next, an improved version of Lumina-T2X, showcasing stronger generation performance with increased training and inference efficiency. We begin with a comprehensive analysis of the Flag-DiT architecture and identify several suboptimal components, which we address by introducing the Next-DiT architecture with 3D RoPE and sandwich normalizations. To enable better resolution extrapolation, we thoroughly compare different context extrapolation methods applied to text-to-image generation with 3D RoPE, and propose Frequency- and Time-Aware Scaled RoPE tailored for diffusion transformers. Additionally, we introduce a sigmoid time discretization schedule for diffusion sampling, which achieves high-quality generation in 5-10 steps combined with higher-order ODE solvers. Thanks to these improvements, Lumina-Next not only improves the basic text-to-image generation but also demonstrates superior resolution extrapolation capabilities as well as multilingual generation using decoder-based LLMs as the text encoder, all in a zero-shot manner. To further validate Lumina-Next as a versatile generative framework, we instantiate it on diverse tasks including visual recognition, multi-views, audio, music, and point cloud generation, showcasing strong performance across these domains. By releasing all codes and model weights at https://github.com/Alpha-VLLM/Lumina-T2X, we aim to advance the development of next-generation generative AI capable of universal modeling.

Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT

Abstract

Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers (Flag-DiT) that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions. Despite its promising capabilities, Lumina-T2X still encounters challenges including training instability, slow inference, and extrapolation artifacts. In this paper, we present Lumina-Next, an improved version of Lumina-T2X, showcasing stronger generation performance with increased training and inference efficiency. We begin with a comprehensive analysis of the Flag-DiT architecture and identify several suboptimal components, which we address by introducing the Next-DiT architecture with 3D RoPE and sandwich normalizations. To enable better resolution extrapolation, we thoroughly compare different context extrapolation methods applied to text-to-image generation with 3D RoPE, and propose Frequency- and Time-Aware Scaled RoPE tailored for diffusion transformers. Additionally, we introduce a sigmoid time discretization schedule to reduce sampling steps in solving the Flow ODE and the Context Drop method to merge redundant visual tokens for faster network evaluation, effectively boosting the overall sampling speed. Thanks to these improvements, Lumina-Next not only improves the quality and efficiency of basic text-to-image generation but also demonstrates superior resolution extrapolation capabilities and multilingual generation using decoder-based LLMs as the text encoder, all in a zero-shot manner. To further validate Lumina-Next as a versatile generative framework, we instantiate it on diverse tasks including visual recognition, multi-view, audio, music, and point cloud generation, showcasing strong performance across these domains. By releasing all codes and model weights at https://github.com/Alpha-VLLM/Lumina-T2X , we aim to advance the development of next-generation generative AI capable of universal modeling.

1 Introduction

Scaling diffusion transformers has unveiled significant improvements in text-conditional image and video generation [15, 14, 31, 35, 62]. Notably, Lumina-T2X [35] introduces flow-based large diffusion transformer (Flag-DiT), which has proven to be a stable, scalable, flexible, and trainingefficient architecture for generative modeling across various data domains. For instance, the use

Resolution Extrapolation (2K):

Inka warrior with a war

make up, medium shot, natural light, Award winning wildlife.

Resolution Extrapolation (2K):

A regal swan glides

gracefully across the surface of a tranquil lake, its snowy

white feathers ruffled by the gentle breeze.

𝟏𝟎𝟐𝟒×𝟒𝟎𝟗𝟔

Resolution Extrapolation (Panorama):

The majestic Eiffel Tower standing tall against the Parisian skyline at dusk..

Multi-view: Elegant figurine of a girl with blue hair, wearing a golden dress and headpiece.

Audio:

A dreamy and trippy instrumental jam with an easygoing and mellow single electric guitar playing a simple tune, accompanied by a guitar solo, echo, and effect peda.

Point Cloud:

A blue chair with long red chair legs.

of advanced normalization techniques like KQ-Norm and RMSNorm enable stable mixed-precision training when scaling Lumina-T2X to a 5B Flag-DiT with a 7B LLaMA [80]. Besides, the flow matching framework and Rotary Position Embedding (RoPE) unlock the potential of Lumina-T2X to generate data with arbitrary resolutions, aspect ratios, and durations during sampling. While Lumina-T2X achieves superior visual aesthetics quality with remarkably low training resources, it suffers from weak image-text alignment, slow inference, and extrapolation artifacts due to inadequate training, insufficient training data, and inappropriate context extension strategy.

To fully unleash the potential of scaling diffusion transformers for generative modeling, we present the next generation of Lumina-T2X, Lumina-Next with improved architecture, scaled dataset, optimized sampling techniques, and better context extrapolation strategy, all of which make LuminaNext stronger and faster. Specifically, the key improvements of Lumina-Next are listed as follows:

Architecture of Next-DiT We revisit the architecture design of Flag-DiT and find suboptimal components for scalable generative modeling in the visual domain. First, we replace the 1D RoPE with 3D RoPE to eliminate the inappropriate positional priors to model images and videos using the attention mechanism. We further remove all learnable identifiers in Flag-DiT, such as [nextline] and [nextframe] , since our 3D RoPE already provides sufficient 3D positional information. Next, we task an in-depth analysis of the instability during both training and sampling and find it stems from the uncontrollable growth of network activations across layers. Therefore, we introduce the sandwich normalization block [25] in attention modules, which is proven to effectively control the activation magnitudes. Additionally, we employ the Grouped-Qurey Attention [4] to reduce com-

putational demand especially when generating high-resolution images. The improved architecture of Next-DiT is validated on the ImageNet-256 benchmark, demonstrating a faster convergence rate compared to both Flag-DiT and SiT [59].

Frequency- and Time-Aware Scaled RoPE Inspired by the recent progress of context extrapolation in LLMs [18, 1, 64], Lumina-T2X directly applies NTK-Aware Scaled RoPE to generate higher resolution images thanks to the design of 1D RoPE. However, a comprehensive analysis of using 3D RoPE for length extrapolation in the vision domain is still lacking, especially tailored for diffusion and flow models. To bridge this gap, we first conduct a holistic comparison between existing extrapolation methods applied to text-to-image diffusion transformers, including Position Extrapolation, Position Interpolation [18], and NTK-Aware Scaled RoPE [1]. We rethink the relationship between the encoding frequency in RoPE and the generated content and propose a Frequency-Aware Scaled RoPE, significantly reducing content repetition during extrapolation. Furthermore, considering the difference between the auto-regressive generation of LLMs and the time-aware generation process with fixed sequence length in diffusion models, we propose a novel Time-Aware Scaled RoPE for diffusion transformers to generate high-resolution images with global consistency and local details.

Optimized Time Schedule with Higher-Order Solvers In contrast to various time schedules [61, 74, 42] and advanced SDE/ODE solvers [53, 54, 26, 42, 81] in diffusion models, most flow models [45, 50, 31] still adopts Euler's method with uniform time discretization during sampling. SiT [59] conducts a preliminary exploration of using different diffusion schedules and higher-order ODE solvers on the ImageNet benchmark. In this work, we demonstrate that existing diffusion schedules are not suitable for flow models and propose novel time schedules tailored for flow models to minimize discretization errors. We further combine the optimized schedules with higher-order ODE solvers, achieving high-quality text-to-image generation samples within only 5 to 10 steps.

Time-Aware Context Drop To reduce the network evaluation time of each single step, we propose dynamically merging latent tokens to reduce redundancy in attention blocks. Different from the complex merge-and-unmerge algorithm in Token Merging [9], we employ a simple average pooling for keys and values to aggregate similar context tokens in spatial space. Additionally, we augment this Context Drop with time awareness to align with the dynamic sampling process of diffusion models. The resulting Time-Aware Context Drop is validated to effectively enhance inference speed while maintaining visual quality, achieving a 2 × inference-time speed up in 1K resolution image generation. When integrated with advanced attention inference techniques like Flash Attention [22], our method further improves inference speed of generating ultra-high resolution images.

By composing all improvements over Lumina-T2X, we reach Lumina-Next. Our Lumina-Next, featuring a 2B Next-DiT and Gemma-2B as text encoder, achieves better text-to-image generation compared to Lumina-T2X with a 5B Flag-DiT and LLaMA-7B [80] as text encoder, while significantly reducing training and inference costs. Unlike previous models [67, 65, 31, 15, 14] that rely on CLIP [66] or T5 [20], Lumina-Next demonstrates strong zero-shot multilingual ability with LLMs as text encoder. Moreover, equipped with improved architecture and the proposed inference techniques, Lumina-Next can achieve tuning-free 2K and few-step generation. We also show that Lumina-Next's framework can be easily extended to other modalities with excellent performance, demonstrating that Lumina-Next is a unified, versatile, powerful, and efficient framework for generative modeling. All codes and checkpoints of Lumina-Next are released. We hope the reproducibility of Lumina-Next can foster transparency and innovations in the generative AI community.

2 Improving Lumina-T2X

2.1 Architecture of Next-DiT

In Lumina-T2X [35], Flag-DiT serve as the core architecture with flow matching formulation, ensuring training stability and scalability for larger models. In this section, we propose Next-DiT, an improved version of Flag-DiT with the following key modifications. The comparison of detailed architectures of Next-DiT and Flag-DiT can be found in the Appendix B.1.

Replacing 1D RoPE and Identifiers with 3D RoPE Flag-DiT replaces absolute positional embedding (APE) with RoPE to enable flexibility of extrapolating to unseen resolution during training.

It further introduces learnable tokens including [nextline] and [nextframe] as identifiers to model images and videos with different aspect ratios and durations. We discovered that 1D RoPE is a lossy representation to encode accurate spatial-temporal positions and the learnable identifiers introduce additional design complexity, all of which can be simplified to 3D RoPE without losing any information.

Specifally, Lumina-T2X encode input from different modalities into latent frames of shape [ H,W,T,C ] , where T = 1 for images, T = numframes for videos, and T = numviews for multi-view images. Then, 1D RoPE is applied

(a)

(b)

to the flattened 1D token sequences, which is formulated as follows given the m -th query and n -th key q m , k n ∈ R d head :

where Θ = Diag ( θ 1 , ..., θ d head / 2 ) is the frequency matrix. However, 1D RoPE directly treats flattened frame tokens as 1D language sequences, discarding the spatial and temporal relations between different positions. As shown in Figure 2(a), when applying 1D RoPE to the 2D images, the resulting long-term decaying property is obviously incorrect for the 2D scenario. Inspired by recent works that introduce 2D RoPE in the image domain [55, 32], it is natural to extend the original 1D RoPE to 3D RoPE for spatial-temporal modeling. We divide the embedding dimensions into three independent parts and calculate positional embedding for the x -axis, y -axis, and z -axis separately. Given then 3D coordinates of the m -th query and n -th key, the 3D RoPE is defined as:

where Θ t = Θ h = Θ w = Diag ( θ 1 , ..., θ d head / 6 ) is the divided frequency matrix for each axis and ∥ denotes concatenating complex vectors of different axes at the last dimension. The attention scores with 3D RoPE are calculated by taking the real part of the standard Hermitian inner product:

where Re [ · ] denotes the real part of complex numbers and ∗ indicates complex conjugates.

With the introduction of 3D RoPE, we provide a unified and accurate spatial-temporal representation of positional encoding for different modalities, without the need to add extra learnable tokens such as [nextline] and [nextframe] to indicate the row, column, and frame index. Our 3D RoPE also incorporates the 2D RoPE formulation since it naturally becomes 2D RoPE when applied to images with only 1 frame. Figure 2 shows the 2D RoPE assigns high attention scores to positions within the same row and column, which is a natural prior for images. Furthermore, our decoupled implementation of 3D RoPE on spatial and temporal axes better unlocks the potential for length extrapolation when combined with inference-time techniques mentioned in Section 2.2. We demonstrate the superior zero-shot extrapolation results with arbitrary resolutions in Section 3.1.

Controling Magnititude of Network Activations To stabilize the training process when scaling up model size and token length, Lumin-T2X replaces all LayerNorm with RMSNorm and add QKNorm before key-query dot product attention computation. Though effective to some extent, this recipe is insufficient for stabilizing the training of extremely large diffusion transformers on long sequences. A closer examination of the Flag-DiT architecture reveals the presence of long signal paths without any normalization due to the residue structure of attention and mlp layers. We argue that these unnormalized paths appeared in many diffusion transformer architectures [63, 15, 59] can accumulate signal values across layers, causing uncontrollable growth of network activations, especially in deeper layers. We validate this hypothesis by visualizing the evolution of activation magnitudes over different depths of the text-to-image model using 500 random samples at different timesteps, as shown in Figure 3. What's worse, the uncontrollable network activations not only lead

×104

×106

25

7

Mean - w/ sandwich norm

6'0

Discretization Error

1.2

250

Mean - w/o sandwich norm

1.0

Curvature

Mean of Activation Norms

6

Max - w/o sandwich norm

8'0

Max of Activation Norms

Errors

0.8

0.7

1.0

20

225

200

4

3

0.6

Discretization

0.6

0.5

0.8

0.6

Curvatures

FID

→

15

Flag-DiT

Next-DiT

SiT

150

175

0.4

0.4

10

125

2

0.2

0.3

0.4

100

1

0.2

0.2

75

0

0

5

10

Layer

15

20

25

30

0.0

0.1

0.0

0.2

0.4

Time

0.6

0.8

1.0

Step

50

to an unstable training process but also deteriorate the sampling process. Particularly for resolution extrapolation, small errors in the early layers or sampling steps are amplified in subsequent layers or steps. These compounding errors gradually cause extremely out-of-distribution inputs for the network, leading to failure in generated samples.

Therefore, we propose a simple yet effective approach that adds RMSNorm both before and after each attention and mlp layer. The modified architecture is equivalent to using the sandwich normalization [25] in transformer blocks. Importantly, both pre-norm and post-norm are placed before the scale operation in AdaLN-Zero to prevent normalizing an all-zero tensor at the start of training, which is caused by the zero-initialized scale parameter in AdaLN-Zero. However, after the scale operation, the activation magnitudes still cannot be preserved. We further add a tanh gating to the second scale prediction after post-norm to prevent extremely large modulation values from contributing to the residual branch. With all these efforts including the original QK-Norm in Flag-DiT, we effectively control both then input and output activation magnitudes of each attention and mlp layer, as validated in Figure 3.

Experiments on ImageNet To quantitatively assess the effects of Next-DiT with the above improvements, we conduct experiments on the label-conditional ImageNet-256 benchmark. We follow the training setups and evaluation protocols of SiT [59] and Flag-DiT [35]. As depicted in Figure 5, Next-DiT converges significantly faster than both Flag-DiT and SiT evaluated by FID and Inception Score (IS). This observation confirms that the improved design of Next-DiT can enhance its fundamental generative capabilities. Additionally, we ablate the effectiveness of long-skip connections[6] designed to provide shortcuts for low-level features. Unfortunately, long-skip connections lead to training instability and significantly worsen the performance of Next-DiT. Therefore, Next-DiT chooses to not incorporate long-skip connection.

2.2 Improving NTK-Aware Scaled RoPE with Frequency- and Time-Awareness

In the large language models (LLMs) community, tuning-free or few-shot tuning length extrapolation is a popular focus currently and has been thoroughly analyzed, e.g. , position interpolation [18], NTK-Aware Scaled RoPE [1], YaRN [64], etc . In Lumina-T2X, due to the adoption of 1D RoPE same as LLMs, NTK-Aware Scaled RoPE has been directly applied to achieve a certain degree of resolution extrapolation. In Lumina-Next, we further investigate the effects of different length extrapolation methods on 3D RoPE for the first time. Based on our findings, we introduce frequencyand time-awareness grounded in NTK-Aware Scaled RoPE, resulting in novel Frequency-Aware Scaled RoPE and Time-Aware Scaled RoPE targeted to the visual generation task. In Figure 6 (a), we demonstrate the wavelength of each dimension of the RoPE embedding under different extrapolation techniques with a toy setting of b = 5 , h head = 24 , and T = H = W = 16 .

Revisiting NTK-Aware Scaled RoPE 3D RoPE encodes position information of each axis using a frequency matrix Θ = Diag ( θ 1 , · · · , θ d , · · · , θ d head / 6 ) with θ d = b -6 d/d head , where b is the rotary base. When we want to perform a s times resolution extrapolation, the most straightforward way is encoding unseen positions with no change of RoPE, known as Position Extrapolation . However, these unseen positions will directly confuse the model's spatial understanding, leading to the generation of unreasonable or repetitive content (refer to Figure 6 (c)).

(g) Time-aware

Scaled RoPE

(f) Frequency-aware

Scaled RoPE

(e) NTK-aware

Scaled RoPE

(d) Position Interpolation

(c) Position Extrapolation

(b) Original Resolution

(1K Inference)

(a) Wavelength of the RoPE embeddings

under different strategies

Extrapolation

Interpolation

NTK-aware.

Frequency-aware.

Time-aware.

Wavelength

40

Dimension Index

35

20

24

In contrast, another straightforward approach, Position Interpolation [18], involves linearly rescaling the frequency matrix as θ inter d = θ d · s , thereby bringing all position embeddings back into the training value domain. However, as noted by [1], this linear interpolation makes it difficult for RoPE to distinguish between positions of tokens that are very close to each other. The same conclusion exists in the diffusion transformer - as shown in Figure 6 (d), the global structure of the image is maintained at the cost of losing all local details.

To address these issues, NTK-Aware Scaled RoPE [1] is proposed to achieve training-free length extrapolation for LLMs. Particularly, it scales the rotary base as b ′ = b · s and then calculates θ ntk d = b ′-6 d/d head such that the lowest frequency term of RoPE is equivalent to performing position interpolation, allowing a gradual transition from position extrapolation of high-frequency terms to position interpolation of low-frequency ones. In Lumina-T2X [35], NTK-Aware Scaled RoPE has been shown to seamlessly integrate with diffusion transformers to achieve resolution extrapolation to a certain extent. However, failure cases still occur, such as the repetition issue shown in Figure 6 (e), indicating that the model still struggles with spatial awareness during resolution extrapolation.

Frequency-Aware Scaled RoPE Rethinking the limitations of NTK-Aware Scaled RoPE, we believe that performing interpolation on the lowest frequency component is a suboptimal design. Considering that RoPE uses periodic functions with frequencies Θ to encode relative positions, many dimensions encounter unseen repeated cycles during resolution extrapolation, which in turn causes the model to output repetitive content. This echoes the statement of YaRN [64] that position embedding should be divided into parts using wavelength as the boundary and treated differently. To solve this problem, we first identify the dimension with the wavelength equivalent to the training sequence length as d target = d head · log b ( L 2 π ) , where L is the maximum training sequence length. After that, a scaled base with frequency-awareness can be formulated by b ′ = b · s head d target , which enable the component at d target equivalent to position interpolation. Note that d target may not be an integer, but we do not need to use it as an index so that we can use its exact value. Additionally, to prevent the part where d > d target from being excessively interpolated, we take the larger value comparing with the position interpolation result as θ freq d = max ( b ′-6 d/d head , b -6 d/d head · s ) . This Frequency-Aware Scaled RoPE further addresses the issue of content repetition, as shown in Figure 6 (f).

d

Time-aware Scaled RoPE Observing Figure 6, we can conclude that: (1) while position interpolation confuses local details, it effectively ensures a reasonable global structure, and (2) while NTK-Aware Scaled RoPE has content repetition, it retains good local details. By combining these observations with the time-conditioned feature of diffusion models, we can achieve a more targeted resolution extrapolation approach. A common understanding of diffusion models is that they first restore global concepts before local details [19]. This motivates us to design a time-aware strategy to benefit from different approaches at different denoising steps - using position interpolation in the early stages of denoising to ensure the overall structure and gradually shifting to NTK-Aware Scaled RoPE to preserve local details. This echoes DemoFusion's [29] strategy of gradually reducing global guidance over the denoising process, allowing the model to focus on adding local details. Specifically, the previously proposed Frequency-Aware Scaled RoPE is naturally adaptable to this progressive shift. When d target = 1 , it is equivalent to position interpolation; when d target = d head, it is equivalent to NTK-Aware Scaled RoPE. Therefore, we design a coefficient d t

Sigmoid

Rational

Uniform

10 Steps

30 Steps

10 Steps

30 Steps

10 Steps

30 Steps

10 Steps

30 Steps

that varies with time t in the most direct way, making the frequency base time-conditioned, i.e. , b ′ t = b · s d head d t , where d t = ( d head -1) · t +1 . Then the time-aware frequency base can be expressed as θ time d,t = max ( b ′ -6 d/d head t , b -6 d/d head · s ) . This Time-Aware Scaled RoPE makes globally consistent high-detail resolution extrapolation possible, as shown in Figure 6 (g). In the following experiments on resolution extrapolation, we use Time-Aware Scaled RoPE for the best tuning-free performance.

2.3 Improving Sampling Efficiency with Fewer and Faster Steps

Different from one-step generative models, the sampling of diffusion and flow models involves an iterative denoising process. The overall time complexity of sampling can be written as N × T , which is controlled by the number of function evaluations N and the inference time of a single function evaluation T . Therefore, to improve the efficiency during inference, we propose corresponding strategies for each factor to reduce N and T respectively.

Fewer Sampling Steps with Optimized Time Schedule and Higher-Order Solvers Both diffusion models and flow-based models numerically solve an SDE or ODE within the interval [ t min , t max ] during sampling. A common approach is to discretize the continuous time interval into multiple sub-intervals and use Euler's method with O ( h 2 ) discretization error with respect to step size h or other advanced solvers. Given a constant budget of N sampling steps, the time discretization strategy { t 0 , t 1 , ..., t N } , or time schedule, determines the grid of timesteps on which the network is evaluated. The resulting accumulated truncation errors can greatly impact the quality of generated samples, especially when N is small. Various time schedules have been explored for diffusion models, such as Cosine [61], Linear [74], and EDM [42]. However, most flow-based models still adopt the simplest uniform time intervals and first-order Euler's method for sampling, resulting in poor-quality images in low-resource scenarios, as shown in Figure 7.

Therefore, we provide a detailed analysis of truncation errors at different timesteps using Euler's method to solve the Flow ODE for text-to-image generation. We begin by setting the anchor sampling steps, N = 50 , with uniform intervals for evaluation. For each timestep t i , the sample produced by one-step Euler's method is defined as ˆ x t i +1 = x t i +( t i +1 -t i ) v θ ( x t i , t i ) . To approximate the local truncation error τ t i at time t i , we estimate the ground truth sample x t i +1 by running a large number of Euler steps from t i to t i +1 with uniform intervals and compute the discretization error as τ t i = x t i +1 -ˆ x t i +1 . In addition, since the flow model builds linear interpolation between noise and data and predicts the constant velocity field, it is natural to use the curvature to describe local errors on the trajectory, i.e. , the straighter the trajectory, the smaller the discretization error. Following [60], we compute the second-order derivative to estimate the local curvature of each step as κ t i = x t i -( x t i +1 + x t i -1 ) × 0 . 5 . The average L2-norm of discretization error τ t i and curvature κ t i are visualized in Figure 4. Our analysis reveals that discretization errors are much larger when t is near pure noise ( t = 0 ) and are relatively larger near clean data ( t = 1 ), compared to intermediate timesteps. This observation contradicts the findings in diffusion models, where EDM [42] discover

the discretization errors monotonically increase as samples get closer to clean data. Hence, diffusion schedules like EDM are not ideal for flow-based models.

To optimize the time schedules for sampling, we propose two parametric functions to map t ∈ [0 , 1] :

These time schedules are tailored for flow-based models. The first, a rational function parametrized by σ , maps all timesteps to smaller values, minimizing the discretization errors near pure noise. This schedule reduces to the uniform time schedule when σ = 1 and is also used in prior works [35, 31, 38] for high-resolution image generation. The second schedule, a piecewise sigmoid-like function, ensures that the step sizes at the start and end of sampling are larger than those at intermediate steps. The central point is controlled by µ , while the weights on lower and higher timesteps are adjusted by α and β , respectively. As shown in Figure 7, both rational and sigmoid schedules improve few-step sampling performance compared to all-black or all-white images generated by the uniform schedule. This highlights the importance of the early stage of sampling in creating the main subject in the image. The sigmoid schedule generates images with better details and is adopted for the remainder of this paper with µ = 0 . 6 , α = 6 , and β = 20 .

Notably, this optimized schedule requires no extra computation but a free launch for fast sampling using our flow model. It is also compatible with advanced solvers applicable to flow models, which can unleash its full potential. However, limited work has explored the use of higher-order solvers for solving the Flow ODE. Unlike the Probability Flow ODE [75] in diffusion models, which has a semi-linear structure, the Flow ODE derived from the continuity equation is in a simpler form of ˙ x = v θ ( x t , t ) . This simplicity allows for the direct use of existing higher-order ODE solvers, such as the traditional explicit Runge-Kutta (RK) methods in the form of x t = x s + ∫ t s v θ ( x τ , τ )d τ , where the n -th order solver ensembles n intermediate steps between [ s, t ] to reduce the approximation errors. During sampling, explicit RK family solvers also need to specify the timesteps { t 0 , t 1 , ..., t N } in advance and adopt the uniform intervals as default. Thus, our improved schedules, such as the rational or sigmoid schedule, can be directly applied to these higher-order solvers. Algorithm B.1 illustrates the pseudocode for combining the midpoint method and sigmoid schedule for sampling.

Faster Network Evaluation with Time-Aware Context Drop After reducing the denoising steps during sampling, our next goal is to optimize the network evaluation time T for each single step, which is constrained by the quadratic complexity of attention blocks. Among various inferencetime acceleration methods, Token Merging (ToMe) [9] is a simple yet effective approach that merges similarly visual tokens to reduce redundancy in attention blocks, thereby speeding up inference for image classification. Although ToMe has been successfully applied to diffusion U-Net [10, 73], we find that a straightforward application of ToMe to Next-DiT, a purely transformer-based diffusion model, completely fails to produce satisfactory images. We hypothesize that performing the original token merging in all transformer layers across all diffusion steps is clearly unsuitable for the task of generative modeling with diffusion transformers.

To bridge this gap, we propose Time-Aware Context Drop, an optimized method to drop redundant visual tokens during inference for Next-DiT. First, the original bipartite soft matching in ToMe not only introduces additional complexity but also contradicts the design of 3D RoPE in Next-DiT. Therefore, we replace this complex token partitioning strategy in ToMe with a simple average pooling for downsampling, aligning with the spatial prior introduced in 3D RoPE. Similar to [73], we find that applying the merge-and-unmerge operation to all queries, keys, and values greatly undermines the quality of generated images as this strategy is designed for visual recognition. To preserve the complete visual content, we only perform average pooling on keys and values for downsampling. Finally, considering the time-awareness in diffusion models, we perform token dropping for all layers at t = 0 to enhance efficiency and gradually transform to no token dropping at t = 1 to maintain visual quality. The Context Drop without time-awareness is similar to the key-value (KV) token compression in PixArtΣ [14], which merges KV tokens in spatial space using 2 × 2 convolution to reduce complexity. However, unlike their fine-tuning methods, Context Drop is a training-free technique with an adaptive compression rate that aligns with the diffusion time schedule. In Appendix C.1, We demonstrate samples generated by our methods, which reduce inference time while achieving comparable visual quality.

SDXL

Lumina-Next

Lumina-T2I

PixArt-

𝜶

Lumina-Next

(1K)

MultiDiffusion

DemoFusion

ScaleCrafter

3 Lumina-Next by Composing Everything

By composing all improvements, we finally reached Lumina-Next, a powerful generative framework for various modalities with improved efficiency. We first instantiate Lumina-Next for text-to-image generation, which is stronger and faster than Lumina-T2X. Besides, compared with SDXL and PixArtα , it demonstrates preliminary supports of multilingual prompts (refer to Appendix C.2), better visual quality with smaller sampling steps, and more flexibility at synthesizing images with any resolution. After this, we further verified that the Lumina-Next framework also exhibits excellent performance in any-resolution recognition (refer to Appendix D) and multi-view images, music, audio, point clouds generation (refer to Appendix E).

3.1 Flexible Resolution Extrapolation

Failure of SDXL and PixArtα SDXL [65] and PixArtα [15] are representative methods of U-Net and transformer-based diffusion models, respectively. Despite their ability to generate impressive images, they fail to produce images beyond the training resolution, resulting in repetitive and unreasonable content at higher resolutions, as shown in Figure 8. This is because the receptive field of U-Net and the absolute position embedding of transformers are significantly biased towards the training resolution or sequence length.

Comparison with Tuning-free Resolution Extrapolation Methods Due to the enormous cost of directly training high-resolution (greater than 2K) generative models, researchers have sought to enable pre-trained diffusion models to infer beyond the training resolution in a tuning-free manner. MultiDiffusion [7] is the first attempt to generate high-resolution panoramas via sampling overlapped sliding windows at the original resolution. DemoFusion [29] further utilizes the generated

5 Steps

10 Steps

5 Steps

10 Steps

5 Steps

10 Steps

5 Steps

10 Steps

Lumina-Next

+Midpoint Solver

PixArt-

α

+DPM-Solver

SDXL

+DPM-Solver

results at training resolution as global guidance, achieving object-oriented high-resolution image generation. Meanwhile, ScaleCrafter [37] adopts dilated convolution to achieve resolution extrapolation tiled for U-Net based diffusion models ( e.g. , SDXL [65]).

In Figure 8, we compare Lumina-Next with the methods above on the 2K generation task. We can conclude that MultiDiffusion, proposed for panorama generation, lacks global awareness, leading to the issue of repetitive content still being present. DemoFusion and ScaleCrafter work well, but DemoFusion requires multiple runs, resulting in a non-negligible inference budget, and ScaleCrafter is built upon dilated convolution, which is unsuitable for modern transformer-based diffusion models. In contrast, Lumina-Next inherently supports a direct resolution extrapolation with the carefully designed Next-DiT architecture and frequency- and time-aware scaled RoPE at inference time, offering a more fundamental solution in the era of diffusion transformer.

On the Importance of Next-DiT for Resolution Extrapolation Regarding Figure 8, we particularly emphasize that Lumina-Next's 2K extrapolation performance significantly surpasses that of Lumina-T2X. In addition to the newly proposed time-aware scaled RoPE, we attribute this to the advanced design of Next-DiT: (1) 3D RoPE enables independent modelling of the t , h , w dimensions, decoupling extrapolation in three dimensions so that they do not affect each other, and (2) sandwich norm stabilizes the output magnitude of each transformer block, reducing the model's sensitivity to sequence length. Composing them together, Lumina-Next possesses excellent resolution extrapolation capabilities without the need for any special inference algorithms like DemoFusion [29].

3.2 Few-Step Text-to-Image Generation

We demonstrate how our improved sampling schedules together with higher-order ODE solvers can significantly boost the sampling efficiency for flow models. Specifically, we employ the midpoint solver, a second-order Runge-Kutta method, combined with our sigmoid schedule following Algorithm B.1. For comparison, we choose open-source text-to-image models, including SDXL [65] and PixArtα [15] equipped with DPM-Solver [53, 54]. Figure 9 shows Lumina-Next using the midpoint solver significantly improves the sample quality in 10-20 number of function evaluations (NFEs), consistently achieving better conditional sampling performance compared to PixArtα and SDXL using DPM-Solver. Switching to higher-order solvers can further enhance few-step sampling performance with the cost of linearly increasing NFEs, presenting a speed-quality trade-off.

4 Conclusion

In this paper, we introduce Lumina-Next, which successfully addressed the limitations of its predecessor, Lumina-T2X. With the improved Next-DiT architecture, context extrapolation together, and fast sampling techniques tailored for our flow-based diffusion transformers, Lumina-Next showcases strong generation capabilities, such as generating high-quality images significantly larger than its training resolution and multilingual text-to-image generation. Remarkably, these results are all produced in a training-free manner and outperform previous methods including SDXL and PixArtα . Weextend Lumina-Next's versatility to other modalities, such as multiviews, audio, music, and point clouds, with minimal modifications, achieving superior results across these diverse applications.

References

Ntk-aware Scaled Rope Allows Llama Models to Have Extended (8k+) Context Size Without Any Fine-tuning and Minimal Perplexity Degradation. https: //www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_ allows_llama_models_to_have/ , 2024. Accessed: 2024-4-10.

Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International Conference on Machine Learning , pages 40-49. PMLR, 2018.

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325 , 2023.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 4895-4901, 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22669-22679, 2023.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference on Machine Learning , pages 1737-1752, 2023.

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8, 2023.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations , 2022.

Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4598-4602, 2023.

Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. arXiv preprint arXiv:2008.06520 , 2020.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 , 2024.

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], 2015.

Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixartσ : Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692 , 2024.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixartα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 , 2023.

Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1206-1210. IEEE, 2024.

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 , 2023.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11472-11481, 2022.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and others. Scaling instructionfinetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation, 2023.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch npack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems , 36, 2024.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13142-13153, 2023.

Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems , 34:19822-19835, 2021.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems , 35:30150-30166, 2022.

SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372 , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high-resolution image generation with no $$$. In CVPR , 2024.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. 2022.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331 , 2023.

Seth Forsgren and Hayk Martiros. Riffusion-stable diffusion for real-time music generation. URL https://riffusion. com , 2022.

Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935 , 2024.

Peng Gao, Le Zhuo, Ziyi Lin, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 , 2024.

Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-toaudio generation using instruction-tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731 , 2023.

Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representations , 2023.

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning , pages 1321313232. PMLR, 2023.

Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474 , 2023.

Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403.12963 , 2024.

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models, 2023.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually Guided Audio Generation.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 , 2022.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models.

Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308.05734 , 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9298-9309, 2023.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. arXiv preprint arXiv:2312.08754 , 2023.

Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 , 2023.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376 , 2024.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2306.07279 , 2023.

Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems , 36, 2024.

Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740 , 2024.

Bao Nguyen, Binh Nguyen, and Viet Anh Nguyen. Bellman optimal step-size straightening of flow-matching models. arXiv preprint arXiv:2312.16414 , 2023.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

OpenAI. https://openai.com/sora. In OpenAI blog , 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071 , 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Text-to-music generation with long-context latent diffusion, 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 , 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multiview diffusion for 3d generation. arXiv preprint arXiv:2308.16512 , 2023.

Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE International Conference on Computer Vision , pages 3859-3868, 2019.

Ethan Smith, Nayan Saxena, and Aninda Saha. Todo: Token downsampling for efficient generation of high-resolution images. arXiv preprint arXiv:2402.13573 , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

Paul Taylor. Text-to-speech synthesis . Cambridge university press, 2009.

Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22721-22731, 2023.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Fu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint arXiv:2405.18407 , 2024.

Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 , 2023.

Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.05034 , 2024.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete Diffusion Model for Text-to-sound Generation.

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In IEEE International Conference on Computer Vision , pages 4541-4550, 2019.

scale

LN

N

RoPE

RMS

Predicted Velocity

Linear Proj.

or

Proj.

LN

Proj.

Proj.

Proj.

scale & shift

Feedforward

scale

Upgrade From Flag-DiT

To Next-DiT

Label

Embedding

Time

Embedding

RMS

RMS

LN

N

2D

RoPE

RMS

Noisy input

Predicted Velocity

Linear Proj.

or

scale

Proj.

Proj.

scale

Feedforward

Dog

Patch Embed

RMS

QKV Proj.

Label

Embedding

Time

Embedding

Dog

Patch Embed

RMS

scale & shift

QKV Proj.

Proj.

Proj.

LN

Noisy input

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

Next-line token

Prediction head

Prediction head

tanh

tanh

scale

scale

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

\n

LN

: LayerNorm

RMS

: RMS Norm

A Broader Impacts and Limitations

A.1 Broader Impacts

In this work, we propose a fundamental framework for generative modeling. On the positive side, the open-source of our code and checkpoints will foster transparency as well as innovations in the generative AI field. On the negative side, our text-to-image generation model, for example, may be used to generate images with potential biases and malicious information, similar to any generative AI tool. However, we have designed and used a filtering pipeline to filter out all malicious content in the training dataset.

A.2 Limiations

In this work, we demonstrate the superior high-resolution and multilingual text-to-image generation capabilities of Lumina-Next compared to open-source models like SDXL and PixArtα . However, our text-to-image generation model still fall short of state-of-the-art closed-source models like Midjourney and DALLE 3 in terms of text-image alignment and visual aesthetics. A notable gap lies in the size of text-image pairs used for multi-stage training. Although we expand the size of our training dataset to 20 million, it remains considerably smaller than the datsets used by these closedsource models. In addition, fine-tuning models with human preference data using techniques like Direct Preference Optimization is also important to improve image quality. We plan to address these challenges in future research.

B Additional Details Lumina-Next

B.1 Illustration of Next-DiT's Architecture Improvements

In Figure 10, we illustrate the structural difference between Flag-DiT and Next-DiT. The main improvements include 3D RoPE, sandwich normalization, group-query attention, etc . Specifically, We leverage grouped-query attention [4] in transformer blocks, which is an interpolation between multihead and multi-query attention to improve inference speed as well as reduce memory consumption. For instance, our 2B model divides 32 query heads into 8 groups, resulting in 8 shared key-value heads. This modification reduces the parameter size while achieving similar performance compared to the original multi-head attention.

B.2 Sampling with Optimized Schedule and Higher-Order ODE Solver

In the main text, we propose two optimized time schedules and combine them with higher-order ODEsolvers for few-step sampling. Algorithm B.1 illustrates the pseudocode for using the midpoint method with the optimized schedule.

B.3 Improved Image-Text Pairs

Lumina-T2I constructs 14M high-aesthetic synthetic text-image pairs as the training dataset. While the limited data size leads to fast convergence, these data only cover a small proportion of real image distribution, which contains diverse contents and styles. Besides, the quality of text captions in this dataset is imbalanced, consisting of incomplete or inaccurate image descriptions. As pointed out in recent works [8, 14], improving the diversity and overall quality of text prompts can greatly boost text-image alignment. Therefore, it is important to re-examine the construction of training datasets.

LAION Augmented High-Quality Images We first expand the size of our training dataset from 14M to 20M using an additional 6M text-image pairs from the LAION [69] dataset. To efficiently filter high-quality images from large-scale internet datasets, we employ a two-stage data filtering pipeline:

In the first stage, we aim to obtain a coarse-grained candidate pool from massive images. Specifically, we first filter out all images with resolutions lower than 512 × 512 . Then, we further filter out images with low aesthetic scores ( < 5 . 5 ), high watermark probabilities ( > 0 . 5 ), and high unsafe probabilities ( > 0 . 5 ). With these filtering thresholds, we construct an image pool with approximately 30 million items. In the second stage, we introduce the overall image quality rank as a fine-grained filtering metric to further obtain a subset with superior overall quality. Concretely, we rank the 30 million coarse-filtered images based on aesthetic score, watermark probability, and safety probability, respectively. Then we sum the three types of ranks into an overall image quality rank, and the top-ranking 6 million images are selected to form our final high-quality image dataset. Upon manual evaluation, the 6 million image data filtered by our two-stage pipeline exhibit high resolution, high aesthetic appeal, and minimal probabilities of watermarking and unsafe content.

Mixture of Captioners

GPT-4V

BLIP2

a cup of coffee and a piece of cake on a table

Sharegpt4v

In the tranquil setting of a park, a wooden picnic

table serves as the stage for a simple yet inviting

scene. On the table, a slice of chocolate cake rests

on a white plate, its rich, dark color contrasting

with the plate's pristine whiteness. A silver fork

lies next to the cake, ready to dive into the sweet

treat.

Beside the cake, a white cup cradles a hot beverage,

steam gently rising from it, hinting at the warmth

within. A spoon rests on the saucer, its polished

surface reflecting the soft light of the day.

A black bag is casually placed on the table, its

contents unknown but its presence adding to the

charm of the scene. A book lies open nearby, its

pages whispering tales of adventure and knowledge.

The table is set against a backdrop of lush greenery,

with trees standing tall and proud, their leaves

rustling softly in the breeze. A path meanders

through the trees in the background, inviting

onlookers to take a leisurely stroll and enjoy the

serene

Cogvlm

A slice of chocolate cake with a rich, moist texture

sits on a white plate, accompanied by a cup of hot

chocolate and a spoon, all resting on a wooden table

with a book and a black bag. The setting is outdoors,

under a clear sky, suggesting a leisurely day in a

tranquil outdoor cafe.

LLaVA-next

The image captures a tranquil outdoor setting, likely a café or a picnic area. The main focus is a wooden table, which is adorned with a variety of items, each telling a story of its own.

On the table, there's a white plate holding a slice of cake. The cake appears to be a rich, dark brown color, suggesting it might be a chocolate cake. The slice is neatly cut, revealing the

moist interior.

Beside the cake, there's a cup of coffee. The coffee is a light brown color, indicating it might be a latte or a cappuccino. The cup is white, which contrasts nicely with the darker color of

the coffee.

Next to the coffee cup, there's a black bag. The bag is closed, and it's not clear what's inside. It could be a purse, a laptop bag, or a camera bag.

In the background, there's a green umbrella. The umbrella is closed and leaning against the wall, ready to provide shade on a sunny day.

The table is set on a wooden deck, which is surrounded by lush greenery. The deck is made of wooden planks, adding to the rustic charm of the setting.

Overall, the image paints a picture of a peaceful moment, perhaps during a break from work or a leisurely afternoon spent enjoying a slice of cake and a cup of coffee in the company

of nature.

SPHINX-X

The image features a wooden dining table

with a variety of items on it. There is a

partially eaten slice of cake on a plate,

accompanied by a cup of coffee. A knife is

placed next to the cake, and a spoon is also

present on the table.

In addition to the food and utensils, there

are several books scattered around the

table, indicating a relaxed and comfortable

atmosphere. A backpack can be seen on the

table, suggesting that someone might have

brought these items to enjoy a meal or a

break.

Role 1: Concise elements:

* Outdoor wooden table

* Piece of cake on a white plate

* Cappuccino in a white cup with

saucer

* Fork and spoon

* Sugar packet

* Mobile phone

* Wallet

* Black camera bag

* Sunglasses

* Leaflets

* Wood-chip ground

* Outdoor chairs

* Trees

* Outdoor umbrellas

Role 2: Detailed description of nine regions:

left: In the left section, there is a wooden table surface with a corner of a white plate

visible, displaying a piece of cake. A black camera bag sits to the left of the plate, and

behind it, partially out of the frame, are outdoor chairs and trees, suggesting a garden

or café setting.

right: The right section shows a white cup of cappuccino on a saucer placed on the

right side of the wooden table, next to which is a mobile phone, a wallet, and a pair of

sunglasses. In the blurred background, outdoor chairs and umbrellas indicate a public

outdoor space.

upper: This section captures the upper part of the scene with leaflets and a sugar

packet lying on the table, alongside a black camera bag. The background includes

outdoor umbrellas and foliage.

lower: The lower section includes the front of the table where a piece of cake on a

white plate is visible along with silverware. Just below the table's edge, the ground is

covered in wood chips.

central: The central area of the image has a piece of cake on a white plate, a cup of

cappuccino on a saucer, a mobile phone, and a wallet placed on a wooden table. A sugar

packet lies near the cup.

top-left: The top-left includes a corner of a black camera bag and leaflets on the

wooden table, with green trees and white outdoor umbrellas in the soft-focus

background.

top-right: This area shows a white cup of cappuccino on a saucer, a mobile phone, and a

pair of sunglasses. The background softly blurs into the colors of chairs and umbrellas.

bottom-left: The bottom-left section displays a piece of cake on a white plate, with

silverware on the side. The wooden table surface and wood-chip ground are also visible.

bottom-right: In the bottom-right, there is a close-up view of the white saucer with a

cappuccino cup, alongside a mobile phone and a wallet, set upon the wooden table. Below

the table is the wood-chip ground.

Role 3: Detailed description of the whole image:

The image captures a personal moment during a daytime outing at an outdoor café or garden. The focus is on a wooden table

which holds the remnants of a casual meal or snack: a piece of cake on a white plate with a fork and a spoon, and a

cappuccino in a white cup and saucer. Personal items including a mobile phone, wallet, sunglasses, and a black camera bag are

scattered on the table. Leaflets, possibly menus or flyers, and a single sugar packet lie next to the camera bag. The

background, although out of focus, indicates a leisurely environment with other tables, chairs, and umbrellas suggesting

other patrons in the vicinity. Trees provide a natural backdrop to the setting. The ground is covered with wood chips,

further reinforcing the casual, rustic ambiance of the place. The light and shadows suggest it is a bright, sunny day,

perfect for enjoying an outdoor setting.

Role 4: Concise description of

the whole image:

An outdoor café scene featuring a

casual snack of cake and

cappuccino on a wooden table,

surrounded by personal items, with

a blurred background of trees,

chairs, and umbrellas.

(a) Generated Samples

(b) Inference Time

80

Baseline

Flash Attention

74.1

70

Flash Attention + Context Drop

60

Inference Time(s)

50

50.3

45.7

30

22.5

20

10

11.6

10.2

OOM! OOM!

0

1024x1024

2048x2048

Resolution

Mixture-of-Captioners Although PixArtσ [14] and DALLE-3 [8] highlight the importance of improving the quality of text descriptions, they only leverage a single captioner to enhance caption accuracy. As illustrated in Figure 11, we compare captions generated from open-sourced VisionLanguage Models (VLMs) such as BLIP2 [44], LLaVa [48], SPHINX [34], and Share-GPT4V [17]. We observe different VLMs describe different information of the given image in different styles at different visual granularities. Moreover, these generated captions are complementary to each other since none of them achieves 100% captioning accuracy without any hallucination. Motivated by this observation, we propose a novel Mixture-of-Captioners (MoC) approach which employs multiple pretrained VLMs to generate mutual complementary captions. Besides, we collect a high-quality and high-resolution dataset consisting of 100K images and leverage GPT-4V to perform multi-facet holistic descriptions over these high-resolution images, depicted in Figure 11. After re-captioning using our proposed MoC, each image is paired with a caption pool with multiple descriptions from various models. During the training stage, we randomly sample one caption from collected MoC caption pools. This also resembles a kind of data augmentation that enhances model robustness during inference with all types of user inputs.

Modern Standard Arabic

:

ﻓﺘﺮة إﯾﺪو ،

Russian

:

Токийская башня,

высококачественное

Укиё-э, период Эдо

Chinese

:

东京塔，

江户时代

Japanese

:

東京タワー、

最⾼品質の浮世絵、

江戸時代

Korean

:

도쿄

타워

,

최고

품질의

우키요에

,

에도

시대

Vietnamese

chất lượng Ukiyo-e tốt nhất,

thời kỳ Edo

Ukrainian

:  Токійська вежа,

найякісніше Укійо-е,

період Едо

Thai

:

โตเยวทาวเวอ

,

โยะเอะณภาพด

,

สยเอโดะ

Persian (Iranian) :

ﺑﺮج ﺗﻮﮐﯿﻮ، اوﮐﯿﻮ-اِ ﺑﺎ ﺑﺎﻻﺗﺮﯾﻦ

ﮐﯿﻔﯿﺖ، دوره ادو

Polish

:  Wieża Tokijska,

najwyższa jakość Ukiyo-e,

okres Edo

Marathi

:

टो  ो

टॉवर

,

उ  ृ

गुणव  ा

उ  कयो

-

ए

,

एदो

कालावधी

Kurdish

Ukrîyo-e bi bilindî başi,

Turkish

en kaliteli Ukiyo-e,

Edo dönemi

English

:  Tokyo Tower,

finest quality Ukiyo-e,

Edo period

PixArt-

Lumina-Next

SDXL

PixArt-

Lumina-Next

SDXL

C More Text to Image Generation Results

C.1 Results of Time-Aware Context Drop

As shown in Figure 12 (a), Next-DiT using the proposed Time-Aware Context Drop with a 75% downsampling ratio generates comparable and even better 2K images compared to the baseline. The inference time comparison in Figure 12 (b) highlights that our Context Drop can further enhance inference speed when combined with advanced inference techniques for transformers, such as Flash Attention [22], which is more significant for higher-resolution image generation. Notably, our Context Drop is similar to masked attention or dilated convolution, which has been proven to be the key of resolution extrapolation in diffusion U-Net [37, 40]. Hence, this high-level resemblance in our diffusion transformer architecture may explain why Context Drop can reduce repetitive and unreasonable artifacts in high-resolution image generation, presenting an interesting avenue for future research.

C.2 Results of Zero-shot Multilingual Generation

Previous T2I models, such as Imagen [67], PixArtα [15] and Stable Diffusion [31, 65], utilized CLIP and T5 as text encoder for generating high aesthetics images. Such choices took advantage of pretrained text encoders or multimodal aligned text encoders as good representations for textconditional image synthesis. However, employing T5 and CLIP as text embedding shows limited

In the sky above, a giant, whimsical

cloud shaped like the

😊

a soft, golden light over the scene

👧

with

🌹

in the

❄

☃

with

🌹

in the

❄

A small

🍎

and

🍊

with

😁

emoji in the Sahara desert

👽🤖👹👻

🐔

playing

🏀

🐶

wearing

😎

flying on

🌈

PixArt-

Lumina-Next

SDXL

effectiveness in handling multilingual prompts. Different from those approaches, the Lumina series adopts decoder-only LLMs as text encoders. For instance, Lumina-T2X uses a 7 billion LLaMa as text encoder, while Lumina-Next employs the smaller Gemma-2B to decrease GPU memory costs and increase training throughput. Remarkably, both LLaMa-7B and Gemma-2B endow Lumina T2I models with the ability to understand multilingual prompts in a zero-shot manner, despite being primarily pretrained on English-only corpora. As shown in Figure 13, our T2I models not only accurately interpret multilingual prompts but also generate images with cultural nuances and show a preliminary ability to understand emojis, even though it was mainly trained with English-only image-text pairs. Compared to SDXL and PixArtα , which use CLIP and T5, there is a significant improvement in the understanding of multilingual prompts. We argue that decoder-only LLMs with strong language understanding abilities better align multilingual semantics than the encoder-decoder architecture of T5 or the multimodal alignment of CLIP text embeddings. To further explore the relationships between multilingual generation abilities and text encoders, we train Lumina-Next with different open-source LLMs, including Gemma-2B, Qwen-1.8B [5], and InternLM-7B [12]. As demonstrated in Figure 15, Lumina-Next with Qwen-1.8B and InternLM-7B showcase significantly better text-image alignment compared to Gemma-2B when using diverse Chinese prompts as inputs. Remarkably, Lumina-Next with Qwen-1.8B and InternLM-7B accurately captured the meanings and emotions of classic Chinese poems, reflecting these nuances in the generated images. This observation corresponds with the stronger multilingual abilities in Qwen-1.8B and InternLM-7B. Therefore, we argue that using more advanced LLMs as text encoders enhances text-to-image generation performance.

C.3 Resolution Extrapolation with Any Aspect-ratio

In Figure 16, we further compare the panorama ( 1024 × 4096 images) generation performance of the proposed Lumina-Next with MultiDiffusion [7] and DemoFusion [29] to highlight its any aspect ratio extrapolation ability. We found that although panorama generation is one of the primary goals of MultiDiffusion, its 'copy-paste' inference scheme easily leads to unreasonable content. DemoFusion adopts a progressive upscaling strategy, limiting its ability to generate arbitrary aspect ratios by the base model - in extreme aspect ratio scenarios, it also fails to produce satisfactory content. In contrast, Lumina-Next perfectly maintains the ability to generate at any aspect ratio under the training resolution, even when performing 4 × resolution extrapolation.

C.4 Generation with Long Prompt

Lumina-Next can handle image generation with long prompts. We give examples in Figure 17. As seen, with long prompts as inputs, Lumina-Next can produce high-quality images that strictly follow the details in the prompt. This ability remains when using other languages such as Chinese.

⼤漠孤烟直，

⻓河落⽇圆

孤⾈蓑笠翁

两只⻩鹂鸣翠柳

秋⻛起兮⽩云⻜，

⼀个安静的乡村⽊屋，

背景是⼀⽚⻘翠的⼭坡

⼀颗古⽼的橡树，

树⼲粗壮，

树枝向四周延展，

地⾯上覆盖着

厚厚的落叶

⼀只海⻳在热带

海底缓慢游动，

周围是五彩斑斓

的珊瑚和⻥群

Gemma-2B

InternLM-7B

Qwen-1.8B

MultiDiffusion

DemoFusion

Lumina-Next

D Any Resolution Recognition with Lumina-Next

Lumina-Next is not only a generative modeling framework for generating images with various resolutions but also a framework for recognizing images at any resolution. As illustrated in Figure 18, our Next-DiT can be seamlessly adapted into a robust backbone for image representation learning. By incorporating the sandwich normalization and 2D RoPE design into our architecture, Next-DiT achieves superior performance and faster convergence while maintaining computational efficiency compared to the original vision transformer in image recognition tasks. Furthermore, Next-DiT overcomes the limitations of previous vision transformers in processing images of varying resolutions, demonstrating remarkable generalization capabilities to handle input images with arbitrary resolutions and aspect ratios.

D.1 Pipelines

The unified framework of Lumina-Next for image generation and understanding at any resolution is depicted in Figure 18. Specifically, we introduce a learnable gate factor in the attention module, thereby avoiding the incorporation of time embeddings and label embeddings in image recognition tasks. We apply global average pooling (GAP) to all image tokens to obtain the class token, which is then fed into the classification head implemented by an MLP network. By integrating 2D RoPE during attention computation and incorporating RMSNorm before and after each attention and MLP layer, our Next-DiT architecture exhibits enhanced resolution extrapolation capabilities.

在⼴袤⽆垠的沙漠中，夜幕降临时，天际被⽆数璀璨的星星点缀 得如同⼀幅银⾊的华丽锦缎。⽉亮将温柔的⽉光洒在⾦⻩⾊的沙 丘上，投射出柔和的光影。顽强的树挺⽴在沙丘之间，静静地伫 ⽴在这寂静的夜⾊中。⼀个孤独的旅⼈缓缓⾏⾛，仿佛在追寻着 星⾠的指引。

A grand, medieval castle perched atop a rocky cliff, overlooking a vast, undulating landscape of forests and meadows. The castle's imposing stone towers and turrets rise majestically against a backdrop of a brilliant blue sky. A drawbridge spans a deep, mist-filled moat, and banners flap in the gentle breeze, displaying regal coats of arms.

A serene, coastal fishing village nestled along a rocky shoreline. Quaint, colorful houses with terracotta roofs line the narrow, cobblestone streets. Fishing boats bob gently in the harbor, their nets drying in the sea breeze. Seagulls circle overhead as the sun casts a golden glow over the calm, turquoise waters.

An elderly man sits comfortably in a cozy library, reading a thick, leather-bound book. He's wearing a grey cardigan over a white shirt, with round spectacles perched on his nose. His white hair is neatly combed back. Behind him, wooden bookshelves are filled with rows of books, some old and some new. A vintage lamp on a side table casts a warm light, illuminating his serene face. A classic grandfather clock ticks softly in the background.

A young girl plays joyfully with a small, fluffy white dog on a sandy beach. She's wearing a pink swimsuit with white polka dots and a matching headband. Her blonde hair is tied into pigtails. The girl and the dog run towards the gently crashing waves, leaving footprints behind them. In the distance, there are colorful beach umbrellas and people lounging on beach towels. The sky is clear and blue, with seagulls soaring above.

We employ a progressive training scheme with a fixed-resolution pre-training stage followed by an arbitrary-resolution fine-tuning stage, enabling Next-DiT to achieve robust image understanding across various resolutions.

Fixed Resolution Pre-training We first pre-train the Next-DiT architecture at a fixed resolution by cropping and resizing images to 224 × 224 , following the approach of the original ViT. This resolution balances the trade-off between training speed and input resolution, as higher resolutions require more FLOPs and reduced throughput. After pre-training, we directly evaluate our model on unseen resolutions to guide a subsequent fine-tuning stage with arbitrary resolutions.

Any Resolution Fine-tuning To enable our model to handle images with any resolution and aspect ratio, we employ an any-resolution fine-tuning stage. While previous works [23, 78] also utilize a multi-resolution training approach, they rely on a set of predefined input resolutions and randomly sample different resolutions for each input image from this set. These methods often alter the original resolution of aspect ratio of the input images a lot, potentially causing a performance drop, and they incur additional computational costs by resizing low-resolution images to higher resolutions unnecessarily. In contrast, we propose a dynamic padding scheme that allows efficient model training while preserving the original resolution and aspect ratio of the input images. This approach enables us to fully exploit the multi-resolution understanding capability of Next-DiT in a computationally efficient manner.

Dynamic Partitioning and Padding with Masked Attention Our aim is to retain the original resolution and aspect ratio of each input image to the fullest extent during the subsequent fine-

Dog

Any Res. Recognition

RMS

RMS

LN

N

RMS

Input

Linear Proj.

LN

Feedforward

Patch Embed

RMS

QKV Proj.

Label

Embedding

Time

Embedding

RMS

RMS

LN

N

2D

RoPE

RMS

Noisy input

Predicted Velocity

Linear Proj.

or

scale

Proj.

Proj.

scale

Feedforward

Dog

Patch Embed

RMS

QKV Proj.

Proj.

Proj.

LN

Prediction head

tanh

tanh

scale

scale

2D

RoPE

tanh

tanh

AVG

pool

Prediction head

LN

: LayerNorm

RMS

: RMS Norm

tuning stage. Additionally, we seek to ensure that the width and height of the input image are divisible by the patch size of the patch embed layer, enabling us to fully utilize the image information during the embedding process. To achieve this, we propose a dynamic partitioning and padding scheme to optimize the handling of image tokens instead of relying on a fixed resolution, as illustrated in Figure 19. Specifically, given constraints on the maximum number of patches N and the maximum aspect ratio R max , we define a set of candidate patch partitions C = { ( H p , W p ) | H p · W p ≤ N and max( H p ,W p ) min( H p ,W p ) ≤ R max } . For an input image of size ( H I , W I ) , we determine its optimal patch partition by calculating the matching ratio between the original input size and the target size, given by ( H ∗ p , W ∗ p ) = argmax ( H p ,W p ) ∈C ) min ( Hp HI , Wp WI ) max ( Hp HI , Wp WI ) . Subsequently, we resize the input image to ( H ∗ p · P, W ∗ p · P ) , where P denotes the patch size. This dynamic partitioning process leads to varying sequence lengths of patch tokens for each input image. To handle batches containing inputs with different token lengths, we pad all image token sequences to match the length of the longest sequence using a pad token. Additionally, we introduce attention masks within each attention module to prevent unwanted interactions between pad tokens and regular image tokens.

Flexible Resolution Inference By incorporating NTK-Aware Scaled RoPE and sandwich normalization, Next-DiT demonstrates exceptional resolution extrapolation during inference, even without fine-tuning on varied resolutions. Furthermore, the proposed dynamic partitioning and padding scheme allows for efficient fine-tuning while preserving image resolutions and aspect ratios. Consequently, our model can perform flexible inference on images with arbitrary resolutions and aspect ratios.

(0,0)

(0,1) (0,2)

(1,0)

(1,2)

(1,1)

(2,0)

(2,1) (2,2)

(0,0) (0,1)

(1,0)

(1,1)

(2,0) (2,1)

(3,0) (3,1)

(0,0)

(0,1) (0,2)

(1,0)

(1,2)

(1,1)

(2,0)

(2,1) (2,2)

(0,3)

(1,3)

(2,3)

Image 1

Image 2

Image N

Image Token

Seq 1

Seq 2

Seq N

Padding Token

2D Rope at position (i,j)

(i,j)

Dynamic

Partition

Image tokens with Dynamic Padding

Resize

Resize

Resize

0

0

0

0

0

0

0

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

0

0

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

0

0

0

0

0

0

0

0

-inf

0

0

0

0

0

0

0

0

-inf

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

0

0

0

0

0

0

0

0

0

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

-inf

Mask 1

Mask 2

Mask N

&

Masked Attention

Attention Masks

D.2 Setups

We conduct experiments on the ImageNet-1K dataset to validate the effectiveness of Next-DiT for image recognition. We build our Next-DiT model following the architecture hyperparameters of ViT-base [28], stacking 12 transformer layers with a hidden size of 768 and 12 attention heads. This configuration ensures that our architecture has a comparable number of parameters to the original ViT. During the fixed-resolution pre-training stage, we train the models from scratch for 300 epochs with an input size of 224 × 224 . We use the AdamW optimizer with a cosine decay learning rate scheduler, setting the initial learning rate, weight decay, and batch size to 1e-3, 0.05, and 1024, respectively. We follow the same training recipe as the DeiT model [79], including its data augmentation and regularization strategies. In the subsequent any-resolution fine-tuning stage, we further train the pre-trained model using the proposed dynamic partitioning and padding scheme. This finetuning is performed for an additional 30 epochs with a constant learning rate of 1e-5 and a weight decay of 1e-8.

D.3 Experiments

We report the performance of our Next-DiT in Table 1. We first test our model at a fixed resolution of 224 × 224 . Compared with DeiT-base, which has comparable parameters, Next-DiT exhibits superior performance and faster convergence. Notably, training the model for only 100 epochs achieves a Top-1 accuracy of 81.6%, nearly matching DeiT-base's result achieved after 300 epochs. Under the same training setting with 300 epochs of pre-training, Next-DiT surpasses DeiT-base with an accuracy of 82.3% versus 81.6%, respectively, demonstrating its effectiveness as a robust image backbone. We also adopt the flexible resolution inference strategy, preserving the original resolution and aspect ratio of the input images to validate the model's generalization capability across varied resolutions. After the additional fine-tuning stage, Next-DiT achieves 84.2% accuracy when handling arbitrary resolutions. We report the performance of DeiT-base pre-trained at 224 × 224 , using the same inference configuration. To align with the input resolution, we interpolate its positional embeddings accordingly. Compared to our fine-tuned Next-DiT, the performance of DeiTbase drops significantly, suggesting its limitations in processing images with varied resolutions.

224

384

512

1024

60

65

70

75

80

Top-1 Accuracy (%)

DeiT

Next-DiT-pretrain

Next-DiT-finetune

Resolution

We also evaluate the performance across different image resolutions, as illustrated in Figure 20. Next-DiT demonstrates better generalization to larger image sizes compared to DeiT-base, even without fine-tuning. With additional fine-tuning using the proposed dynamic partitioning scheme, our model significantly enhances its ability to handle varied resolutions, particularly high resolutions such as 1024 × 1024 , outperforming DeiT-base by a significant margin. These experimental results highlight the effectiveness of Next-DiT as a powerful image backbone for recognizing images at any resolution, in addition to its capability to generate images at various resolutions.

E Generating Multi-Views, Music and Audio with Lumina-Next

Lumina-Next is a versatile and expandable framework for generative modeling that goes beyond text-conditioning image generation. In this section, we demonstrate the application of Lumina-Next to multi-view generation using both image and text conditioning. Additionally, we extend LuminaNext to encompass a wider range of modalities, including text-conditional audio, music, and point cloud generation.

E.1 Image- and Text-conditional Multi-View Generation

Pipelines Having witnessed the success of Next-DiT on text-to-image generation, we use NextDiT's text-to-image model as a pre-trained model and extend it to image- and text-conditional multiview generation, called MV-Next-DiT, as shown in Figure 21. Firstly, we introduce relative pose control to Next-DiT to distinguish between views. Specifically, for N multi-view images with the same elevation and random azimuths rendering from 3D object, taking the azimuth of the first image as the reference view, the relative azimuths of the remaining N -1 images to the first image are calculated sequentially. The obtained relative azimuths are encoded into the same dimension as time embedding using a 2-layer MLP network after sin and cos transformation, and are added to the time embedding to inject camera control into the model. Secondly, we expanded the original text condition of Next-DiT into text and image conditions. Specifically, the same encoding and injection method as Next-DiT is used for text condition. For the image condition, we use VAE to encode the input image, and then concatenate it to the multi-view noises ( N + 1 ). This image is not used for supervision during training. In addition, when building the training data, this condition image has the same azimuth as the first supervised view (both are 0), but it is a random elevation. It is worth noting that the image condition is dropped based on probability during training, and the input of the image condition is optional during inference. Finally, a progressive training strategy of views and resolution is adopted during training, and views flexible inference strategy is used during inference. Specifically, during training, we first performed training at 256 × 256 resolution with N = 4 . Then it was expanded to the training of 512 × 512 resolution with N = 4 , and finally it was expanded to the training of 512 × 512 resolution with N = 8 . In inference, since each object is trained using all random views, any number of view can be inferenced at one time. These improvements allow MV-Next-DiT to output multi-view images with high quality and flexible azimuth.

Setups Data preparation: we use the Objaverse [24] list and caption provided by Cap3D [58] as training data. For data rendering, we first use the same elevation and 8 random azimuths for each object, render with the same camera setting, and obtain supervision data. Then the azimuths corresponding to 4 images are randomly selected from the 8 rendered views, and cooperate random

MV-Next-DiT

(Any View Inference)

MV-Next-DiT

(All View Training)

Image Condition

(Optional)

Noised

View-1

Noised

View-2

…

Noised

View-N

Velocity

View-1

Velocity

View-2

…

Velocity

View-N

Text

Condition

Time

Embedding

Any View Inference

Text/Image Conditional All View Finetune

Relative Pose

Embedding-1

Relative Pose

Embedding-2

Relative Pose

Embedding-N

Training from

Next-DiT's

initialization

Text

Condition

Time

Embedding

Image Condition

(Optional)

Noised

View x N

Relative Pose

Embedding x N

Velocity

View x N

Inference

elevations to render input images. Each object then has 4 groups of training samples. Training details: We use Next-DiT with 600M parameters for training of MV-Next-DiT. The specific training settings are shown in Table 2.

Experiments We collected images provided by previous work, such as CRM [83], to test the performance of our MV-Next-DiT. In particular, we provide the generated results of only text condtion, text and image conditions, and the results of sampling 4 views, 6views, and 8views respectively from the setting of N=8, as shown in Figure 22. In addition, we compared the capabilities with existing multi-view methods as shown in Table 3. The overall performance demonstrates the advantages of our MV-Next-DiT. The rich higher-resolution flexible view generated results demonstrate MV-NextDiTs excellent multi-view generation capabilities and the superiority of the Next-DiT architecture.

E.2 Text-Conditional Audio and Music Generation

Pipelines Weuse a modified audio VAE with a 1D-convolution-based model to derive audio latent. The audio signal is a sequence of mel-spectrogram sample x ∈ [0 , 1] C a × T , where C a , T respectively denote the mel channels and the number of frames. Our spectrogram autoencoder is composed of 1) an encoder network E which takes samples x as input and outputs latent representations z ; 2) a decoder network G reconstructs the mel-spectrogram signals x ′ from the compressed representation z ; and 3) a multi-window discriminator learns to distinguish the generated samples G ( z ) from real ones in different multi-receptive fields of mel-spectrograms.

In text2audio generation, we include the dual text encoder architecture consisting of a main text encoder CLAP [30] that takes the original natural language caption y as input and a temporal encoder FLAN-T5 [20] which takes the structured caption y s passed by LLM as input. The final conditional

Chibi-style character with red hair, cat ears, and a happy expression.

Adorable kitten with big eyes and a playful expression.

Text & Image condition

Text condition

Input

- ------

+

---------

I

1

6,0

representation is expressed as:

Where f text is the main text encoder and f temp is the temporal encoder. For the text encoder in text2music generation, we use the FLAN-T5 [20] text encoder that takes the original natural language caption y as input.

Setups of Text-to-Music Generation We employ a diverse combination of datasets to facilitate the training process of our model. In the context of text-to-music synthesis, we exclusively employ the LP-MusicCaps [27] dataset. The culmination of these efforts yields a dataset comprising 0.92 million audio-text pairs, boasting a cumulative duration of approximately 3.7K hours .

We conduct preprocessing on both text and audio data as follows: 1) We convert the sampling rate of audios to 16kHz. Prior works [84, 41, 46] pad or truncate the audio to a fixed length (10s), while we group audio files with similar durations together to form batches to avoid excessive padding which could potentially impair model performance and slow down the training speed. This approach also allows for improved variable-length generation performance. We truncate any audio file that exceeds 20 seconds, to speed up the training process. 2) For audios without natural language annotation, we apply the pseudo prompt enhancement method from Make-An-Audio [41] to construct captions aligned with the audio. We traverse the AudioCaps training set and the LLM augmented data with a probability of 50%, while randomly selecting data from all other sources with a probability of 50%. For the latter dataset, we use "<text & all>" as their structured caption.

Experiments of Text-to-Music Generation In this part, we compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) MusicGen [21]; 3) MusicLM [3]; 4) Mousai [68]; 5) Riffusion [33]; 6) MusicLDM [16]; 7) AudioLDM 2 [47]. The results are presented in Table 4, and we have the following observations: Regarding audio quality, our model consistently surpasses diffusion-based methods and language models across FAD and KL metrics. In terms of subjective evaluation, our model demonstrates the strong text-music alignment faithfulness.

Setups of Text-to-Audio Generation Following the benchmark studies [43, 41], we train on AudioSet-SL [46] and finetune the model in the Audiocaps dataset. Overall, we have ∼ 3k hours with 1M audio-text pairs for training data. For evaluating text-to-audio models, the AudioCaption validation set is adopted as the standard benchmark, which contains 494 samples with five humanannotated captions in each audio clip.

We conduct preprocessing on the text and audio data: 1) convert the sampling rate of audios to 16kHz and pad short clips to 10-second long; 2) extract the spectrogram with the FFT size of 1024, hop size of 256 and crop it to a mel-spectrogram of size 80 × 624 ; 3) non-standard words (e.g., abbreviations, numbers, and currency expressions) and semiotic classes [77] (text tokens that represent particular entities that are semantically constrained, such as measure phrases, addresses, and dates) are normalized.

Our models conduct a comprehensive evaluation using both objective and subjective metrics to measure audio quality, text-audio alignment fidelity, and inference speed. Objective assessment includes Kullback-Leibler (KL) divergence, Frechet audio distance (FAD), and CLAP score to quantify audio quality. In terms of subjective evaluation, we conduct crowd-sourced human assessments employing the Mean Opinion Score (MOS) to evaluate both audio quality (MOS-Q) and text-audio alignment faithfulness (MOS-F).

Audio/Music

Encoder

Latent

Audio/Music

Decoder

Noised Latent

Lumina-Next

Training

Inference

Velocity

Noise

Lumina-Next

Velocity

ODE

Solver

After N steps

Vocoder

Init.

Audio/Music Autoencoding

Generated

Audio/Music

Text

Embedding

Time

Embedding

Text

Embedding

Time

Embedding

CLAP

+FLAN-T5

FLAN-T5

Audio

Music

CLAP

+FLAN-T5

FLAN-T5

Audio

Music

Experiments of Text-to-Audio Generation We conduct a comparative analysis of the quality of generated audio samples and inference latency across various systems, including GT (i.e., groundtruth audio), AudioGen [43], Make-An-Audio [41], AudioLDM-L [47], TANGO [36], Make-AnAudio 2 [39], and AudioLDM 2 [47], utilizing the published models as per the respective paper and the same inference steps of 100 for a fair comparison. The evaluations are conducted using the AudioCaps test set and then calculate the objective and subjective metrics. The results are compiled and presented in Table 6. From these findings, we draw the following conclusion:

In terms of audio quality, our proposed system demonstrates outstanding performance with the FAD of 1.03, demonstrating minimal spectral and distributional discrepancies between the generated audio and ground truth. Human evaluation results further confirm the superiority, with MOS-Q and MOS-F scores of 77.53 and 76.52, respectively. These findings suggest a preference among evaluators for the naturalness and faithfulness of audio synthesized by our model over baseline approaches.

For ablation, we 1) explore the effectiveness of dual text encoders for a deep understanding of arbitrary natural language input and observe that the drop in CLAP encoders has witnessed the distinct degradation of text-audio alignment faithfulness; 2) study the usage of Audioset data, and find that without fine-tuning on Audiocaps following previous works [47], a significant drop on objective evaluation happens.

E.3 Label- and Text-Conditional Point Cloud Generation

Pipelines We propose a density-invariant point cloud generator to validate the effectiveness of Lumina-Next in the 3D domain. The density-invariant property allows an efficient training on a fixed point number ( e.g ., 256 points) and an inference of any point number ( e.g ., 1024, 2048, or more points). Different from 2D images, increasing the density of a point cloud does not affect the overall shape but rather enhances the details. To achieve any point inference, we combine the Time-aware Scaled RoPE with Fourier features [76] and design a Time-aware Scaled Fourier feature.

(

[cos(2 π θ ⊤ 1 p ) , sin(2 π θ ⊤ 1 p ) , ..., cos(2 π θ ⊤ d p ) , sin(2 π θ ⊤ d p ) , ... ] . To inject the time condition of diffusion steps into the Fourier feature, we scale the frequency via θ ′ d = θ d · s d head d t , where d t = d head -1) · t + 1 . This leads the generator to focus on the global information in the early steps of denoising, and include more shape details in later steps. This Time-aware Scaled Fourier feature can effectively distinguish continual positions in 3D space, thus our model guarantees any point inference even though the model is only trained on a fixed low density.

Setups We train a label and a textual conditioned 3D generator respectively. For the label conditional model, we employ the ShapeNet dataset [13] and follow [56] to preprocess it. We randomly sample 256 points from each point cloud to train the model. After training, we synthesize point clouds of densities in { 256 , 512 , 1024 , 2048 , 4096 , 8192 } . To evaluate the generation, we adopt the Minimum Matching Distance (MMD) and the Coverage score (COV) as quality metrics and use Chamfer Distance (CD) as the distance measure to calculate them. For the textual guided generator, we adopt the Cap3D dataset [57] to train the model, where we use RoPE positional embedding instead of the time-aware Fourier feature.

8192 Points

4096 Points

2048 Points

1024 Points

512 Points

256 Points

Airplane

Bench

Lamp

Chair

山

人

Experiments We compare the generation performance of two categories: airplane and chair. We fine-tune our pre-trained generator on 2048 points to further improve the generation quality. We mainly compare our performance with PC-GAN [2], TreeGAN [72], PointFlow [85], ShapeGF [11], and PDiffusion [56] in Table 7. We observe that our generator achieves a satisfactory performance compared to existing models. We also visualize some generated point clouds in Figure 24, where we directly apply the generator pre-trained on 256 points to sample higher-density point clouds. We observe the density of generated point clouds decreases when we increase the density ( e.g ., airplane and chair), which could be due to the fact that more points would cause a smoother attention matrix in self-attention layers.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, our paper proposes a unified framework for generative modeling, which is corresponded with our abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation at the beginning of the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not focus on theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We depict all implementing details of our method. We are also releasing the code and checkpoints for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the training and inference code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give implementing details as well as configs for our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The evaluation of large generative models is expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We list GPU usage for each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential societal impacts in our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We explain in our paper about filtering the dataset to remove harmful content.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite the original paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide readme for our codebase.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLumina-Next is an improved version of Lumina-T2X, featuring a core architecture  that employs a Flow-based Large Diffusion Transformer (Flag-DiT). Through empirical experiments and analysis, Lumina-Next introduces an enhanced Next-Dit architecture and develops a fast sampling algorithm, which boosts the model's generalization capability and improves both training and inference efficiency.\n\nSTRENGTHS:\nThis paper focuses on Lumina-T2X and provides empirical analysis and discussion on the training stability, inference efficiency, and extrapolation performance of this type of Flow-based Large Diffusion Transformer. Its advantages are as follows:\n\n- This paper uses 3D RoPE to replace 1D RoPE, enabling the perception of positional information in both spatial and temporal dimensions without the need for learnable [nextline] and [nextframe] identifiers. These improvements help stabilize training and benefit the extrapolation of visual sequence context.\n- Sandwich normalization is applied to the model structure to prevent the accumulation of errors due to increased model depth, thereby preserving model performance.\n- Different extrapolation schemes is ablated, and Frequency- and Time-Awareness Scaled RoPE was proposed to improve resolution extrapolation.\n- A time schedule suitable for flow-based diffusion models was proposed, improving the visual quality of fast sampling produced with 10-20 NFEs.\n\nOverall, this paper addresses some of the shortcomings of Lumina-T2X and provides insights for the development of large-scale flow-based diffusion transformers.\n\nWEAKNESSES:\n- Whether identifiers such as [nextline] and [nextframe] are unnecessary when using 3D RoPE is not sufficiently discussed in the paper.\n- The paper lacks a comparison with stable-diffusion-3, which is also a flow-based diffusion model.\n- Is the proposed Time Schedule universally applicable at low NFEs (<10)? Figure 7 contains too few examples to reflect the actual effect. Are there any numerical comparisons? \n\nOverall, this paper leans towards empirical research and analysis, making it more akin to a technical report.\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nAlthough the paper discusses limitations in the supplementary materials, I believe it could benefit from adding some discussion on whether the training data might introduce biases related to violence, pornography, race, etc.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces Lumina-Next, an enhanced version of the Lumina-T2X model, which is a Flow-based Large Diffusion Transformer (Flag-DiT) aimed at transforming noise into various modalities like images and videos based on text instructions. Compared with Lumina-T2X, Lumina-Next introduces the following improvements:
* A redesigned architecture named Next-DiT, featuring 3D Rotary Position Embedding (RoPE) and sandwich normalization to stabilize training and inference.
* Frequency- and Time-Aware Scaled RoPE for better resolution extrapolation in text-to-image generation.
* An optimized time discretization schedule combined with higher-order Ordinary Differential Equation (ODE) solvers for faster and high-quality generation.

The model demonstrates stronger performance in text-to-image generation, resolution extrapolation, and multilingual capabilities in a zero-shot manner. It also shows versatility across different tasks such as visual recognition, multi-views, audio, music, and point cloud generation.

STRENGTHS:
- This paper, like SD3 and PixArt, explores image generation within the DiT architecture, and the codes and models have been open-sourced, which is beneficial to the community.
- Strong functionality. Without additional training, Lumina-Next can easily generate higher-resolution images and produce images of decent quality in fewer inference steps.

WEAKNESSES:
- The paper mainly demonstrates the advantages over comparative methods in terms of functionality but does not show comparisons with these state-of-the-art methods in terms of text-image alignment and image quality.
- Although the paper's method is about DIT, it does not demonstrate the manifestation and properties of scaling laws in the generation domain.

QUESTIONS:
na

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new multi-modal generation model, Lumina-Next, which extends the previous Lumina-T2X approach by several innovations:  i) 3D rotary position embedding (RoPE) ii) extra normalization to stablize training iii) frequency- and time- aware scaled RoPE for training free resolution extrapolation iv) improved time scheduling and higher-order ODE solvers for reverse sampling. Lumina-Next aims to achieve faster, better, and more efficient generation on multiple text-to-X tasks, e.g. text-to-image/audio/point-cloud, etc.\n\nSTRENGTHS:\nTopic: the paper studies multi-modal generation using a unified framework, which would be of great interest to the community. The paper also promises to release the source code to support future research\n\nMethodology: the proposed technical extensions for Lumina-T2X are technically sound. \n\nExperiment: Lumina-Next demonstrates promising qualitative results on text-to-image generation, training-free resolution extrapolation, and text-to-point-cloud generation. The paper also provides quantitative evaluation on text-to-audio/music generation.\n\nWEAKNESSES:\nThe reviewer appreciates the authors’ effort on improving text-to-X generation. However, as a technical submission for NeurIPS, the paper is fully convincing as it provides limited quantitative evaluation. \n\nAs a representative reader, the reviewer would expect\n\ni) quantitative evaluations on all generation tasks\n\nii) extra apples-to-apples evaluations on any resolution recognition, as for now, there is only one baseline, DeiT-Base. The comparison is also apples-to-oranges as the models have different architectures and were trained on different data.\n\niii) comprehensive quantitative ablations on the proposed technical extensions, a) 3D RoPE,  b) extra normalization, c) frequency- and time- aware scaled RoPE, d) improved time scheduling and higher-order ODE solvers for reverse sampling, etc\n\nQUESTIONS:\nPlease refer to the weaknesses\n\nLIMITATIONS:\nLimitations have been discussed in the Appendix", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper introduces the next generation of Lumina-T2X, Lumina-Next, which offers improved architecture, a scaled dataset, optimized sampling techniques, and a more efficient context extrapolation strategy. The improved architecture shows faster convergence rates, while the optimized sampling technique enables high-quality text-to-image generation with fewer steps. Through visual results, the authors validate the improvements provided by refining Flag-DiT in Lumina-T2X.

STRENGTHS:
* Next-DiT brings excellent high-resolution extrapolation generation.

* Compared to previous work, it provides better generation quality under a few-step generation setting.

WEAKNESSES:
* There is a lack of quantitative experiment comparison results on some Text2Image benchmarks to demonstrate the superiority and effectiveness of the proposed method. Most of the comparisons are shown through visualization.

QUESTIONS:
* Text-to-image benchmark result should be evaluated and reported

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLumina-T2X encounters challenges including training instability, slow inference, and extrapolation artifacts. This paper introduces Lumina-Next, which improves Lumina-T2X with improved architecture, scaled dataset, optimized sampling techniques, and better context extrapolation strategy.\n\nOn the architecture side, they replace 1D RoPE with 3D RoPE to eliminate the inappropriate positional priors to model images and videos using the attention mechanism. They further removed all learnable identifiers in Flag-DiT, and introduced the sandwich normalization block in attention modules to control the activation magnitudes.\n\nOn the context extrapolation side, the authors propose a Frequency-Aware Scaled RoPE, reducing content repetition during extrapolation, and a novel Time-Aware Scaled RoPE for diffusion transformers to generate high-resolution images with global consistency and local details.\n\nOn the sample techniques side, the authors propose time schedules tailored for flow models to minimize discretization errors. They further combine the optimized schedules with higher-order ODE solvers, achieving high-quality text-to-image generation samples within 5 to 10 steps. \n\nWith the improved architecture, context extrapolation, and fast sampling techniques, the authors show that Lumina-Next yields strong generation capabilities, such as generating high-quality images much larger than its training resolution and multilingual text-to-image generation. \n\nFurthermore, they extend Lumina-Next’s versatility to other modalities, such as multiviews, audio, music, and point clouds, with minimal modifications, achieving superior results across these diverse applications.\n\nSTRENGTHS:\n- The paper is well written with extensive results, applications and details, which is valuable to the community.\n- The paper provides practical improvements over architecture, dataset, sampling techniques, and context extrapolation strategy. \n- The rethinking of existing model design is appreciated, especially showing a much smaller network can get better performance.\n\nWEAKNESSES:\n- the changes over Lumina-T2X are a bit incremental, considering Lumina-T2X is just released a month before the submission. Some of the designs in Lumina-T2X itself such as learnable tokens are not essential and only exist in Lumina-T2X, so the improvement (removing it) is not applicable to other DiTs, so removing extra learnable tokens itself can hardly be considered innovation. \n- the 3D rope does not show advantage over native designs, such as normal 2D + temporal positional embedding. There is no application showing the usage of 3D RoPE, such as video generation.\n- the ImageNet experiments can hardly justify the improvements. The comparison between Lumina-Next and SiT/Flag-DiT is not clear, such as whether the model sizes are the same.\n- there is a lack of explanation why model size can be reduced from the Flag-DiT 5B/7B model significantly. \n- The adoption of newer and stronger text encoders is also not ablated, so the improvement over Lumia-T2X cannot fully be attributed to the proposed changes.\n- There is a lack of quadratic evaluation metrics such as TIFA and ImageReward scores for fair comparison across models.\n\nQUESTIONS:\nWith all those changes on controlling the activation magnitudes, what about simply using grad clip norm to reduce gradient norms?\n\nLIMITATIONS:\nThere are more interesting contents in the appendix, which can be moved to the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}