EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing

## Abstract

Abstract Diffusion transformers have been widely adopted for text-to-image synthesis. While scaling these models up to billions of parameters shows promise, the effectiveness of scaling beyond current sizes remains underexplored and challenging. By explicitly exploiting the computational heterogeneity of image generations, we develop a new family of Mixture-of-Experts (MoE) models ( EC-DiT ) for diffusion transformers with expert-choice routing. EC-DiT learns to adaptively optimize the compute allocated to understand the input texts and generate the respective image patches, enabling heterogeneous computation aligned with varying text-image complexities. This heterogeneity provides an efficient way of scaling EC-DiT up to 97 billion parameters and achieving significant improvements in training convergence, text-to-image alignment, and overall generation quality over dense models and conventional MoE models. Through extensive ablations, we show that EC-DiT demonstrates superior scalability and adaptive compute allocation by recognizing varying textual importance through end-to-end training. Notably, in text-to-image alignment evaluation, our largest models achieve a state-of-the-art GenEval score of 71.68% and still maintain competitive inference speed with intuitive interpretability.

## 1 Introduction

Diffusion models (Ho et al., ; Rombach et al., ; Podell et al., ) have demonstrated remarkable success in generation across various modalities. There has been growing interest in scaling these models to billions of parameters to improve performance and control computation over the generation process.
One of the primary focuses of scaling diffusion models is to enhance scalability for *text-conditioned image generation*.
As a promising approach for this synthesis task, diffusion transformers (DiT) (Peebles & Xie, ) combine the strengths of diffusion models and transformers and exhibit great potential in scalability.
Current DiT-based models, such as Stable Diffusion 3 (SD), reach up to 8 billion parameters (Esser et al., ) and achieve promising performance in text-to-image generation. However, the effectiveness and optimal approaches for scaling DiTs beyond this size remain underexplored, primarily due to the significant impacts on training and inference efficiency associated with larger models.

The sparse Mixture-of-Experts (MoE) technique (Zoph et al., ; Lepikhin et al., ; Du et al., ; Zhou et al., ) has proven to be an efficient method for scaling diffusion models like DiT. Sparse MoEs effectively scale up model capacity while maintaining relatively low computational overhead. In MoE architectures, the router selects a subset of parameters (experts) to process each group of token embeddings and then combines the final outputs from each expert.
Current works (Yatharth Gupta, ; Fei et al., ; Xue et al., ) on sparse DiT often adopt a token-choice routing strategy (Zoph et al., ; Lepikhin et al., ; Fedus et al., ), which routes *each text token* to the top-ranked experts. However, this routing strategy is suboptimal with diffusion models, which
requires a more adaptive routing approach. Specifically, unlike autoregressive generation, DiT processes the entire image sequence at a time. This suggests that the global information of generated samples is accessible throughout the denoising procedure. Additionally, different image areas often contain varying levels of detail. Thus, an optimal router should adaptively allocate heterogeneous computation for different patterns.

Driven by these motivations, we propose scaling DiT with adaptive expert-choice routing (EC-DiT), which is of independent interest, and tailor it for text-to-image generation. This approach leverages global information from each image to optimize computational resource allocation that is aligned with varying image complexities, thus naturally aligning with the diffusion procedure. We scale EC-DiT up to 97 billion parameters with 64 experts.
Figure 1 showcases the promising scalability of EC-DiT.
Despite a less than 30% increase in computational overhead, our approach significantly improves training convergence, text-to-image alignment, and overall generation quality compared to dense and sparse variants with conventional token-choice MoEs (Lepikhin et al., ; Du et al., ; Zoph et al., ). Notably, in text-to-image alignment evaluation, EC-DiT achieves a new GenEval score of 71.68% while maintaining competitive inference speed compared to other strong baselines.
Our main contribution is summarized as follows:

- •
Scaling DiT with adaptive computation. We introduce EC-DiT, a sparsely scaled DiT incorporating expert-choice routing for text-to-image synthesis. This novel approach leverages global context information at each generation step to achieve heterogeneous compute allocation adaptive to different patterns within the generated images.
- •
Promising performance at scale. We examine EC-DiT’s scalability and effectiveness by scaling up to 97 billion parameters. At larger model scales, EC-DiT demonstrates faster loss convergence, improves image quality, and reaches state-of-the-art text-image alignment. Moreover, scaling with more experts consistently improves performance across various dimensions without significantly increasing computational overhead.
- •
Comprehensive experiments and analysis. We conduct extensive experiments to validate EC-DiT’s superior scalability compared to dense and token-choice variants. Visualizations further demonstrate the model effectively learns an adaptive computation allocation strongly aligned with textual significance.

## 2 Methodology

Starting with preliminaries of rectified flow model and MoE, this section presents the main design and properties of our EC-DiT models.

### 2.1 Preliminaries

##### Image generation via rectified flow.

DiT processes images as sequences of patches in latent space. An input image is first equally divided into a grid of patches, then flattened and projected into a sequence of embeddings. We denote $\mathbf{x}\in\mathbb{R}^{S\times d_{x}}$ as the image sequence of length $S$ in latent space, with $d_{x}$ representing the model’s hidden dimension. Diffusion models progressively perform time-dependent denoising procedures on the image sequence $\mathbf{x}$.
Given two observations $\mathbf{x}_{0}\sim p_{0}(\mathbf{x})$ and $\epsilon\sim p_{1}=\mathcal{N}\left(0,1\right)$, rectified flow (Liu et al., ; Lipman et al., ; Esser et al., ) defines a probability flow ordinary differential equation (ODE) to transfer between $p_{0}$ and $p_{1}$ with a straight-line path: $d\mathbf{x}_{t}=v_{\theta}(\mathbf{x}_{t},t)dt$.
We parametrize the velocity (*i.e.*, the path direction) $v_{\theta}(\mathbf{x}_{t},t)$ with DiT and optimize the following L2 objective (Liu et al., ; Esser et al., ):

$$ $\mathcal{L}\left(\theta\right)=\mathbb{E}_{t\sim\pi\left(t\right),\epsilon\sim \mathcal{N}\left(0,1\right)}\left[\|(\mathbf{x}_{1}-\mathbf{x}_{0})-v_{\theta} (\mathbf{x}_{t},t)\|^{2}\right],\quad\mathbf{x}_{t}=t\mathbf{x}+(1-t)\epsilon,$ (1) $$

where $\pi(\cdot)$ refers to the timestep density during training, for which we select the logit-normal sampling strategy (Atchison & Shen, ): $t=\log\frac{u}{1-u}$, with $u\sim\mathcal{N}(\mu_{t},\sigma_{t})$. The hyperparameters $\mu_{t}$ and $\sigma_{t}$ control the frequency for intermediate timesteps being sampled during the training process.

##### Mixture-of-Experts.

Shazeer et al. () first introduced the MoE layer that consists of a set $\{{\mathcal{E}}_{i}(x)\}_{i=1}^{E}$ of $E$ experts (each of which is often a FeedForward network (FFN)) and a learnable router with the weight $\mathbf{W}_{r}$. For a given token representation $\mathbf{x}$, the router selects the highest top-$k$ experts based on the gating value of $\mathbf{x}\cdot\mathbf{W}_{r}$. The output of the MoE layer will then be the weighted combination of the selected expert’s computation where the expert weights are the normalized gating values via the softmax distribution. Since the token chooses its best set of experts, this top-$k$ routing is also termed as the token-choice routing.

### 2.2 Text conditioning with cross-attention modules

In the text-to-image synthesis, the model generates image samples $\mathbf{x}$ conditioned on the text prompt $\mathbf{y}$.
We utilize a text encoder $\text{Enc}\left(\mathbf{y}\right)\in\mathbb{R}^{L\times d_{y}}$ to extract the text embeddings of $\mathbf{y}$ where $L$ is the number of text tokens, and $d_{y}$ is the text’s hidden dimension.
Following the approach in Chen et al. (), we incorporate a multi-head cross-attention layer (Vaswani et al., ) between the self-attention and FFN layers in each DiT block. The cross-attention $\mathbf{C}_{h}$ for head $h$ is written as:

$$ $\displaystyle\mathbf{Q}_{x}=\mathbf{x}\cdot\mathbf{W}_{x}^{\text{query}},\quad \mathbf{K}_{y}=\text{Enc}\left(\mathbf{y}\right)\cdot\mathbf{W}_{y}^{\text{key }},\quad\mathbf{V}_{y}=\text{Enc}\left(\mathbf{y}\right)\cdot\mathbf{W}_{y}^{ \text{value}},$ (2) $$

$$ $\displaystyle\textstyle\mathbf{C}_{h}=\texttt{softmax}\left(\frac{\mathbf{Q}_{ x}\cdot\mathbf{K}_{y}^{\top}}{\sqrt{d_{x}}}\right)\cdot\mathbf{V}_{y},$ (3) $$

where $\mathbf{Q}_{x}$ is the query matrix for $\mathbf{x}$, and $\mathbf{K}_{y}$, $\mathbf{V}_{y}$ are the key and value matrices for $\mathbf{y}$, respectively;
$\mathbf{W}_{x}^{\text{query}}\in\mathbb{R}^{d_{x}\times d_{x}}$ and $\mathbf{W}_{y}^{\text{key}},\mathbf{W}_{y}^{\text{value}}\in\mathbb{R}^{d_{y}
\times d_{x}}$ are the projection matrices for the image and text representations.
Then, the outputs from all $H$ heads are concatenated and linearly transformed by $\mathbf{W}^{\text{out}}$ to obtain the final output, *i.e.*, $\texttt{MHCA}(\mathbf{x})=\texttt{concat}([\mathbf{C}_{1},\dots,\mathbf{C}_{H}
])\cdot\mathbf{W}^{\text{out}}$.
This modification effectively injects textual information into the image generation.

### 2.3 Scaling DiT with expert-choice routing

Figure: Figure 2: EC-DiT architecture. The router leverages information from the entire sequence to adaptively route the most suitable tokens to each expert. Through this heterogeneous routing, more computation is allocated to detailed image areas, such as the space station and moon, while less computation is used for rendering the background.
Refer to caption: x2.png

Because the majority of computation takes place in the dense FeedForward (FFN) layers (Peebles & Xie, ) of DiT, replacing the FFN layer with a MoE layer is one efficient method of scaling. The introducing of the MoE layer decouples the effective computation from the model capacity. That is, we can effectively scale up the model capacity without significantly sacrificing the inference speed since only a selected subset of experts will be activated conditioned on the input representation. While this approach seems plausible and effective, it can be further improved for the following reasons:

- •
Uniform computation. The computation assigned to each input token is the same regardless of its representation even though the selected set of experts can be different. However, different image patches naturally demand varying amount of compute, e.g,. foreground needs more attention while background and low-frequency areas may be disregarded.
- •
Local context. Token-choice routing cannot access to the global information of the sequence due to the constraint of auto-regressive decoding. However, this global information can be better exploited in the diffusion process.
- •
Load balancing. To improve the load balance of each expert, some sorts of auxiliary load balance loss (Shazeer et al., ; Lepikhin et al., ; Du et al., ; Zoph et al., ) are often required for token-choice routing to work properly.

Therefore, we propose to scale up the diffusion models by better exploiting the structure of the diffusion process. Specifically, following Peebles & Xie (); Chen et al. (), we embed the timestep using a two-layer MLP with dimensionality equal to the hidden dimension $d_{x}$, and feed this embedding into each AdaLN layer.
Additionally, as shown in Figure 2, let
$\mathbf{x}_{s}$ be the output of the self-attention module, and $\mathbf{x}^{\prime}$ be the input to the router, *i.e.*,

$$ $\mathbf{x}^{\prime}=\mathbf{x}_{s}+\texttt{MHCA}(\mathbf{x}_{s})\in\mathbb{R}^ {S\times d_{x}}.$ (4) $$

Therefore, the input sequence $\mathbf{x}{\prime}$ contains timestep information and integrates different modalities through equation 3.

Each expert is a two-layer FeedForward component, where the $i$-th expert is represented as ${\mathcal{E}}_{i}\left(\mathbf{x}\right)=\texttt{GeLU}(\mathbf{x}\cdot\mathbf{
W}_{\text{1}}^{i})\cdot\mathbf{W}_{\text{2}}^{i}$. Here, $\mathbf{W}_{\text{1}}^{i}\in\mathbb{R}^{d_{x}\times d_{x}^{\prime}}$ and $\mathbf{W}_{\text{2}}^{i}\in\mathbb{R}^{d_{x}^{\prime}\times d_{x}}$ are the weight matrices for the $i$-th expert.
For each MoE layer, the router is parameterized by the expert embedding $\mathbf{W}_{r}\in\mathbb{R}^{d_{x}\times E}$ where $E$ is the total number of experts of the layer. Given the input $\mathbf{x}^{\prime}$, the router first produces a token-expert affinity score tensor $\mathbf{A}\in\mathbb{R}^{S\times E}$ via softmax along the expert dimension, *i.e.*,

$$ $\mathbf{A}_{s,i}=\frac{\exp\left(\left(\mathbf{x}^{\prime}\cdot\mathbf{W}_{r} \right)_{s,i}\right)}{\sum_{i=1}^{E}\exp\left(\left(\mathbf{x}^{\prime}\cdot \mathbf{W}_{r}\right)_{s,i}\right)}.$ (5) $$

This affinity score tensor assesses the relevance between each pair of expert and input token.

Unlike the token-choice routing where each token $\mathbf{x}^{\prime}$ selects the top-$k$ experts from $\mathbf{A}_{s,i}$, EC-DiT works from the expert-choice (Zhou et al., ) view where each expert independently selects the top-$C$ tokens in the descending order from $\mathbf{A}_{s,i}$, and $C=B\times S\times f_{c}/E$ represents the average capacity of each expert, where $B$ is the batch size of image patches, $f_{c}$ denotes the capacity factor and reflects the average number of experts assigned to process each token.

Figure: Algorithm 1 Pseudocode of EC-DiT’s Routing Layer

Comparing to the token-choice routing, which assigns each token independently, EC-DiT selects the most relevant $C$ tokens from the *entire* sequence.
To achieve this, we compute the gating tensor $\mathbf{G}\in\mathbb{R}^{S\times E}$ as follows:

$$ $\mathbf{G}_{s,i}=\begin{cases}\mathbf{A}_{s,i},\quad&\mathbf{A}_{s,i}\in \texttt{top-k}\left(\left\{\mathbf{A}_{s,i}\mid 1\leqslant s\leqslant S\right \},\texttt{k}=C\right)\\ 0,\quad&\text{otherwise},\end{cases},$ (6) $$

where $\mathbf{G}_{s,i}$ is the weighting score representing expert $i$’s preference over the $s$-th token.

With equation 3 and equation 6, EC-DiT directs the computational focus of the experts towards tokens with significant textual information and visual patterns, while also being adaptive to varying denoising timesteps. Specifically, such allocation is not one-to-one. For instance, as shown in Figure 2, tokens corresponding to the moon and space station contain substantial detail and are therefore assigned to multiple experts for finer processing. In contrast, tokens representing a plain background may not be assigned to any experts at all. This makes the computation more adaptive and efficient, concentrating resources where they are most needed.

We define a set of indexing vector $\{\mathcal{I}_{i}\mid 1\leqslant i\leqslant E\}$ to filter the input tokens allocated to each expert, *i.e.*,

$$ $\mathcal{I}_{i}=\left\{s\mid\mathbf{G}_{s,i}>0,1\leqslant s\leqslant S\right\}$ (7) $$

The output $\mathbf{x}_{\text{out}}\in\mathbb{R}^{S\times d_{x}}$ of the sparse layer is then obtained by combining the results from each expert using the gating tensor $\mathbf{G}$, *i.e.*,

$$ $\mathbf{x}_{\text{out}}=\sum_{i=1}^{E}\left(\mathbf{G}_{\mathcal{I}_{i},i} \right)^{\top}{\mathcal{E}}_{i}\left(\mathbf{x}^{\prime}_{\mathcal{I}_{i},:} \right).$ (8) $$

Note that equation 6 inherently ensures full utilization of each expert and perfect load balance by design. Therefore, EC-DiT is trained directly via optimizing equation 1, without the need for an additional load-balancing loss (Zoph et al., ). The pseudocode of EC-DiT’s routing strategy is presented in Algorithm 1.

##### Remark:

The proposed EC-DiT shares several key benefits, making it particularly well-suited for DiT-based diffusion models in text-to-image generation tasks.

- •
Adaptive computation. EC-DiT is implicitly aware of the multimodal context and denoising stages. As we will demonstrate in Section 3, EC-DiT effectively enables context-aware routing and dynamically adjusts its computation across both model depth and denoising timesteps.
- •
Global context information. Diffusion models, unlike autoregressive models, process the entire sequence at each denoising step. As described in equation 6, EC-DiT leverages this global context, combining multi-modal information to optimize token-to-expert allocation.
- •
Load balancing. While token-choice routing often requires auxiliary load-balancing loss, EC-DiT ’s routing mechanism inherently ensures balanced utilization of all experts. This prevents bottlenecks and enhances overall efficiency by allocating computations more effectively.

## 3 Evaluations

In this section, we empirically evaluate EC-DiT’s performance across various metrics for text-image alignment and image quality and validate the training and inference efficiency. We then visualize the heterogeneous compute allocation within the proposed EC-DiT. Finally, we present comparative results against dense and token-choice baselines. Throughout the paper, we use the naming convention Dense-<Config> for dense models and EC-DiT-<Config>-<n>E, where <Config> is the model configuration name; <n> is the number of experts.

### 3.1 Experiment Setup

**Table 1: Model size and specifications across four backbone configurations. For Total Params., nE denotes EC-DiT with n experts in a single MoE layer. Activated Params. refer to the number of activated parameters in the forward pass. In Model Arch., #Head is the number of query heads, and #KV represents the number of key/value heads for grouped-query attention (GQA). ^*Note that a trade-off was made in the model depth for the M configuration, where Dense-M has 46 layers, while EC-DiT-M has 38 layers in order to be able to fit into the HBM of training accelerators.**
| Config. | Total Params. | Activated Params. | Model Arch. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dense | 8E | 16E | 32E | 64E | EC-DiT | #Layers | Hidden dim. | #Head | #KV |  |
| XL | 1.47B | 2.51B | 3.70B | 6.08B | – | 1.62B | 28 | 1,152 | 18 | 6 |
| XXL | 2.35B | 4.87B | 7.73B | 13.47B | – | 2.71B | 38 | 1,536 | 24 | 6 |
| 3XL | 4.50B | 10.74B | 17.87B | 32.15B | – | 5.18B | 42 | 2,304 | 36 | 6 |
| M | 8.03B | – | – | – | 97.21B | 8.27B | 38/46^* | 3,072 | 48 | 12 |

##### Model architecture.

Following Chen et al. (), we adopt a modified DiT architecture with additional cross-attention modules for text-to-image generation. For all models evaluated, we use a 670M T5-based (Raffel et al., ) encoder for the text input and a 34M variational autoencoder (VAE) with 8 channels (CLIP-ViT-bigG) for the image input (Hessel et al., ). Both encoders remain frozen during DiT training. The transformer component is configured with four model sizes: XL, XXL, 3XL, and M, as detailed in Table 1.
Starting from the second layer, we scale each dense model by interleavingly replacing all even-numbered dense layers with sparse MoE layers, where each expert is an FFN with dimensions identical to the dense counterpart.
We set the capacity factor $f_{c}=2.0$ throughout the training and inference stages for all sparse models.
We scale the XL-3XL dense models with 8, 16, and 32 experts. To further test the model’s scalability, we scale the M configuration with 64 experts.
To explore the trade-off between model depth and width during scaling, we reduce the number of layers in EC-DiT-M-64E to 38, while maintaining 46 layers in the Dense.
With these settings, our EC-DiT-3XL-32E reaches 32.15 billion parameters, and the largest EC-DiT-M-64E attains 97.21 billion parameters beyond all the current sparse diffusion models (Fei et al., ; Lin et al., ; Balaji et al., ).

##### Training details.

We collect and utilize approximately 1.2 billion text-image pairs from the Internet (McKinzie et al., ; Lai et al., ). The model resolution is set to $256\times 256$, with a patch size of 2. To accelerate training, we employ the masking technique proposed in (Zheng et al., ) with a masking ratio of 0.5. This results in the input sequence length of 128 per image.
Model training is conducted on v4 and v5p TPUs with a batch size 4096.
We use the RMSProp with momentum optimizer (Hinton, ) with a learning rate of 1e-4 and 20K warmup steps. All models are trained with Distributed Data Parallelism (DDP) or Fully Sharded Data Parallel (FSDP) for 800K steps. The expert dimension of EC-DiT is fully shared across the TPU mesh.
To enhance training efficiency, we regroup the first two dimensions of the input sequence (batch_size, sequence_length) into (outer_batch, num_group, group_size). We maintain a group_size of 1024 for training and 512 for inference, which allows EC-DiT to process two images in a sequence at inference time.

##### Evaluation metrics.

To assess the model’s text-to-image capabilities, we primarily use GenEval (Ghosh et al., ) and Davidsonian Scene Graph (DSG) scores (Cho et al., ). GenEval segments and detects objects in the generated samples to evaluate their alignment with the conditioned texts. For DSG, we use an internal version of Gemini to perform Visual Question Answering (VQA) to assess the text-image alignment. To evaluate the image quality of the generated images, we measure zero-shot Fréchet Inception Distance (FID) (Heusel et al., ) along with CLIP Score (Hessel et al., ) on the MS-COCO $256\times 256$ dataset using 30K samples (Lin et al., ). We also provide generated samples from a subset of Partiprompts (Yu et al., ) in Appendix C.

### 3.2 Text-to-image Alignment

**Table 2: GenEval comparison. Our largest scaled model (EC-DiT-M with 64 experts) surpasses all existing models in overall performance and most sub-tasks of the GenEval Metrics (Ghosh et al., ), including the DPO-finetuned version of the current state-of-the-art, SD3 (Esser et al., ) (depth=38). The scores for models preceding ours are sourced from Esser et al. (). The best and second-best entries are highlighted.**
| Model ($\downarrow$) $/$ Score (%) ($\rightarrow$) | Overall | Single obj. | Two obj. | Counting | Colors | Position | Color attr. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SD v1.5 (Rombach et al., ) | 43.00 | 97.00 | 38.00 | 35.00 | 76.00 | 4.00 | 6.00 |
| PixArt-$\alpha$ (Chen et al., ) | 48.00 | 98.00 | 50.00 | 44.00 | 80.00 | 8.00 | 7.00 |
| SD v2.1 (Rombach et al., ) | 50.00 | 98.00 | 51.00 | 44.00 | 85.00 | 7.00 | 17.00 |
| DALL-E 2 (Ramesh et al., ) | 52.00 | 94.00 | 66.00 | 49.00 | 77.00 | 10.00 | 19.00 |
| SDXL (Podell et al., ) | 55.00 | 98.00 | 74.00 | 39.00 | 85.00 | 15.00 | 23.00 |
| SDXL Turbo (Podell et al., ) | 55.00 | 100.00 | 72.00 | 49.00 | 80.00 | 10.00 | 18.00 |
| IF-XL (Saharia et al., ) | 61.00 | 97.00 | 74.00 | 66.00 | 81.00 | 13.00 | 35.00 |
| DALL-E 3 (Shi et al., ) | 67.00 | 96.00 | 87.00 | 47.00 | 83.00 | 43.00 | 45.00 |
| SD3 (Esser et al., ) | 68.00 | 98.00 | 84.00 | 66.00 | 74.00 | 40.00 | 43.00 |
| SD3 (Esser et al., ) w/ DPO | 71.00 | 98.00 | 89.00 | 73.00 | 83.00 | 34.00 | 47.00 |
| Dense-3XL | 68.92 | 99.69 | 86.00 | 69.41 | 81.67 | 20.58 | 56.19 |
| EC-DiT-3XL-32E | 70.91 | 99.64 | 87.88 | 72.53 | 83.84 | 21.19 | 60.40 |
| EC-DiT-M-64E | 71.68 | 99.84 | 88.67 | 73.69 | 85.77 | 21.33 | 60.80 |

We present the comparative evaluation results on GenEval and DSG, both of which are commonly used to measure text-to-image alignment. Table 2 shows the comprehensive GenEval results of EC-DiT compared to current state-of-the-art models. Our largest sparse model, EC-DiT-M-64E, surpasses SD3 in the overall score and most sub-tasks. Notably, the solely pretrained EC-DiT-M-64E even outperforms the SD3 model fine-tuned with DPO. Moreover, our second-largest model, EC-DiT-3XL-32E, achieves comparable alignment capability to SD3 with DPO.
It is worth noting that EC-DiT-M-32E is scaled from a 6B dense model, while the SD3 model listed in Table 2 is an 8B dense model with $512\times 512$ resolution. The difference in resolution may narrow the gap between our models and SD3, as SD3 generally yields higher GenEval scores at higher resolutions (Esser et al., ). This further underscores the potential benefits of scaling dense model with EC-DiT.

Figure: Figure 3: DSG comparison. Scaling up with EC-DiT elevates performance.
Refer to caption: x3.png

We further validate the effectiveness of EC-DiT through DSG scores (Cho et al., ), as depicted in Figure 3. EC-DiT consistently improves DSG performance over the dense variants across different model configurations. Additionally, for each model size, increasing the number of experts further enhances text-image alignment. In both GenEval and DSG evaluations, increasing the model’s capacity with the EC-DiT directly enhances its text-following ability and leads to more accurately aligned image generation. A detailed DSG score breakdown is provided in Appendix A.

### 3.3 Inference-time Efficiency

Figure 5 illustrates the inference efficiency of EC-DiT compared to dense models. Across all base model configurations,EC-DiT consistently delivers improved performance in text-image alignment. The actual inference overhead introduced by EC-DiT ranges from 20% to 28% across different configurations. This practical overhead is greater than the theoretical one of up to 15%, which is calculated in Table 1. For EC-DiT-M, although the theoretical overhead is around 3%, the actual overhead is measured at 23%. This difference might be attributed to the varying efficiency in inference-time parallelism: EC-DiT-M uses model parallelism to fit on $8\times$H100 GPUs, whereas the dense model utilizes FSDP. Despite these factors, scaling with EC-DiT reliably enhances text-to-image alignment up to the state-of-the-art level with less than 30% additional overhead.

Figure: Figure 4: Inference-time efficiency. The circle size is proportional to the total activated parameters. Inference time represents the time elapsed to generate 500 samples on $8\times$H100 GPUs. EC-DiT shows superior performance compared to dense models, with less than 30% additional overhead.
Refer to caption: x4.png

### 3.4 Heterogeneous Compute Allocation

Figure: (a) Allocation per sparse layer at the denoising step $t=40$
Refer to caption: x6.png

In Figure 6, we visualize the heterogeneity of compute allocation through heatmaps that show the number of experts assigned to each image token.
We observe that EC-DiT allocates more computation to areas with clear textual significance, such as major objects or detailed patterns. For example, in Figure 6(a), the main object (such as the moon) and the rendered text receive the most computation. In contrast, the background (composed of nearly monotone colors) receives much lighter computation. This allocation heterogeneity tends to be more pronounced in later layers.
Figure 6(b) also demonstrates that later layers exhibit heterogeneity in fewer denoising steps. We hypothesize that the early layers handle most ”low-frequency” areas, such as the background tokens, and provide context for the later stages. Meanwhile, the routers from later layers more quickly converge to the most textually representative tokens. This coincides with previous observations in pixel space (Lei et al., ).
Overall, EC-DiT ’s compute allocation exhibits high heterogeneity, where a single image token can be processed by up to 44% of the total computation to capture intricate details, while only a few experts process non-detailed areas or even skipped in certain layers. Notably, EC-DiT learns to achieve this adaptive and heterogeneous compute allocation through end-to-end training. We hypothesize that the input to the router, derived from the cross-attention module, likely contains essential cross-modal information from both text and image, which can be effectively leveraged to achieve this adaptive routing.

### 3.5 Comparison with Token-choice Routing

Figure 5 compares EC-DiT with models scaled using token-choice MoE, as proposed in Lepikhin et al. (). This routing strategy has been widely adopted in several sparse diffusion models (Yatharth Gupta, ; Xue et al., ; Fei et al., ). The token-choice baseline uses top-2 routing, which matches the activation size of EC-DiT with a capacity factor of $f_{c}=2.0$.
Our method consistently demonstrates superior training convergence and performance throughout the entire training period.
Notably, EC-DiT with 8 experts rivals the token-choice baseline with 16 experts in both generation quality and text-image alignment, while EC-DiT with more experts significantly outperforms this token-choice baseline.
One reason for the slower training convergence in token-choice routing is the equal compute allocation to different image areas, which makes the compute less adaptive. Detailed image areas may not receive sufficient computation to render complex patterns, while simpler areas waste the computation they receive. Moreover, token-choice routing processes each token independently without considering sequence information, leading to a lack of perception of the overall relationships between local tokens.
Additionally, the inherent load imbalance in token-choice routing results in some experts being overloaded with an excessive number of tokens and creating a training bottleneck (Zhou et al., ). Appendix B provides a comprehensive comparison of EC-DiT and the token-choice baseline regarding GenEval performance. As shown in Table 4, EC-DiT outperforms the token-choice model across all tested configurations, which further validates the advantages of EC-DiT in text-image alignment.

### 3.6 Scaling EC-DiT Models with More Experts

Figure: Figure 7: FID vs. CLIP Score curve during training. The closer a point is to the upper right corner (☆), the better the model performs in terms of higher image quality (lower FID) and text-image alignment (higher CLIP Score). For each model, eight points are plotted from left to right, representing the model’s performance at 100k training step intervals.
Refer to caption: x8.png

From previous results, scaling a dense model with more experts consistently brings performance gains in both convergence and generation quality. In Figure 8, increasing the number of experts leads to slightly improved loss convergence. Figure 3 and Figure 5 collectively demonstrate that adding more experts to the same dense model consistently enhances the model’s text-image alignment capability. Additionally, Figure 5 indicates that increasing the number of experts effectively results in a more powerful model without increasing the activation size nor the inference time, as the expert capacity $C$ is dynamically adjusted based on the number of experts.
In Figure 7, we further examine the trend of FID vs. CLIP Score during training. Despite fluctuations in the curve, the general trend shows that adding more experts shifts the curve towards improved image generation quality and text-image alignment.
Therefore, we suggest increasing the number of experts if the computational budget allows, as this consistently yields performance gains without additional inference overhead.

### 3.7 Training Dynamics

Figure: Figure 8: Training loss of the EC-DiT. The results show a noticeable loss gap between the dense model and the sparse EC-DiT. The training loss and convergence speed is further improved by increasing the number of experts.
Refer to caption: x9.png

We also analyze the training dynamics and convergence of dense models and EC-DiT with varying model configurations and numbers of experts, as shown in Figure 8. Across all model settings, we observe a significant improvement in loss reduction between the dense models and sparse EC-DiT throughout the training period. EC-DiT demonstrates a more effective ability to learn denoising for rectified flow generation and leads to better loss convergence. For example, EC-DiT-XXL models trained with 200K steps achieve a training loss comparable to Dense-XXL models that require around 400K steps. Additionally, the sparse EC-DiT-XXL models eventually stabilize at a lower loss than their dense counterparts.

## 4 Related Work

##### Mixture-of-Experts.

Many MoE models utilize a token-choice routing strategy, where each token independently selects the top-$k$ experts for processing.
Sparsely Gated MoE (Shazeer et al., ) first introduced top-$k$ gating in LSTM models. Such routing strategy was later extended to transformers for language modeling in GShard (Lepikhin et al., ), Switch Transformer (Fedus et al., ), GLaM (Du et al., ), DeepSpeed-MoE (deepspeed), and ST-MoE (Zoph et al., ).
To address potential load imbalances among experts, these token-choice approaches typically incorporate an auxiliary load-balancing loss (Shazeer et al., ).
Alternatively, Zhou et al. () proposed an expert-choice routing in text domain that selects the most suitable text tokens for each expert.
Inspired by this, we adapt and scale up the DiT model using expert-choice routing tailored specifically for diffusion-based generative tasks.
With this approach, each image token from different image areas receives heterogeneous computation allocation. Additionally, the method inherently achieves perfect load balance, as each expert processes an equal number of image tokens, which optimizes both efficiency and performance for the diffusion process.

##### Scaling diffusion models with MoE.

Recent advancements have leveraged MoEs to scale up dense diffusion models for image generation tasks.
Several studies have focused on *class-conditioned image generation*.
DTR (Park et al., ) and Switch-DiT (Park et al., ) approach the routing problem as multitask learning and utilize token-choice MoE for different denoising stages. Similarly, MEME (Lee et al., ) employs an ensemble of denoising experts specialized in distinct timestep ranges.
Furthermore, some models have extended this approach for *text-to-image generation*. ERNIE-ViLG 2.0 (Feng et al., ) and eDiff-I (Balaji et al., ) employ ensembles of specialized experts for different denoising stages. Using U-Net as their backbone, these methods scale up to 24 billion parameters. However, these models activate only one expert at each timestep, which reduces efficiency at large scales.
Recently, some works have shifted towards sparse MoEs. SegMoE (Puigcerver et al., ) employs top-k routing to scale up SDXL (Podell et al., ) with up to four experts. RAPHAEL (Xue et al., ) combines the concepts of distinct timestep experts and token-choice MoEs. DiT-MoE (Fei et al., ) applies token-choice routing to produce a sparse DiT with up to 16 billion parameters. Nevertheless, these methods often suffer from load imbalance issues inherent to the token-choice scheme, requiring an additional auxiliary balancing loss to mitigate this problem.
The most similar work to ours is MoMa (Lin et al., ), which also utilizes expert-choice routing. However, MoMa explores model scaling only on a relatively small scale, with its largest model reaching 7 billion parameters. In contrast, we leveraged the proposed adaptive routing to scale EC-DiT up to *97 billion parameters* and demonstrated the DiT scaling potential on a much larger scope.

## 5 Conclusion and Future Work

We propose EC-DiT, a novel scaling approach for DiT using expert-choice routing. EC-DiT leverages global image information and allocates computation to each image patch adaptive to its pattern complexity. Our model effectively scales up to 97 billion parameters and significantly improves training efficiency, text-to-image alignment, and overall generation quality compared to dense models and sparse DiTs with token-choice routing.

In this work, our approach primarily focuses on adaptive computation based on image information, such as pattern complexity. There may also be additional factors, such as object semantics or compositional relationships, that could further improve generation quality, and we plan to investigate how these additional sources of information can be leveraged to further boost EC-DiT’s performance.

## 6 Ethics Statement

In this paper, we propose a new scaling strategy for text-to-image generation, improving both the quality of generated images and their alignment with given textual inputs. However, this general approach potentially carries ethical concerns. For example, the generative model could be misused to generate harmful or malicious content, such as imagery based on violent or discriminatory text prompts. We acknowledge these concerns and are committed to addressing them by developing more advanced dataset filtering mechanisms and model calibration techniques. Our future work will prioritize increasing the safety and ethical use of text-to-image generation models.

#### Acknowledgments

We would like to thank John Peebles, Xianzhi Du, Chen Chen, Wenze Hu, Riley Roberts, Wei Liu, Zhengfeng Lai, Vasileios Saveris, Peter Grasch, and Yinfei Yang for constructive feedbacks, data curations, and discussions.

## Appendix A Detailed DSG Scores

**Table 3: DSG Score breakdown. An internal version of Gemini was used to assess each sample’s quality through VQA, as proposed in Cho et al. ().**
| Model ($\downarrow$) $/$ Score ($\rightarrow$) | Overall | Counting | Real user | Text | Paragraph | Poses | TIFA-160 | Relation | Defying |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dense-XL | 72.50 | 71.50 | 50.00 | 60.70 | 86.70 | 74.80 | 83.70 | 80.40 | 73.90 |
| EC-DiT-XL-8E | 74.03 | 72.63 | 52.13 | 65.90 | 87.10 | 75.63 | 85.83 | 78.70 | 76.20 |
| EC-DiT-XL-16E | 74.03 | 74.20 | 52.17 | 64.57 | 87.00 | 73.50 | 86.13 | 79.60 | 76.63 |
| EC-DiT-XL32E | 74.58 | 73.53 | 52.20 | 66.33 | 86.58 | 74.60 | 86.85 | 80.48 | 79.15 |
| Dense-XXL | 74.80 | 74.40 | 51.60 | 66.20 | 87.30 | 76.10 | 86.30 | 82.10 | 77.80 |
| EC-DiT-XXL-8E | 75.03 | 74.17 | 53.55 | 67.95 | 86.47 | 73.58 | 87.15 | 81.35 | 78.60 |
| EC-DiT-XXL-16E | 75.63 | 76.33 | 53.60 | 68.52 | 87.40 | 72.80 | 88.18 | 82.40 | 78.60 |
| EC-DiT-XXL-32E | 75.93 | 76.55 | 53.63 | 71.57 | 88.18 | 73.63 | 87.80 | 80.85 | 78.08 |
| Dense-3XL | 75.03 | 74.47 | 52.83 | 70.33 | 87.73 | 69.40 | 86.77 | 83.00 | 78.10 |
| EC-DiT-3XL-8E | 75.60 | 76.43 | 52.23 | 72.60 | 87.80 | 72.00 | 88.38 | 82.13 | 76.55 |
| EC-DiT-3XL-16E | 76.10 | 75.63 | 54.10 | 73.10 | 88.07 | 73.70 | 88.00 | 81.57 | 77.90 |
| EC-DiT-3XL-32E | 76.20 | 78.27 | 54.47 | 71.33 | 87.63 | 73.17 | 87.90 | 82.00 | 78.13 |
| EC-DiT-M-64E | 76.40 | 77.33 | 54.67 | 72.50 | 88.23 | 74.33 | 87.27 | 82.40 | 77.97 |

Table 3 presents the detailed DSG scores for Dense and EC-DiT across various model configurations. Scaling with EC-DiT consistently improves performance in several categories, such as Counting and Text, and leads to better overall performance.

## Appendix B GenEval Comparison with Token-choice Baseline

**Table 4: GenEval comparison with token-choice baselines. EC-DiT (EC) and GShard (GS) are implemented with two base model configurations (XL and XXL). GShard uses top-2 token-choice routing.**
| Model ($\downarrow$) $/$ Score (%) ($\rightarrow$) | Overall | Single obj. | Two obj. | Counting | Colors | Position | Color attr. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dense-XL | 67.08 | 99.80 | 82.47 | 64.28 | 82.46 | 19.61 | 53.86 |
| GS-XL-8E | 67.82 | 99.69 | 81.19 | 68.85 | 80.95 | 20.83 | 55.44 |
| GS-XL-16E | 68.00 | 99.71 | 81.64 | 66.39 | 82.85 | 20.56 | 56.84 |
| GS-XL-32E | 68.52 | 99.92 | 82.88 | 69.18 | 83.23 | 20.30 | 55.63 |
| EC-XL-8E | 68.62 | 99.69 | 83.71 | 68.44 | 81.85 | 19.78 | 58.28 |
| EC-XL-16E | 68.76 | 99.69 | 83.21 | 66.45 | 82.36 | 22.05 | 58.83 |
| EC-XL-32E | 69.38 | 99.79 | 84.14 | 68.61 | 83.68 | 21.02 | 59.03 |
| Dense-XXL | 67.82 | 99.67 | 82.28 | 69.77 | 81.25 | 19.09 | 54.88 |
| GS-XXL-8E | 68.30 | 99.61 | 83.96 | 68.83 | 80.19 | 19.81 | 57.38 |
| GS-XXL-16E | 68.49 | 99.59 | 84.88 | 69.04 | 80.37 | 19.97 | 57.09 |
| GS-XXL-32E | 69.96 | 99.38 | 86.17 | 70.00 | 85.51 | 20.44 | 58.25 |
| EC-XXL-8E | 69.11 | 99.84 | 85.43 | 68.59 | 81.85 | 20.31 | 58.61 |
| EC-XXL-16E | 69.43 | 99.34 | 83.46 | 71.62 | 83.83 | 20.94 | 57.42 |
| EC-XXL-32E | 70.86 | 99.53 | 86.47 | 72.34 | 84.39 | 21.00 | 61.39 |

Table 4 further compares the GenEval performance of EC-DiT with the token-choice baselines (Lepikhin et al., ). By leveraging global image information and adaptive computation, EC-DiT demonstrates superior performance over the token-choice baseline across most evaluation categories.

## Appendix C More Generated Samples

Figure: Figure 9: Generated samples from the Partiprompts subset (Yu et al., ). For each prompt, the model generates four candidates, and the one with the best visual satisfaction is selected.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - We know that DiT performs the diffusion in the latent space instead of the actual image pixel space. Does it undermine the motivation of this paper to allocate more computations to image regions of higher complexity? Please explain how working in latent space affects their approach to computational allocation.
- From Section 2.1, it seems that the authors use the Rectified Flow method on the DiT architecture with MoE. Can the authors explain why they chose to use Rectified Flow and how it integrates with their overall approach? This would help clarify the role of Rectified Flow in their method.
soundness: 3
strengths: - The proposed method is well-motivated. An image typically contains regions of different complexity, which require different computation costs during generation. It would be beneficial to use the MoE method to allocate computation to different image regions adaptively. In Figure 6, the visualization of the number of experts for each token shows that more experts are used for the image tokens corresponding to the objects (compared to the background) on the images. The figure provides good support for the motivation of this work.
- The quantitative scores of EC-DIT show its effectiveness (Table 2). The evaluation includes multiple metrics, including GenEval, Davidsonian Scene Graph (DSG) scores, FID, and the CLIP Score.
- The paper is well-written and the method is clearly explained with a pseudocode provided.
summary: This paper proposes a Mixture-of-Experts (MoE) model, called EC-DIT, for diffusion transformers. By applying the prior work of expert-choice routing to Diffusion Transformer (DiT), EC-DIT allocates a fixed amount of tokens to each expert. As the authors suggest, the method has the advantage that tokens on the more complex parts of images can be assigned to more experts, resulting in higher computation allocated to more important tokens and thus higher efficiency.
weaknesses: - For Table 2, can the authors include the FID metric to compare the generated images' visual quality across the models? 
- Can the authors add a column for inference time or throughput (e.g. images/second) in Table 2? It would help the readers better understand the efficiency of different models.
- This paper has a good contribution to the community by showing the benefits of applying expert-choice routing in DiT, although the main approaches are mainly based on existing works, which makes the proposed method slightly less novel. 
- Can the authors make the notations in Section 2.1 regarding the rectified flow clearer? Please explain $p_0$, $x_0$, $x_t$, and $v_\theta$ in more details.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: see weakness
soundness: 3
strengths: 1. EC-DIT can scale up to 97 billion parameters, achieving a state-of-the-art GenEval score of 71.68% in text-to-image synthesis.
2. The adaptive expert-choice routing is a novel design. It enables efficient compute allocation by having experts select relevant tokens, ensuring load balance without an extra load-balancing loss. This approach better utilizes computational resources, adapting to the complexity of different image regions and text, and improving overall performance and efficiency in handling inputs.
3. By embedding timestep and using cross-attention modules, EC-DIT effectively integrates global information, which allows the model to understand the context of the image and its relationship with the text, guiding compute allocation at different stages. It results in hierarchical processing, enhancing image quality and text-to-image alignment, and making the generation process more context-aware.
summary: This paper focuses on scaling diffusion transformers for text-to-image synthesis. It proposes EC-DIT, a new family of Mixture-of-Experts (MoE) models with adaptive expert-choice routing. By leveraging the computational heterogeneity of image generation, EC-DIT adaptively allocates compute to understand input texts and generate image patches, enabling heterogeneous computation aligned with text-image complexity. It scales up to 97 billion parameters and shows significant improvements in training convergence, text-to-image alignment, and overall generation quality compared to dense and conventional MoE models. Its contributions include introducing EC-DIT for adaptive computation in text-to-image synthesis, achieving promising performance at scale with faster loss convergence and better quality, and conducting comprehensive experiments to validate its superiority and demonstrate effective adaptive compute allocation.
weaknesses: 1. The best score for SD3 (Esser et al., 2024) w/ DPO is 74% in the original paper, which is actually higher than that of EC-DIT (71.68%). Even with the value 71% listed in Table 2,  EC-DIT is only gaining marginally. Given its additional overhead, it is hard to see the value of the gain. 
2. The difference between the theoretical and actual inference overhead (e.g., for EC-DIT-M, the theoretical is around 3% but the actual is 23%) indicates that there may be room for optimization in the inference process. The authors may want to explore the factors contributing to this overhead or propose specific strategies for mitigating it, which could further enhance the practical applicability of the model, especially for applications where real-time or fast inference is crucial.
3. The generalizability of the adaptive expert-choice routing is unclear. The performance of EC-DIT is mainly demonstrated in the context of text-to-image synthesis. How would it fit into the task of pre-training mixed-modal, early-fusion language models? It would be interesting to explore its effectiveness and efficiency in other related tasks. 
4. This paper could benefit from more detailed ablation studies on specific components of the proposed model. For example, understanding the individual contributions of the timestep embedding, cross-attention modules, and the expert-choice routing mechanism in more detail. This lack of in-depth analysis makes it harder to understand which parts are truly driving its performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: While efforts have been made to enhance the scalability of diffusion models for improved performance and efficiency, several issues remain:
i) The novelty of this approach requires clarification, especially in comparison to existing diffusion models incorporating MoE, as detailed in the related work section (lines 057-063).
ii) Certain claims should be refined or revised. For instance, the statement: “our approach significantly improves training convergence, text-to-image alignment, and overall generation quality compared to dense and sparse variants with conventional token-choice MoEs (Lepikhin et al., 2020b; Du et al.,2022; Zoph et al., 2022).” First, I do not see the comparison with these methods in experiments; Second, the performance is only comparable with SOAT, thereby questioning the usage of 'significantly.'; Third, the reason of improvement might be mainly attributed to increased model parameters, e.g., in experience, more parameters make the training converge faster. Thus, deeper analysis could be given.
“EC-DIT leverages this global context, combining multi-modal information to optimize token-to-expert allocation.” multi-modal?
“The model resolution is set to 256 × 256, with a patch size of 2.” “This results in the input sequence length of 128 per image.” Patch size is 2*2?
iii) Those existing diffusion models with MoE should be compared in experiments, if they are available.
iv) Fig. 3 only shows the performance on DSG score. Additional evaluation on other metrics should be showed, as the performance across various metrics might exhibit different trends. 
v) Fig. 4 should show more comparisons with other methods (e.g., in Table 2), rather than solely focusing on the proposed methods themselves. 
vi) How to set C in Top-k? There lacks the comparison with the different C values.
vii) The performance is only comparable with SOAT methods, such as SD3. In fact, the parameter quantities are in the same magnitude, SD3 (8B) vs EC-DIT-M-64E(>8B, observed in Fig. 4). Thus, this raises questions about the core contributions of this work. Is the primary focus acceleration? If so, there lacks many comparisons to current methods.
soundness: 2
strengths: i) The technique line is reasonable.
ii) The writing is easy to follow.
summary: This paper proposes a MoE based diffusion transformer framework to scale diffusion models. The basic idea is to combine DIT and MoE. The experimental results demontrate the effectiveness.
weaknesses: i) This work should be incremental in essence, as the basic idea is to combine DIT and MoE.
ii) Certain statements exhibit overclaims that need refinement. (see Questions in detail)
iii) Crucial experiment comparisons with diffusion methods utilizing MoE are notably absent. (see Questions in detail)

The rebuttal addresses my concerns, so I raise the score.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses part.
soundness: 3
strengths: - EC-DiT successfully scales the DiT model to 97 billion parameters, demonstrating exceptional performance.
- Expert-choice routing outperforms traditional token-choice routing strategies while maintaining competitive inference speed.
- The paper presents extensive experiments to showcase EC-DiT’s capabilities.
summary: This paper explores how to scale diffusion transformers (DiT) to larger model capacities. The authors developed EC-DiT, a MoE-based (Mixture of Experts) DiT model leveraging expert-choice routing, expanding the DiT model parameters to 97 billion. EC-DiT enables context-aware routing and dynamically adjusts computation, inherently ensuring load balancing across experts. Extensive experiments were conducted to verify and analyze the effectiveness of EC-DiT.
weaknesses: - While I recognize the authors’ contribution in scaling up DiT and support the paper’s acceptance, I believe it does not demonstrate sufficient technical innovation beyond prior work, appearing more as a practical integration of existing techniques
- A clearer and more precise definition of ‘Activated Params.’ should be provided. In Algorithm 1, it appears that all parameters are activated. Does this refer to the average number of parameters activated per token?
- I would like the authors to include a comparison of memory usage during training and inference.
- It would be beneficial if the authors could open-source the relevant scripts and model weights upon acceptance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: 1. Investigating methods to scale up diffusion transformers is a crucial research direction, as it directly impacts the effectiveness and potential of large-scale text-to-image synthesis models. 

2. The proposed EC-DIT model's effectiveness is verified by quantitative metrics.
summary: The paper presents EC-DIT, a new Mixture-of-Experts (MoE) model specifically designed for diffusion transformers in text-to-image synthesis. EC-DIT leverages computational heterogeneity through expert-choice routing, which enables the model to allocate computational resources adaptively based on the complexity of the input text and the corresponding image patches. This approach supports scaling up to 97 billion parameters, leading to improvements in text-to-image alignment, and generation quality.
weaknesses: 1. As shown in Figures 1 and 9, the generated images are relatively simple and lack sufficient complexity. For more intricate images, the aesthetic quality appears subpar, and the benefits of scaling up are not convincingly perceptible. Offering 64-128 more generated images with more complex scenes could better demonstrate the value of the proposed method.

2. Models like SD3 and Flux, which have significantly fewer parameters than EC-DIT-64E, manage to produce images with notably superior visual quality. Please clarify why this discrepancy exists and address what specific advantages scaling up provides if the visual quality is still lacking. This raises questions about the true value that scaling up offers in terms of image quality.

Meta-review: In this paper, authors propose mixture of experts for diffusion models. Specifically, a new family of Mixture-of-Experts (MoE) models with adaptive expert-choice routing is proposed. By leveraging the computational heterogeneity of image generation, EC-DIT adaptively allocates compute to understand input texts and generate image patches, enabling heterogeneous computation aligned with text-image complexity. Experimental results show good qualitative and quantitative performance improvement.

Decision: Accept (Poster)