Kati8/ICLR_2023_2025 huggingface dataset:
authors: Haotian Sun
authors: Tao Lei
authors: Bowen Zhang
authors: Yanghao Li
authors: Haoshuo Huang
authors: Ruoming Pang
authors: Bo Dai
authors: Nan Du
reviews: 
Comments: The paper presents EC-DIT, a new Mixture-of-Experts (MoE) model specifically designed for diffusion transformers in text-to-image synthesis. EC-DIT leverages computational heterogeneity through expert-choice routing, which enables the model to allocate computational resources adaptively based on the complexity of the input text and the corresponding image patches. This approach supports scaling up to 97 billion parameters, leading to improvements in text-to-image alignment, and generation quality.
Strengths: 1. Investigating methods to scale up diffusion transformers is a crucial research direction, as it directly impacts the effectiveness and potential of large-scale text-to-image synthesis models. 

2. The proposed EC-DIT model's effectiveness is verified by quantitative metrics.
Weaknesses: 1. As shown in Figures 1 and 9, the generated images are relatively simple and lack sufficient complexity. For more intricate images, the aesthetic quality appears subpar, and the benefits of scaling up are not convincingly perceptible. Offering 64-128 more generated images with more complex scenes could better demonstrate the value of the proposed method.

2. Models like SD3 and Flux, which have significantly fewer parameters than EC-DIT-64E, manage to produce images with notably superior visual quality. Please clarify why this discrepancy exists and address what specific advantages scaling up provides if the visual quality is still lacking. This raises questions about the true value that scaling up offers in terms of image quality.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper explores how to scale diffusion transformers (DiT) to larger model capacities. The authors developed EC-DiT, a MoE-based (Mixture of Experts) DiT model leveraging expert-choice routing, expanding the DiT model parameters to 97 billion. EC-DiT enables context-aware routing and dynamically adjusts computation, inherently ensuring load balancing across experts. Extensive experiments were conducted to verify and analyze the effectiveness of EC-DiT.
Strengths: - EC-DiT successfully scales the DiT model to 97 billion parameters, demonstrating exceptional performance.
- Expert-choice routing outperforms traditional token-choice routing strategies while maintaining competitive inference speed.
- The paper presents extensive experiments to showcase EC-DiT’s capabilities.
Weaknesses: - While I recognize the authors’ contribution in scaling up DiT and support the paper’s acceptance, I believe it does not demonstrate sufficient technical innovation beyond prior work, appearing more as a practical integration of existing techniques
- A clearer and more precise definition of ‘Activated Params.’ should be provided. In Algorithm 1, it appears that all parameters are activated. Does this refer to the average number of parameters activated per token?
- I would like the authors to include a comparison of memory usage during training and inference.
- It would be beneficial if the authors could open-source the relevant scripts and model weights upon acceptance.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes a MoE based diffusion transformer framework to scale diffusion models. The basic idea is to combine DIT and MoE. The experimental results demontrate the effectiveness.
Strengths: i) The technique line is reasonable.
ii) The writing is easy to follow.
Weaknesses: i) This work should be incremental in essence, as the basic idea is to combine DIT and MoE.
ii) Certain statements exhibit overclaims that need refinement. (see Questions in detail)
iii) Crucial experiment comparisons with diffusion methods utilizing MoE are notably absent. (see Questions in detail)

The rebuttal addresses my concerns, so I raise the score.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 4
reviews: 
Comments: This paper proposes a Mixture-of-Experts (MoE) model, called EC-DIT, for diffusion transformers. By applying the prior work of expert-choice routing to Diffusion Transformer (DiT), EC-DIT allocates a fixed amount of tokens to each expert. As the authors suggest, the method has the advantage that tokens on the more complex parts of images can be assigned to more experts, resulting in higher computation allocated to more important tokens and thus higher efficiency.
Strengths: - The proposed method is well-motivated. An image typically contains regions of different complexity, which require different computation costs during generation. It would be beneficial to use the MoE method to allocate computation to different image regions adaptively. In Figure 6, the visualization of the number of experts for each token shows that more experts are used for the image tokens corresponding to the objects (compared to the background) on the images. The figure provides good support for the motivation of this work.
- The quantitative scores of EC-DIT show its effectiveness (Table 2). The evaluation includes multiple metrics, including GenEval, Davidsonian Scene Graph (DSG) scores, FID, and the CLIP Score.
- The paper is well-written and the method is clearly explained with a pseudocode provided.
Weaknesses: - For Table 2, can the authors include the FID metric to compare the generated images' visual quality across the models? 
- Can the authors add a column for inference time or throughput (e.g. images/second) in Table 2? It would help the readers better understand the efficiency of different models.
- This paper has a good contribution to the community by showing the benefits of applying expert-choice routing in DiT, although the main approaches are mainly based on existing works, which makes the proposed method slightly less novel. 
- Can the authors make the notations in Section 2.1 regarding the rectified flow clearer? Please explain $p_0$, $x_0$, $x_t$, and $v_\theta$ in more details.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3