RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
Authors: Zeyue Xue
Guanglu Song
Qiushan Guo
Boxiao Liu
Zhuofan Zong
Yu Liu
Ping Luo
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
Text-to-Image Generation

Abstract: Text-to-image generation has recently witnessed remarkable achievements. We introduce a text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by  stacking tens of mixture-of-experts (MoEs) layers, i.e., space-MoE  and time-MoE layers,  enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting a particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal.  Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. Secondly, a single model with three billion parameters, trained on 1,000 A100 GPUs for two months, achieves a state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers of image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https://raphael-painter.github.io/.

RAPHAEL: Text-to-lmage Generation via Large Mixture of Diffusion Paths
Zeyue Xue
Guanglu Song"
The University of Hong Kong
Sense Research
xuezeyue@connect. hku.
songguangguu@emmsciinn..ommm com
Qiushan Guo
Boxiao Liu
Zhuofan Zong
The University of Hong Kong qsguo8cs. hku. hk
SenseTime Research
SenseTime Research
litboziao@sensetime.. com
zongzhuofan@@mail.c com com
Yu Liu
Ping Luo!
SenseTime Research liuyuisanal @gmail. com
The University of Hong Kong
Shanghai AI Laboratory
pluoêes .ku. hk
"When one is painting one does not think."
Raffaello Sanzio da Urbino
Abstract
Text-to-image generation has recently witnessed remarkable achievements. We introduce text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by stacking tens of mixture- of of-experts (MoEs) layers, i.e., space-Mof and time-Mot layers, enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEI. outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2.0. DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, eyberpunk, and ink illustration. Secondly, single model with three billion parameters, trained on 1. 000 A100 GPUs for two months, achieves state-of the-art state-of-the-art zero-shot FID score
6.61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https:
/ printerel-pinterr
painter centtterggittie github.io///
Equaal contribution. Work done during Zeyue's internship at SenseTime Research.
Project lead.
*Corresponding authors.
'More creations can be found in https://miaohua htape///mmsaouuu.ssssttitte cww/zh- http://hh-//iiccreeeeeeeeee wttp://picctureeeeeeeeioooto selection Please select the Artist v0.3.5 model to generate. This latest version based RAPHAEI. This in- formation was last updated on 16th, 2023.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

parrot with pearl earring, style.
playing soccer, digital art.
A Pikachu with an ongry expression and red eyes, with lightning around hyper realistic style.
There five in the street.
Street shot of a fashionable
Chinese lady in Shanghal, wearing black high- trousers.
Moonlight Maiden, girl in school uniform, long white
standing under
the moon, cellulaid
style, Japonese manga style.
Half humon, half robot, repaired
human,
flesh warrior, mech display, man mech, cyberpunk.
O 06
RAPHACK
Raahe
that RAPHAEL.
RAPHAEL
RAPEEL
RAPHAET.
StableDiffusion XI.
DeepFloyd
DALL-E2
FRNTF VII. G 2.0
Figure 1: Comparisons of RAPHAEL with recent representative generators, Stable Diffusion XL (2). Deep- Floyd, DALL-E 2 3 and ERNIE-ViLG 2.0 [ They are given the same prompts, where the words that the human artists yearn preserve within the generated images highlighted in red. These images cherry-picked. We see previous models often fail to preserve the desired concepts. For example, only aegnerr-geeerteed images precisely reflect the prompts such as "pearl earring. Vermeer", "playing soccer", "five cars", "black high-was waisted trouser". "white hair, manga, moon". and "sign, RAPHAEL", while other models generate compromised results. Better zoom in 200%.
Introduction
Recent advancements in text-to-image generators, such as Imagen (1) Stable Diffusion (2]. DALL- 2 B3, eDiff-I [4), and ERNIE- ViLG 2.0 [5), have yielded remarkable success and found wide applications in computer graphics, culture and art, and the generation of medical and biological data.
Despite the substantial progress made text-to-image diffusion models pressing need for research to further achieve more precise alignment between text and image. As
there remains

---

illustrated in [ existing models often fail to adequately preserve textual concepts within the generated images. This is primarily due the reliance on classic cross- coss-attention mechanism for integrating text descriptions into visual representations, resulting in relatively coarse control of the diffusion process, and leading to compromised results.
To address this issue, we introduce RAPHAEL, a text-to-image generator, which yields images with superior artistry and fidelity compared to prior work, as demonstrated in 2 RAPHAEL, an acronym that stands for "distinct image regions align with different text phases in attention learning", offers an appealing benefit not found in existing approaches.
Specifically, we observe that different text concepts influence distinct image regions during the generation process [6]. and the conventional aoss-attention layer often struggles to preserve these varying concepts adequately in an image. To mitigate this issue, we employ diffusion model stacking tens of mfixture-of-exper layers. Concretely, the space-MoE layers are responsible for depicting different concepts in specific image regions, while the time-MoE layers focus on painting these concepts different diffusion timesteps.
(MoE) layers 7 78, including both space-MoF and time-MoF.
This configuration leads to billions of diffusion paths from the network input to the output. Naturally, each path can act as a "painter" responsible for rendering a particular concept to an image region at specific timestep. The result is more precise alignment between text tokens and image regions, enabling the generated images that accurately represent the associated text prompt. This approach sets RAPHAEL apart from existing models and even sheds light on future studies of the explainability of the generation process. Additionally, we propose an edge-supervise learning module to further enhance the image quality and aesthetic appeal of the generated images.
Extensive experiments demonstrate that RAPHAEL outperforms preceding approaches, such as Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2. (1) RAPHAEL exhibits superior performance switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. (2) RAPHAEL establishes
with zero-shot FID-30k score of 6.61 on the COCO dataset. (3) RAPHAEL, a single model with three billion parameters trained on 1, 000 A100 GPUs, significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark.
state-of-the
The contributions of this work are three-fold: (i) We propose a novel text-to-image generator, RAPHAEL, which, through the implementation of several carefully-desinnee techniques, generates images that more accurately reflect textual prompts than previous works. (ii) We thoroughly explore RAPHAEL's potential for switching images in diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration, and for extension using LoRA [), ControlNet [10], and SR-GAN [Τ. (iii) We will release a programming API for RAPHAEL to the public. We believe that RAPHAEL holds the potential to advance the frontiers of image generation in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field.
2 Notation and Preliminary
We present the necessary notations and the Denoising Diffusson Probabilistic Model (DDPM) [22 for text-to-image generation. Given a collection of N images, denoted as [x,],N the aim is to learn generative model, p(x), that is capable of accurately representing the underlying distribution.
In forward diffusion, Gaussian noise is progressively introduced into the source images. At an arbitrary timestep it is possible directly sample from the Gaussian distribution following the T-step noise schedule {a:)Tmm without iterative forward sampling. Consequently, the noisy image at timestep t, denoted as can be expressed as X2 = vI a,X0 + Vāzez where ā = -'' a; In this expression, Xo represents the source image, while E, NO, 1) indicates the Gaussian noise at step t. In the reverse process, denoising neural network, denoted as Do(-), is employed to estimate the additive Gaussian noise. The optimization of this network is achieved by minimizing the loss
function, Ldenoise (o,,,,)) [|ee Do (x,, t]12
By employing the Bayes' theorem, it is feasible to iteratively estimate the image at timestep t 1 through sampling from the posterior distribution, po(xx-1(xx). We have X
3

---

Harvest of vegetobles in a wooden box near the beds vegetables grow naturally, summer light background, backlight and rays, clean sharp focus.
Chinese illustration, orientol landscape painting, above super wide angle, magical, romantic, detailed, colorful, multi- dimensional paper kirigami craft.
Photography closeup portrait adorable rusty broken-down steompunk covered budding vegetation, surrounded by tall grass, misty futuristic sci-fi forest environment.
cute matte low poly Isometric Zelda Breath of wild forest island, woterfolls, soft shadows, trending on Artstation, render, fez video game.
Goddess high fashion, impressionistic line ort, controsting earth tones, vibrant, pen and illustration, splatter, obstract expressionism superimposed onto majestic space queen.
The Coped Crusoder, Gotham skyline, rooftop, mysteriaus, powerful, nighttime, mixed media, expressionism, dark tones, high contrast, style modern, gritty and textured, collage technique.
beautiful woman dressed dress made of outumn leaves in the forest, photography, natural lighting, high detail.
wizard by Hayoshido in the style of Dorohedoro for Elden Ring, with biggest most intricate sword, an sunlit battlefield, breath of the wild, striking illustration.
Milkyway in a glass bottle, 4k, unreal engine, octane render.
These examples show that RAPHAEL can generate artistic images with varying text prompts various styles. The synthesized images have details and semantics. The prompts were written by human artists without cherry-pickingg
Figure 2:
1-a, - -T-a, Do (x,, (x,, t)) + o22, where a, signifies the standard deviation of the newly injected noise the image each step, and represents the Gaussian noise.
Vai X4 -
In essence, the denoising neural network estimates the score function at varying time steps, thereby progressively recovering the structure of the image distribution. The fundamental insight provided by the DDPM lies in the fact that the perturbation of data points with noise serves to populate regions of low data density, ultimately enhancing the accuracy of estimated scores. This results in stable training and sampling.
U-Net with Text Prompts. The denoising network is commonly implemented using U-Net 13 architecture, as depicted in FigBin Appendix 7.3 To incorporate textual prompts (denoted by y) into the U-Net, text encoder neural network, Eg is employed to extract the textual representation.

---

(a) Transformer Block
(b) Spuce-Mol
Festures
Lydgue
A furry bour under
Purry
Expert
N Conv
Text Gatee
Ma
under sky
Alln
Rear
Expert
Self Attention
Cross Attention
Time MoE
Space MoE
Sky
Expert
M.,
M,
Figure Framework of RAPHAEL.. (a) Each block contains four primary components including a self- attention layer, a cross-attention layer, space-MoE layer, and a time-MoE layer. The space-MoE is responsible for depicting different text concepts in specific image regions, while the time-MoE handles different diffusion timesteps. Each block uses edge-supervised cross-attention learning further improve image quality. (b) shows details space-MoE. For example, given a prompt "a furry bear under sky", cach text token and its corresponding image region (given by binary mask) are directed through distinct space experts, i.e., cach expert learns particular visual features at region. By stacking several space-MoEs, we can easily learn to depict thousands of text concepts.
The extracted text tokens are input into the U-Net through a cross-attention layer. The text tokens possess size of ny dy. where Tly represents the number of text tokens, and dy signifies the dimension of a text token (e.g., dy 768 in [4)..
The cross-attention layer can formulated as abtention Q, K, V) = softmax
V, where Q, K, and V correspond the query, key, and value matrices, respectively. These matrices computed as Q (xx) W K (y) w and EE where W dddd and Why, Wval E Rd, represent the parametric projection matrices for the image and text, respectively, Additionally, denotes the dimension of an image token, h(x,) R= indicates the flattened intermediate representation within the U-Net, with nh being the number of tokens in an image. A
QK vd
QKT
cross-attention map between the text and image, M softmax
vd E I X"s is defined,
which plays crucial role in the proposed approach, described in the following sections.
3 Our Approach
The overall framework of RAPHAEL is illustrated in Fig with the network configuration details provided the Appendix 7.. transformer blocks, each containing four components: self-attention layer, cross-attention layer, space-MoF layer, and a time-MoB layer. The space-MoF responsible for depicting different text concepts in specific image regions at given scale, while the time-MoB handles different diffusion timesteps.
Employing a U-Net architecture, the framework consists 16
3.1 Space-MoE and Time-MoE
Space-MoE. Regarding the space-MoE layer, distinct text tokens correspond various regions within an image, as previously mentioned. For instance, when provided with the prompt "a furry bear under the sky", each text token and its corresponding image region (represented by binary mask) are fed into separate experts, as illustrated in Fig The Space-MoE layer's 's output is the
mean of all experts, calculated using the following formula: 1 Σn" Croutofys) (xx) o M.). In
this equation, M, is binary two-dimensional matrix, indicating the image region the i-th text token should correspond to, as shown in Fig 3p. Here, represents hadamard product, and h (xx) is the features from time-MoE. The gating (routing) function route(y;) returns the index of expert in the space-MoE, with {e1, , e being a set of A experts.
Text Gate Network. The Text Gate Network is employed to distribute image region specific expert, as shown in Fig The function route(y,) = argmax (softmax (g (Eş(y,)) + <)) is used, where G RRd + & is a feed forward network, which uses text token representation Fyly.) as input and assigns space expert. To prevent mode collapse, random noise is incorporated. The
5

---

Expert Indie
Equatindo:
asoollane
banana
cale
cheir
domuu
horse
laptop
4
ocange
pizza
train
Blocktodie
We visualize the diffussin paths (routes) from the network input to the output, utilizing 16 space-MoE layers, each containing spatial experts. These paths closely associated with 100 adjectives, such as "scemic", "peaccful", and "majestic", which represent the most frequently occurring adjectives for describing artworks as suggested by GPT-3.5 [T5] [6]. Given that GPT-3.5 been trained on trillions of tokens, we believe that these adjectives reflect a diverse, real-world distribution. Our findings indicate that different paths distinctively represent various adjectives. Right: We depict the diffusion paths for ten categories (i.e., nouns) within the COCO dataset. Our observations reveal that different categorics activate distinct paths in a heterogeneous manner. The display colors blend together where the routes overlap.
Figure 4: Left:
argmax function ensures that one expert exclusively handles the corresponding image region for each text token, without increasing computational complexity.
From Text Image Region. Recall that M is the cross- cess-attention map between text and image, where each element, Mj. represents correspondence value between the j-th image token and the i-th
text token, In the space-Mofe each entry in the binary mask M, equals "1" if Mj,; 2 7h. otherwise "O" if M, < 7%. as illustrated in Fig 3p. A thresholding mechanism is introduced to determine the values in the mask. The threshold value 77, max(M,,,) is defined, where max(M...) represents the maximum correspondence between text token i and all image regions. The hyper-parameter will be evaluated through an ablation study.
The insight behind the space-MoE is to effectively model the intricate relationships between text tokens and their corresponding regions in the image, accurately reflecting concepts the generated images. As illustrated in Fig 4 the employment of 16 space-MoE layers, each containing experts, results in billions of spatial diffusion paths (i.e., 616 possible routes). It is evident that each diffusion path is closely associated with specific textual concept.
Discussions.
To investigate this further, generate 100 prevalent adjectives that are the most frequently occurring adjectives for describing artworks as suggested by GPT-3.5 [15] 16). Given that GPT-3.5 has been trained on trillions of tokens, posit that these adjectives reflect diverse, real-world distribution. We input each adjective into the RAPHAEL model to generate 100 distinct images and collect their corresponding diffusion paths. Consequently, we obtain treating these pathways as features (i.e., each path is vector of 16 entries), we train straightforward classifier (e.g., XGBoost (17) to categorize the words. The classifier after 5-fold cross-validation achieves over 93% accuracy for open-world adjectives, demonstrating that different diffusion paths distinctively represent various textual concepts. We observe analogous phenomena within the 80 object categories of the COCO dataset. Further details on verbs and visualization are provided in the Appendix 7.5
thousand paths for the 100 words. By
Time-MoF. We can further enhance the image quality by employing a time-mixture--f-exxpee
(time-MoF) approach, which is inspired by previous works such 5 Given that the diffusion process iteratively corrupts an image with Gaussian noise over a series of timesteps f = 1,...,7 the image generator is trained timesteps aim to denoise a noisy image, progressively transforming random noise into an artistic image. Intuitively, the difficulty of these denoising steps varies depending on the noise ratio presented in the image. For example, when t = T, the denoising network's 's input image x, is highly noisy. When the image x; is closer the original image.
denoise the images reverse order from T
All
To address this issue, we employ a time-MoE before each space-MoE in each transformer block. In contrast to 4 which necessitate hand-crafted time expert assignments, implement additional gate network to automatically learn to assign different timesteps to various time experts. Further details can be found in the Appendix 7.3

---

3.2 Edge-supervised Learning
In order to further enhance the image quality, we propose incorporating an edge- suge-superviied learning strategy train the transformer block. By implementing an edge detection module, aim to extract rich boundary information from an image. These intricate boundaries can serve as supervision to guide the model in preserving detailed image features across various styles.
Consider a neural network module, Po (M), with parameters of N convolutional layers (e.g, N = 5). This module is designed to predict edge map given an attention map M (refer to Fig 7 in the Appendix 7.2). We utilize the edge map the input image, denoted as Lddge to supervise the network Po. Ledge can be obtained by the holistically-nese nestedd edge detection algorithm [18] (Fig 7p). Intuitively, the network Po can be trained by minimizing the loss function, Ledge = Focal (Po (M), Ledge), where -.call..-) denotes the focal loss [19] employed to measure the discrepancy between the predicted and the "ground-truth" edge maps. Moreover, as discussed in 5 6). the attention map M is prone to becoming vague when the timestep i. is large. Consequently, it is essential adopt timestep threshold value to inactivate (pause) edge-supervised learning when t is large. This timestep threshold that will be evaluated through an ablation study.
value (T)) is a
Overall, the RAPHAEL model is trained by combining two loss functions, L Ldenoise + Lodge= As demonstrated in Fig 7 in the Appendix 7.2 edge-supervised learning substantially improves the image quality and aesthetic appeal of the generated images.
4 Experiments
This section presents the experimental setups, the quantitative results compared to recent state-of- the-art models, and the ablation study to demonstrate the effectiveness of RAPHAEL. More artistic images generated by RAPHAEL and comparisons between RAPHAEL and other diffusion models can be found in Appendix 7.6 and 7.7
Dataset. The training dataset consists of subset of LAION-SB 20 and some internal datasets, including 730M text-images pairs in total. To collect training data from LAION-5B, we filter the images using the aesthetic scorer same as Stable Diffusion (2] and remove the image-text pairs that have scores smaller than 4.7. We remove the images with watermarks either. Since the text descriptions in LAION-5B are noisy, we clean them by removing useless information such as URLs, HTML tags, and email addresses, inspired by 21
Multi-scale Training. To improve text-image alignment, instead of cropping images fixed scale 2. we resize an image to its nearest size into different buckets, which has 9 different image scales. Additionally, the GPU resources will be automatically allocated to each bucket depending on the number of images it contains, enabling effective use of computational resources]]
Variational Autoencoder (VAE) 22 23 to compress images using Latent Diffusion Model (2]. We first pre-train an image encoder transform image from pixel space latent space, and image decoder convert it back. Unlike previous works, the cross-attention layers in RAPHAEL are augmented with space-MoE and time-Moß layers. The entire model is implemented in PyTorch (24), and is trained by AdamW (25) optimizer with learning rate of le - 4, a weight decay of O, a batch size of 2, 000, on 1, 000 NVIDIA 1100s for two months. More details on the hyper-parametter settings can be found in the Appendix
Implementations. To reduce training and sampling complexity,
7.1
4.1 Comparisons
Results on COCO. Following previous works [ 2 41, we evaluate RAPHAEL on the COCO 256 x 256 dataset using zero-shot Frechet Inception Distance (FID), which measures the quality and diversity of images. Similar the validation set for evaluation. Table shows that RAPHAEL achieves new state-of-the of-th-of-the-rt
32 30, 000 images are randomly selected from
"The dimensions each bucket are as follows: [448, 832]. [512, 768). [512, 704], [640, 640], [576, 640), [640, 576). [704, 512]. [768, 512], and [832, 448). For instance, when images are resized, those with an aspect ratio of 1.0 will be assigned to the bucket of size [640, 640]. GPUs will be allocated to each bucket, based on the images it contains. All GPUs will have the same batch size and will select images from its associated bucket.
7

---

Table 1: Comparisons RAPHIAEL
recent representative text-lo-image generation models MS-COCO 256 x 256 using zero-shot FID-30k. We see that RAPHAEL outperforms all previous works in image quality, even commercial product released recently.
Approach
Venue/Date
Model Type
FID-30K
Zero-shot FID-30K
DF-GAN
20
CVPR 22
GAN
21.42
DM-GAN
CVPR119
GAN
20.79
LAHITE
28
CVPR'22
GAN
8.12
Make-A-Scene Scene
20
BCCV'22
Autoregressive
7.55
LDM 2
CVPR'22
Diffusion
12.63
GLIDE 30
ICML'22
Diffusion
12.24
DALL-E
arXiy, April 2022
Diffusion
10.39
GigaGAN
31
CVPR'23
GAN
9,09
Stable Diffusion
CVPR '2
Diffusion
8.32
Muse-3B B
32
arXiv, Jan. 2023
Non-Autorege Kon-Autoregressive
7.88
Imagen 1
NeurIPS'22 22
Diffusion
7.27
eDiff-I
4
arXiv, Nov. 2022
Diffusion Experts
6.95
ERNIE- 2.0 5
CVPR'25 23
Diffusion Experts
6.75
DeepFloyd
Product, May 2023
Diffusion
6.66
RAPHAEL
Diffusion Experts
6.61
100%
SAPINEL
BALEE
100%
SAPIAKL
100%
MARVAL
100%
Merizz
Ceepfiod
50%
50%
50%
Alignment
Quality
Alignment
Quality
Alignment
Quality Ity
Alignment
Quality
Figure 5: Comparisons of RAPIIALL with DALL-E 2, Stable Diffussin XL (SD XL). ERNIE ViLG 2.0, and DeepFloyd user study using the ViLG-300 benchmark. report the user's preference rates with 95% confidence intervals. We see that RAPHAEL can generate images with higher quality and better conform to the prompts.
performance of text-to-image generation, with 6.61 zero-shot FID-30k MS-COCO, surpassing prominent image generators such as Stable Diffusion, Imagen, ERNIE-ViLG 2.0, and DALL-E 2.
Human Evaluations. We employ the ViL.G-300 benchmark [ bilingual prompt set, which enables to systematically evaluate text-to-image models given various text prompts in Chinese and English. ViLG-300 allows us convincingly compare RAPHAEL with recent-advancee models including DALL-E 2, Stable Diffusion, ERNIE-ViLG 2.0, and DeepFloyd, in terms both image quality and text-image alignment. For example, human artists presented with two sets of images generated by RAPHAEL and competitor, respectively. They are asked to compare these images from two aspects respectively, including image-text alignment, and image quality and aesthetics. Throughout the entire process, human artists are unaware of which model the image is generated from. Fig 5 shows that RAPHAEL surpasses all other models in both image-text alignment and image quality in the user study, indicating that RAPHAEL can generate high-artistry images that conform the text.
Extensions to LoRA, ControlNet, and SR-GAN. RAPHAEL can be further extended by incorporat- ing LoRA, ControlNet, and SR-GAN. In Appendix 7.8 present comparison between RAPHAEL and Stable Diffusion utilizing LoRA. RAPHAEL demonstrates superior robustness against overfitting compared Stable Diffusion. We also demonstrate RAPHAEL with canny-based NentrolNet. Furthermore, by employing a tailormade SR-GAN model, we enhance the image resolution to 4096 6144.
4.2 Ablation Study
Evaluate every module RAPHAEL. We conduct a comprehensive assessment of each module within the RAPHAEL model, utilizing the CLIP [4] score to measure image-text alignment. Given the significance of classifier-free guidance weight controlling image quality and text alignment, present ablation results trade-off curves between CLIP and FID scores across range of

---

Alpha
0.6
172
80
T:
Space-Mol
16.6
Alpha
24
Soace-MoE
Timu-Mot
supervised
16.8
Time MoE Compurational Compleciy
RID-5k
16.4-
8
22
6.0
16.4
162
5.5
16.2
16
16.0
16.0
100
300
500
700
Tc
Score
Number Experts
(a) Impact of and T
(b) FID-CLIP score curves.
(c) Impact of experts.
Figure 6: Ablation Study. (a) examines the selection of a and T.. (b) presents the trade-off between FID and CLIP scores for the complete RAPHAEL model and its variants without space-MoF, time-MoF, and edge-supervised learning. (c) visualizes the correlation between FID-5k and runtime complexity (measured
in terms of the number of DDIM M steps for image per second) function of the number of experts employed. Notably, the computational complexity predominantly influenced by the number of spatial experts.
guidance weights 33 specifically 1.5, 3.0, 4.5, 6.0, 7.5, and 9.0. Fig 6p compares these curves for the complete RAPHAEL model and its variants without space-MoF, lige-supervised learning, and time-MoE, respectively. Our findings indicate that all modules contribute effectively. For example, space-MoB substantially enhances the CLIP score and the optimal guidance weight for the sampler shifts from 3.0 to 4.5. Moreover, at the same guidance weight, space-Moß considerably reduces the FID, resulting in significant improvement in image quality.
Choice of and T.. As depicted Fig we observe that 0.2 delivers the best performance. implying a balance between preserving adequate features and avoiding the use of the entire latent features. An appropriate threshold value for Te terminates edge-supervised learning when the diffusion timestep is large. Our experiments reveal that a suitable choice for T. is 500, ensuring the effective learning of texture information.
Performance and Runtime Analysis on Number of Experts. We offer an examination of the number of experts, ranging from 0 to 8, in Fig For each setting, employ 100 million training samples. Our results demonstrate that increasing the number of experts improves FID (lower values are preferable). However, adding spatial experts introduces additional computations, with the computational complexity bounded by the total number of experts. Once all available experts have been deployed, the computational complexity ceases grow. In the right-hand side of Fig/op. we provide runtime analysis for 40 input tokens, ensuring the utilization of all space experts. For instance, when the number of experts is 6, the inference speed decreases by 24% but yields superior fidelity. This remains faster than previous diffusion models such as Imagen [ and eDiff-1 [4]
Related Work
We review related works from two perspectives, mixtureeoffexxperts and text-to-image generation. More related works can be found in Appendix [7.4 Firstly, the Mixture-of-Experts (MoE) method g partitions model parameters into distinct subsets, each termed "expert". The MoE paradigm finds applicability beyond language processing tasks, extending to visual models (35] and Mixture- of Brlodaaity-Expperte within multi-modal transformers (36], Additionally, efforts are being made to accelerate the training or inference processes for MoE 37 38). Secondly, text-to-image generation synthesize images from natural language descriptions. Early approaches relied on generative to generate images. More recently, with the transfor- mative success of transformers generative tasks, models such DALL-E 43), Cogview [44), and Make-A A-Seene (29 have treated text-to-image generation as sequence- sequence-to-sequenes to-sequence problem, utilizing auto-regressive transformers as generators and employing text/image tokens as input/output sequences. Recently, another research direction has focused on diffusion models by integrating textual conditioning within denoising steps, like Stable Diffusion 2 DALL-E ERNIE-Vile ViLG 2.0 (5], and Imagen [1].
adversarial networks (GANs) 39 40
eDiff-I 4

---

Conclusion
This paper introduces RAPHAEL, a novel text-conditional image diffusion model capable of gen- erating highly-artistic images using a large-scale mixture of diffusion paths. We carefully design space-MoB and time-MoE within an adge-supervise learning framework, enabling RAPHAEL to accurately portray text prompts, enhance the alignment between textual concepts and image regions, and produce images with superior aesthetic appeal. Comprehensive experiments demonstrate that RAPHAEL surpasses previous approaches, such as Stable Diffusion, ERNIE-ViLA 2.0, DeepFloyd, and DALL-EE 2, both FID-30k and the human evaluation benchmark ViL.-300. Additionally, RAPHAEL can be extended using LoRA, ControlNet, and SR-GAN. We believe that RAPHAEL. has the potential to advance image generation research in both academia and industry.
Limitation and Potential Negative Societal Impact. We acknowledge some limitations in our paper that require attention. One limitation is the direct binarization of the attention map, which may result in the loss of some information. An adaptive module should be proposed to address this issue effectively. Additionally, the performance may be affected by failure cases of the edge detector, leading to potential degradation. We plan to explore solutions for these limitations in our future work. The potential negative social impact to use the RAPHAEL API to create images containing misleading or false information. This issue potentially presents in all powerful text-to- image generators. We will solve this issue (e.g. by prompt filtering) before releasing the API to the public.
Acknowledgments and Disclosure of Funding
This paper is partially supported by the National Key R&D Program of China No. N0.2002000000000 and the General Research Fund of Hong Kong No.
References
[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3666 36494, 2022.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint
2tt::2221..1222 2022.
[5] Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Emic--vlg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixuure-o--denoisng--eree... arXiv preprint arXiv:2210.152577 2022.
[6] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022.
[7] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-offexxprrs. layer. arXiv preprint arXiv: ar0iv: 70650063800888.. 2017.
[8] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-5270 2022.
10

---

[9] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini vrXiv:2/06.09685 2021.
[10] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprini arXiv:2302.05543; 2023.
[11] Xintao Wang, Liaggbnn Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1905-1914, 2021.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[13] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assise Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763 PMLR, 2021.
[15] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
[16] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
[17] Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. Xgboost: extreme gradient boosting. package version 0.4-2, 1(4): 1-), 2015.
[18] Saining Xie and Zhuowen Tu. Holistically-nestedd edge detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 1395-1400 2015.
[19] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference Computer Vision, pages 2980-2988, 2017.
[20] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint 22Xiv:2210.0 08402, 2022.
[21] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint or 3033:22303.06555, 2023.
[22] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
[23] Diederik Kingma, Max Welling, et al. An introduction to variational autoencoders. Founda- lions and Trends in Machine Learning, 12(4):307-392, 2019.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
11

---

[25] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[26] Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan:
A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515-16525, 2022.
[27] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-5810, 2019.
[28] Yufan Zhou, Ruiyi Zhang. Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxi- ang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-fice training for text-to-image generation. arXiv preprint arXiv:2
2021.
[29] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 89-106. Springer, 2022.
[30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 12. 10741, 2021.
[31] Minguk Kang, Jun-Yan Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[32] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint wxXi:22301.00704 2023.
[33] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12500 2022.
arXiv preprint
[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[35] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 01888888595 2021.
[36] Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.0720 07722,, 2023.
[37] Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang. Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv preprint arXiv:2/03.13262, 2021.
[38] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 100.:0000.6668,, 2020.
[39] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-65, 2018
[40] Ting- -hhu Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. fiih-resolutton image synthesis and semantic manipulation with conditional gans. In Proceed- ings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 1798-8807 2018.
[41] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/VV Conference Computer Vision and Pattern Recognition, pages 81 8110-8119, 2020.
12

---

[42] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):139-144, 2020.
[43] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shro text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[44] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:1 19822-19835,, 2021.
[45] Zeyue Xue, Jianming Liang. Guanglu Song, Zhuofan Zong, Liang Chen, Yu Liu, and Ping Luo. Large-batch optimization for dense visual predictions. arXiv preprint arXiv:2210. 1078, 2022.
[46] David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXiv:
2013.
[47] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu. Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with tixture-of-expertss In International Conference on Machine Learning. pages 5547-5569. PMLR, 2022.
[48] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettlemoyer. Branch-train-merge Embarrassingly parallel training of expert language models, arXiv preprint arXiv:2208 2rX1v:2008.03366, 2022.
[49] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning rext-to-image diffusion models for subject-driven generation. arXiv preprint a2Xi / 12242, 2022.
[50] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[51] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint urXiv:2210.0927 2022.
[52] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan- Yee K Wong. Vico: Detail-preserving visual condition for personalized text-to-image generation. arXiv preprint urXiv:2306.00971, 2023,
[53] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint urXiv:2303.05/22 2023.
[54] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multiple subjects. arXiv preprini arXiv:2305. 19327, 2023.
[55] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307 09481, 2023.
[56] Lingting Zhu, Zeyue Xue, Zhenchao Jin, Xian Liu, Jingzhen He, Ziwei Liu, and Lequan Yu. Make-a-volume Leveraging latent diffusion models for cross- mods-itdalyty 3d brain mri synthesis. International Conference on Medical Image Computing and Cospuler-Assisted Intervention, pages 592-601. Springer, 2023.
[57] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
13

---

[58] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Xrviv preprint arXiv:2209, 14988, 2022.
[59] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02015, 2023.
[60] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVT Conference on Computer Vision and Puttern Recognition, pages 22563-22574 2023.
[61] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (0544-10553, 2023.
[62] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023.
14


Reviews:
[{"review_id": "9WvvjrAXJs", "reviewer": "Reviewer_PszJ", "summary": "This paper proposes RAPHAEL, a new text-to-image generative model, based on the latent diffusion model framework. The main methodological contribution is the use of space-mixture-of-experts (space-MoE) layers. These are layers that focus on different concepts from the text prompt in different spatial areas of the synthesized image. Different space-MoEs are automatically chosen for the different text tokens, and they are assigned to the relevant regions in the image, which can be found through the cross-attention maps. A similar time-MoE is also incorporated, although that is less novel. The model is validated on standard benchmarks and achieves state-of-the-art performance. Ablations over all relevant new components and hyperparameters are performed. The paper shows various qualitative results and model samples, which are quite impressive. The authors also demonstrate that RAPHAEL can be easily combined with ControlNet, LoRA fine-tuning, and super resolution GANs to enhance the image resolution.", "questions": "I have only one minor question (just curiosity, not impacting the paper rating):\n\nFigure 4 shows the diffusion paths for different simple concepts. Do more related or similar concepts also have more similar diffusion paths? For instance, would \"strawberry\" and \"raspberry\" share much of their paths, while \"strawberry\" and \"car\" would not? ", "limitations": "Potential negative societal impacts have been briefly, but sufficiently addressed. Limitations have not been discussed. What are RAPHAEL's limitations? I would like to encourage the authors to add a critical discussion on this.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the paper are:\n- State-of-the-art text-to-image generation performance, including visually impressive results.\n- Extensive ablation studies on all relevant components.\n- The space-MoE idea is well motivated, novel, and boosts performance non-negligibly.\n- I like in particular the gating mechanisms in both the space- and the time-MoE, which, for instance in the time-MoE case, automatically learn to assign different timesteps to various time experts. Previous work manually assigned experts to different diffusion time intervals (for instance, eDiff-I).\n\n**Clarity:** The paper is well written and easy to read and follow. There are no major concerns regarding clarity. However, some details seem to be missing (see below).\n\n**Originality:** In general, the mixture-of-experts idea is not new and has existed in language models and was also used in the text-to-image literature, for instance in the related eDiff-I (only time experts). However, the space-MoE technique specifically in the context of text-to-image generation is new, to the best of my knowledge, and it is well-motivated and seems to meaningfully boost performance. Hence, while the paper's originality is not groundbreaking, the main technical contribution is well-motivated and novel.\n\n**Significance:** Text-to-image generation is a highly relevant and impactful topic, and RAPHAEL achieves state-of-the-art performance in this competitive area. Its mixture-of-experts approach is well-motivated and may find wider adoption. Apart from the quantitative evaluations, its visual results are stunning. Hence, I think the paper is impactful and significant.\n\n**Quality:** The overall quality of the paper is high. The paper is easy to read, appropriately discusses the related literature, provides a background section, runs extensive ablation studies for all new relevant parameters, and supports its claims by appropriate experiments. The qualitative and quantitative results are strong. There are only relatively minor concerns with respect to missing details (see below).", "weaknesses": "The paper does not have any major weaknesses. However, I have some minor concerns:\n1. Many details are missing:\n    - The paper only mentions on the side (Line 192) that it is using a latent diffusion model framework and does not operate in pixel space. \n       That is fine; however, this requires more details. For instance:\n         - Was the autoencoder regularized? The LDM paper uses either KL-based or VQ-based regularization.\n         - Was the autoencoder trained only with a reconstruction loss? Or also with a (patch-wise) discriminator?\n         - What was the downsampling ratio?\n   - The work should explain the multi-scale training in more detail. How exactly is the model trained at all these different resolutions and aspect ratios at once?\n   - The gating mechanisms incorporate an $argmax$ function. The $argmax$ is usually not differentiable. How did the authors deal with that, to enable regular backpropagation for training? I think it would be helpful to discuss this in a bit more detail.\n   - The paper says that RAPHAEL is trained on LAION \"and a few internal data\". What is this internal data? Even if this data is internal and not released, the authors should describe it and what value it brings on top of LAION. E.g. what is the size of that internal data? Does it have any special characteristics? Only high-quality, for instance? Etc.\n2. A discussion on limitations is missing. This would further strengthen the paper.\n\nIn conclusion, the paper's main weaknesses are all related to missing details and I believe these issues can be addressed easily. I do not see any other major concerns. Considering the paper's strengths discussed above, I am consequently suggesting acceptance of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PszJ,\n\nThank you for appreciating our approach. We will address your concerns below.\n\n**Q1: Details of VAE**\n\nYes, we follow the setup of Stable Diffusion and use KL-based VAE. When training the VAE, we add an extra discriminator following the pipeline of LDM. And the downsampling ratio is 8, z channel size is 12. We also change the ch (hyper-parameter of the LDM's VAE) from 128 to 256.\n\n**Q2: The work should explain the multi-scale training in more detail. How exactly is the model trained at all these different resolutions and aspect ratios at once?**\n\nThanks for your question. I will explain it in detail:\nAs outlined in our research paper, we employ a system comprising 9 buckets, each representing a distinct image scale. The initial step involves resizing an image to its nearest size within these predefined buckets. Subsequently, the allocation of GPU resources to each bucket is automated, based on the number of images they contain. This approach ensures efficient utilization of computational resources. Here is a step-by-step breakdown of the process:\n\n1. Aspect Ratio List ($R$): We maintain a list ($R$) containing aspect ratios for all the images in our dataset. The length of this list corresponds to the total number of images.\n2. Bucket List ($L$): We establish nine buckets ($L$) that encompass various image sizes, including  [448, 832], [512, 768], [512, 704], [640, 640], [576, 640], [640, 576], [704, 512], [768, 512], and [832, 448]. For a given image ratio in $R$, we identify the nearest bucket size in $L$ through a matching process. For example, images with an aspect ratio of 1.0 will be associated with the bucket of size [640, 640].\n3. Mapping Aspect Ratios to Buckets ($R1$): As a result of the previous step, we generate another list ($R1$) of the same size as $R$. Each element in $R1$ indicates the bucket to which the corresponding element from $R$ is assigned.\n4. GPU Allocation: We proceed to compute the GPU allocation based on the information provided by $R1$. Firstly, we calculate the total number of images each bucket contains. Using this information, we create another list ($L1$) of the same size as $L$, and L1.sum() denotes the total images we have. Then, we utilize a simple trick to distribute these buckets across different GPUs. The following code snippet demonstrates this process: We provide codes below:\n\n   bk_gpu_nums = np.clip((L1 / L1.sum() * world_size).astype(int), 0, world_size),\n\n   bk_gpu_nums[bk_gpu_nums.argmax()] = bk_gpu_nums[bk_gpu_nums.argmax()] - (bk_gpu_nums.sum() -world_size),\n\n   Here, \"world_size\" denotes the total number of GPUs available, and \"bk_gpu_nums\" signifies the number of GPUs required for each bucket. Then we assign different GPUs to different buckets.\n5. Image Selection: Each GPU within a specific bucket will select images from the dataset according to the mapping provided by $R1$, and this GPU will train the model with this bucket scale.\n\nWhile we acknowledge that the \"astype(int)\" operation may involve approximation, it becomes negligible when dealing with a large number of GPUs and datasets.\n\nFinally, it is essential to note that each GPU employs the same batch size during the training process.\n\n**Q3: Implementation of $argmax$ function**\n\nGiven a vector $v$, the first step is to compute the softmax value $y_{soft} = softmax(v)$.\n\nNext, the gating functions are determined using the $argmax$ operation, resulting in output $y_{hard} = argmax \\ y_{soft}$.\n\nTo ensure differentiability and enable backpropagation during training, we adopt a technique that bridges the gap between the discrete and differentiable representations. We introduced a soft version of the gating function, denoted as $y_{hard'}$, which can be expressed as follows: $y_{hard'} = (y_{hard} - y_{soft}).detach() + y_{soft}$.\n\nBy using this softened version of the gating function, we could successfully perform backpropagation during training. Although the gating decisions were made based on $y_{hard'}$, the actual backpropagation process was executed on the differentiable representation $y_{soft}$.\n\n**Q4: Internal data**\n\nWe have $\\approx$ 440M data filtered from LAION-5B, and $\\approx$ 290M internal datasets, which is less than Imagen's. Yes, it has special characteristics, such as high-quality, high-aesthetics.\n\nTo collect our internal datasets, we follow the methodology of DALL-E [1]. We curate a dataset on a scale similar to JFT-300M by sourcing images from the Internet. We remove instances with aspect ratios outside the range of [1/2,2], and we follow Stable Diffusion v1.4 to filter out images with low aesthetics scores. For captioning these images, we utilized BLIP-2. The main reason for constructing such an internal dataset with high aesthetics is to compensate for the poor quality of LAION.\n\nFurthermore, most papers on text-to-image generation use a series of internal datasets. For example, Imagen (NeurIPS'22 Outstanding Paper) uses 440M internal data and 400M public data; ERNIE-ViLG 2.0 (CVPR'23) uses LAION-5B and internal Chinese text-image pairs; MUSE (ICML'23) uses the same dataset with Imagen; GigaGAN (CVPR'23) also uses Adobe's internal data for its upsampler.\n\n**Q5: More properties of diffusion paths**\n\nYes, we observe that similar concepts have similar diffusion paths (each concept generates 100 paths with our template). For example, smooth/glossy, minimal/minimalist, dreamy/dreamlike, happy/joyful, sad/gloomy, brave/courageous, tired/exhausted, bright/luminous, honest/sincere, puzzled/confused, brilliant/shining, grateful/thankful, harsh/severe, enormous/huge, humble/modest, etc. We find the 200 diffusion paths generated by the similar concept pair always share at least 11 experts out of 16 blocks.\n\nBut for different concepts, such as strawberry/car, they are **different** things always share a small number of experts (less than 7).\n\n**Q6: Limitations**\n\nPlease refer to the Q9 of reviewer NLKN, and we have discussed limitations in detail."}, {"review_id": "n4obNRpY25", "reviewer": "Reviewer_8jJJ", "summary": "The paper proposed RAPHAEL, a text-to-image diffusion model. The model adopt MoE layers, including space-MoE and time-MoE layers.  In addition, edge-supervised learning is proposed to enhance performance. RAPHAEL establishes a new state-of-the-art with a zero-shot FID-30k score of 6.61 on the COCO dataset, and surpasses its counterparts in human evaluation on the ViLG-300 benchmark.", "questions": "1. Is the space-moe performed only on text-image cross attention? How about applying moe to self-attention within image?\n2. In abstract the authors mentioned that \"RAPHAEL exhibits superior performance in switching images across diverse styles\" (L13). Is it benefit from spatial-moe? if so, is there any pattern in moe paths regarding different styles?", "limitations": "see weakness and questions. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors promised to release a programming API for RAPHAEL to the public. \n2. Explore spatial- and time-moe, and perform some abaltion study. \n3. visualize spatial- and time-moe in the appendix. \n4. achieved sota zero-shot FID score on coco.  ", "weaknesses": "As a paper focusing on pretraining, more clarification of experiment details is needed. Including:\n\na. data. The paper mentioned that \"The training dataset consists of LAION-5B and a few internal data.\". How many internal data is used? Its category distribution and collecting sources?\n\nb. model structure and hyperparameters. including VAE, and each stage of diffusion model. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8jJJ,\n\nThanks for appreciating our work and your advice. We will address your concerns below.\n\n**Q1: The paper mentioned that \"The training dataset consists of LAION-5B and a few internal data.\". How many internal data is used? Its category distribution and collecting sources?**\n\nThe data consists of approximately 440 million entries filtered from LAION-5B and approximately 290 million entries from internal datasets, which is less than what Imagen's dataset contains. These datasets possess special characteristics, notably high-quality and aesthetics.\n\nTo collect our internal datasets, we follow the methodology of DALL-E [1]. We curate a dataset on a scale similar to JFT-300M by sourcing images from the Internet. We remove instances with aspect ratios outside the range of [1/2,2], and we follow Stable Diffusion v1.4 to filter out images with low aesthetics scores. For captioning these images, we utilized BLIP-2. The main reason for constructing such an internal dataset with **high aesthetics** is to compensate for the poor quality of LAION. We don't add additional limits on categories but want to collect datasets with high aesthetics.\n\nRegarding the resolution distribution, adopting the buckets mentioned in the global response Q2, it is as follows: [52,24,161,470,52,81,56,70,34]\n\nFurthermore, in the realm of text-to-image generation, most papers use a combination of internal datasets. For example, Imagen (NeurIPS'22 Outstanding Paper) employs 440M internal data and 400M public data; ERNIE-ViLG (CVPR'23) utilizes LAION-5B and internal Chinese text-image pairs; MUSE (ICML'23) uses the same dataset as Imagen; and GigaGAN (CVPR'23) also leverages Adobe's internal data for its upsampler.\n\n**Q2: Model structure and hyperparameters. including VAE, and each stage of diffusion model.**\n\nThe VAE model structure is based on the setup of Stable Diffusion, utilizing a KL-based VAE. Following the pipeline of LDM, an additional discriminator is introduced to train the VAE. And the downsampling ratio is 8, and z channel size is 12. We also change the ch (hyper-parameter of the LDM's VAE) from 128 to 256.\n\nRegarding the hyperparameters for the diffusion models, our approach consists of a single stage. We adhere to the UNet configurations of stable diffusion v2.1, with the exception of disabling the self-attention module in the largest resolution due to its computational complexity.\n\nWe plan to include these details in our paper, and as soon as we gain access to edit it in openreview, we will upload the information accordingly.\n\n**Q3: Is the space-moe performed only on text-image cross attention? How about applying moe to self-attention within image?**\n\nThe space-moe operation is specifically designed for text-to-image generation. Its purpose is to depict different *text* concepts within specific image regions. Thus, a *text* token is required to perform the cross-attention operation effectively.\n\nHowever, when it comes to the self-attention module, the self-attention map always resembles the \"contour\" of the image and does not have a direct mapping relationship to a particular token. As a result, the space-moe operation cannot be applied to the self-attention module. It can be applied to the cross-attention mechanism because it excels in capturing the correlations between textual descriptions and corresponding image regions.\n\n**Q4: In abstract the authors mentioned that \"RAPHAEL exhibits superior performance in switching images across diverse styles\" (L13). Is it benefit from spatial-moe? if so, is there any pattern in moe paths regarding different styles?**\n\nYes, it benefits from space-moe.\n\nFirstly, as shown in Fig. 6b, space-moe significantly increases the CLIP score, which measures the alignment between images and text descriptions.\n\nSecondly, we observe some patterns (each concept generates 100 paths with our template). For style concepts such as anime/digital/realistic/cyberpunk/artistic/colorful/minimalist/bright, etc, we find that each pair in these styles (such as anime/digital, digital/realistic, etc) always share a relatively small number of experts (less than 7). Moreover, adjectives provided in our appendix also contain many style concepts, and they can be easily classified by XGBoost algorithm.\nSo we think different style concepts have different diffusion paths.\n\nThirdly, we conducted a human evaluation using the model with and without space-moe based on prompts containing style information from the ViLG-300 dataset. The results indicate that most people (76.15%) tend to prefer the model with space-moe, as they believe it better matches the images with the specified style.\n\nFinally, we also observe that diffusion paths reflect semantics, similar concepts/styles have similar diffusion paths. For example, smooth/glossy, minimal/minimalist, dreamy/dreamlike, happy/joyful, sad/gloomy, brave/courageous, tired/exhausted, bright/luminous, honest/sincere, puzzled/confused, brilliant/shining, grateful/thankful, harsh/severe, enormous/huge, humble/modest. We find the 200 diffusion paths generated by each pair share at least 11 experts out of 16 blocks. So we believe space-moe also helps the understanding of semantics."}, {"review_id": "CQSx3BnH8M", "reviewer": "Reviewer_NLKN", "summary": "The paper trains a large-scale latent diffusion model for image synthesis. It trains on a mix of LAION-5B (post-processed + filtered by aesthetic score) and in-house data. It proposes two novel technical contributions: 1) using a \"spatial\" mixture-of-experts (MoE) where an expert is predicted from each text token and processes the attended features; and 2) supervising attention weights via edge maps. Interestingly (and differently from most of the prior works), it also uses multi-scale training. Qualitatively, the generated images look substantially better than the ones from the baselines. It achieves SOTA FID on COCO, which is the main metric/benchmark for text-to-image generators.", "questions": "I raised several concerns in the \"Weaknesses\" section and would be grateful to hearing the author's opinion on them. My main concerns are the lack of details about the method and experiments.", "limitations": "There is a brief limitations discussion on potential negative societal impact. In this regard, I wouldn't demand more discussion from the authors since a potential misuse of powerful image generators is a well-known issue to the community and should be discussed at a \"higher\" level, rather than in this particular work.\n\nHowever, it would be good to see the discussion of other potential limitations of the work, for example, what are the disadvantages of binarizing the attention maps? whether it is possible to ablate the model properly — at least via convergence plots for partial runs (I understand that training a full model for each ablation is infeasible)? what could be a problem with edge maps supervision of the attention maps (i.e., I guess there should be failure cases in edge map detection)? And so on.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method achieves the very best known results for large-scale text-to-image synthesis (among those models which could be rigorously benchmarked against).\n- There are two novel ideas explored in the work: using spatial MoEs (for attended features) and using edge supervision for attention weights. Both ideas make sense and should be easily extendable to other setups. For spatial MoEs, there are also test-time visualizations provided which helps in understanding their influence.\n- The comparison to other methods is careful and thorough: a lot of non-cherry-picked qualitative results provided; human studies are performed, quantitative metrics are reported.\n\n", "weaknesses": "I have two big concerns: 1) spatial MoE and edges supervision are not properly ablated; and 2) the paper does not contain enough technical details to be reproduced. I will elaborate on them below:\n\n1. Improper ablations. After spending ~5 hours on reading the paper, it's still not clear to me where exactly the SotA FID score on COCO is coming from — architecture, data, or optimization, since they are all intermixed in the final model. Fig 6 denotes several ablation experiments, but it's not specified anywhere how were they trained. Does each run in Figure 6 was trained for 2 months on 1,000 GPUs as well? On the same dataset? Such lack of details about ablations makes them impossible to understand and analyze. For Figure 6c, what the FID scores for the line \"Computational complexity\" denote? This is confusing.\n\n2. Lack of details. The current manuscript is something between an academic paper and a technical report. Here are some (of many) missing details:\n  - How exactly does your U-net and VAE look like? Are they equivalent to LDM ones, but with larger channel sizes? Or there are other modifications (apart from spatial/time MoEs)?\n  - How many images are in your in-house dataset, what are their resolutions and how it was collected?\n  - How exactly was multi-scale training implemented? What are the resolution distributions in your final dataset? Do I get it right that different batches on different GPUs have different amount of images in them (since the resolutions are different)? Do you allocate the same amount of GPUs per bucket? Do I get it right that your VAE is multi-scale, while the diffusion model is not? Or vice-versa?\n  - How random noise \\epsilon is sampled for expert routing (L122)?\n  - What is the motivation of using focal loss for edge prediction instead of other loss types? Using focal loss here is quite non-intuitive to me. Did you try ablating it?\n\nThere are also several smaller (but still reasonable) concerns:\n- It's not clear whether spatial experts reflect any text semantics. Judging by Figure 4, the expert assignments are completely random. I would expect to see that visually similar concepts (\"tiger\"/\"cat\"/\"leopard\"; \"dog\"/\"wolf\"/\"fox\"; \"tv\"/\"monitor\") would get assigned to the same expert. Could you please provide any support or refutation to such intuition (e.g., buy checking the clusters)?\n- The writing quality could be improved. There are many variables introduced, and it would ease reading if they would be described in text, i.e., instead of writing \"we set \\lambda to X\", one should write \"we set learning rate \\lambda to X\". Otherwise, a reader needs to jump back and forth trying to recall the variable meaning.\n- There is a quite confusing notation clash:\n  - \\epsilon denotes random noise in diffusion and random noise (L74) in experts routing (L122)\n  - \\alpha denotes variance schedule (L74), focal loss hyperparameter (Figure 6a), routing multiplier (L129)\n- Limitations are not properly discussed (see the \"Limitations\" form below)\n- GigaGAN is a missed reference since they also use MoEs, routed by text tokens.\n- It's not clear why SR-GAN is included into the exposition since there is nothing special about it, and SR-GAN can be combined with any other image generator. Does it work better for RAPHAEL than for other image generators? If so, then it's interesting. But if not — then it's not clear why claiming that RAPHAEL can generate 4096 x 6144 images when combined with it. Following such arguments, what prevents one saying that StyleGAN can generate 100,000 x 100,000 images when combined with tailor-mode bilinear interpolation?\n\nTypos and minor comments:\n- L114: \"mean of all experts\" => \"weighted average of all experts\"?\n\nI look forward to discussing my concerns with the authors and fellow reviewers and improving my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NLKN,\n\nThanks for giving so many constructive suggestions for our paper, I will clarify the settings.\n\n**Q1: The setting of ablation study**\n\nWe conducted an ablation study in the following manner: \n\nFor Fig.6a, we resume the final model trained for two months, and train each point in Fig.6a for another 100M samples to ensure convergence, using the same dataset (LAION and internal datasets) and seed. \n\nFor Fig.6b, we resume the final model trained for two months and then individually deleted the space-moe, time-moe, and edge-supervised learning modules, resulting in three different models. Next, we continued to train each of these models without the respective module with 100M training samples **to ensure their convergence**, using the same dataset (LAION and internal datasets) and seed. **The implementations of space-moe and time-moe are both with residual connection, so the deletion operation is reasonable.** We measure the FID curves for these three models, and the results are presented in Fig. 6b.\n\nFor Fig. 6c, the red and blue lines represent the FID-expert curves (left axis) for space-moe and time-moe, respectively, showing that FID decreases with an increased number of space experts and time experts. We conduct this ablation study following the pipeline of Fig.6b.  We resume the final model trained for two months and then individually deleted the space-moe or time-moe. We add new space-moe or time-moe modules according to the number of experts needed and train each setting for 100M training samples.\n\nThe green line in Fig. 6c represents the inference speed (DDIM steps/s, right axis) with an increased number of space experts. We provide an alternative way to speed up space-moe in our cluster. The definition of space-moe is as follows: $\\frac{1}{n_y}\\sum_{i=1}^{n_y} e_{\\operatorname{route}(y_i)}(h'(x_t) \\circ \\widehat{\\mathbf{M}}_i )$. Each token's corresponding feature will be routed to different experts naively, which is achieved through a \"for\" loop and cannot be optimized in our hardware setup. We propose an implementation approach as follows to address this:\n\n1.We obtain a list $[a_1, a_2,..., a_k]$ = $[e_1(h'(x_t)), e_2(h'(x_t)),..., e_k(h'(x_t))]$.\n2. The output of space-moe can be calculated as: $\\frac{1}{n_y}\\sum_{i=1}^{n_y} a_{\\operatorname{route}(y_i)}\\circ \\widehat{\\mathbf{M}}_i$.\n\nThe above implementation is always faster. We also ablated these two implementations, each trained with ~120M samples, and find that they exhibit similar performances in human evaluation and similar diffusion paths. The inference cost compared with other popular text-to-image diffusion models can be found in the pdf file of the general response.  \n\n**Q2: * How exactly does the U-net and VAE look like**\n\nPlease refer to the Q3 of our global response.\n\n**Q3: In-house dataset**\n\nPlease refer to the Q1 of our global response.\n\n**Q4: Implementation of multi-scale training**\n\nPlease refer to the Q2 of our global response for the detailed implementation.\n\n**Q5: Choice of \\epsilon.**\n\nA small number 1e-6.\n\n**Q6: Choice of Focal Loss**\n\nYes, we have explored other loss types and conducted ablation studies to address this. The most intuitive loss is cross-entropy loss. However, we encounter imbalanced edge-maps and background, where the prediction module tends to classify all pixels as background, negatively impacting the cross-attention maps. To overcome this, we decide to adopt Focal Loss, which effectively handles the imbalance issue.\n\n**Q7: Semantics of space experts.**\n\nWe observe some patterns, similar concepts have similar diffusion paths (each concept generates 100 paths with our template). For example, smooth/glossy, minimal/minimalist, dreamy/dreamlike, happy/joyful, sad/gloomy, brave/courageous, tired/exhausted, bright/luminous, honest/sincere, puzzled/confused, brilliant/shining, grateful/thankful, harsh/severe, enormous/huge, humble/modest. We find the diffusion paths generated by each pair always share at least 11 experts out of 16 blocks.\n\nFor the diffusion paths given by visually similar but **different** pairs, such as tiger/cat/leopard (300 paths); dog/wolf/fox (300 paths); tv/monitor (200 paths), they also always share a relatively small number of experts (less than 7). We guess it is because they don't have similar semantics.\n\n**Q8: Writing quality, notations, references and SR-GAN**\n\nThanks for pointing out these issues. We will polish it in the future since we can't edit the paper in the openreview now.\nWe will also modify the claim about SR-GAN.\nAnd sorry for missing this excellent paper GigaGAN. We will add it once we can edit the paper.\n\n**Q9: Limitations**\n\nIndeed, we acknowledge several limitations in our paper that require attention. One limitation is the direct binarization of the attention map, which may result in the loss of some information. An adaptive module should be proposed to address this issue effectively. Additionally, the performance may be affected by failure cases of the edge detector, leading to potential degradation. We plan to explore solutions for these limitations in our future work.\n\nAnother limitation pertains to the design of our ablation study. Conducting each full setting in the ablation study would be prohibitively expensive and time-consuming. As a result, we opt to run partial settings for the ablation study, which still takes 1.5 months to complete. We recognize the need for a better and more efficient approach to conducting ablation studies of foundation models in the research community.\n\nDuring the 7-day rebuttal period, it is almost impossible to run so many experiments for re-plotting the convergence curves for Fig. 6. However, based on our experience, we observe that edge-supervised learning can converge in less than three days. On the other hand, space-moe and time-moe converge much slower, requiring at least two weeks, given the setting of 6 space experts and 4 time experts.\n"}, {"review_id": "XA04VVH4Tg", "reviewer": "Reviewer_fnVX", "summary": "This paper presents a new large model RAPHEL (short for distinct image regions align with different text phases in attention learning ) for text-to-image generation. Technically, RAPHEL builds upon the LDM pipeline, with VAEs as image encoder-decoder, and then incorporates the MOE layers for spatial and temporal (in terms of diffusion steps) refinement in the diffusion generation process to improve the text and image fidelity. For the experiments and evaluations, the model is trained with LAION-5B and some confidential internal data, comparisons w/ other large models show better performance using the zero-shot FID on the COCO.", "questions": "I have several questions regarding several technical aspects listed below:\n\n- I am still confused on the working mechanism of time-MoE after reading the paper and appendix. For the time-MoE, it is a Time Gate Network at each diffusion step, located between the cross-attention and space MoE. The output of time-MoE is fed into the space-MoE, then what does this info depict for space-MoE at different diffusion steps? \n\n- How does this work in inference, if the info from time-MoE does convey critical information in terms of steps, then how can the info be used in inference especially with the skipping sampling steps?\n\n- What is the inference time cost using the proposed RAPHEL compared to other popular models?\n\n- How does the internal data impact the final performance? Does the performance change evidently w/o the inaccessible internal dataset?\n", "limitations": "The paper discusses the limitations and potential negative impact on the risk of generating images with misleading and false information.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written with a clear structure and easy to follow.\n\n- The model achieves sota FID and qualitative results compared to other strong and powerful similar scale models like Stable Diffusion, DeepFloyd and DALLE2.\n\n- The high-level idea to incorporate MoEs to refine the spatial and temporal details in DPMs for text2image synthesis is intuitive and reasonable, with effectiveness proved in ablation studies.\n", "weaknesses": "- This is another large-scale model work that requires 1000 A100 GPUs with 2-month training on LAION-5B plus internal dataset, while the reviewer acknowledges the popularity of the topic and its superiority in performance, it is another work that can be hardly reproduced by most researchers in the field, especially w/ internal training data inaccessible for the community.\n\n- In terms of the methodology design, while the idea to refine the generation process w/ MoEs is intuitive, the technical novelties are rather limited, and are largely limited to this specific text2img task, as there are several existing works w/ similar ideas [4,5].\n\n- Some technical details remain rather coarse and unclear. See details in my questions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer fnVX,\n\nThanks for your comments. We will address your concerns below.\n\n**Q1: RAPHAEL uses internal datasets and many computing resources**\n\nWe argue that this is not a weakness for rating specific to the text-to-image diffusion model community. Numerous academic papers accepted by top conferences/journels have internal datasets. We provide a table on text-to-image diffusion models here:\n\n| Model          | Venue                        | Internal data | GPUs/TPUs |\n|----------------|------------------------------|---------------|-----------|\n| Ours           | N/A                          | Yes           | 1k  A100s      |\n| Imagen         | NeurIPS'22 Outstanding Paper | Yes           | 512    TPUs   |\n| ERNIE-ViLG 2.0 | CVPR'23                      | Yes           | 320  A100s     |\n\n**Notably, Imagen even received the outstanding paper award at NeurIPS last year.** We also understand ERNIE-ViLG 2.0 uses fewer A100s than us, but they continue to update the model from last year's September and we use their latest API to make comparisons. So it's difficult to compare the GPU hours.\n\nMoreover, we intend to address this issue by releasing an API to make the model more accessible to the public. We firmly believe that RAPHAEL will contribute significantly to the advancement of text-to-image generation in the research community.\n\nOther than the diffusion models mentioned above, **other text-to-image models also use internal datasets, such as MUSE (ICML'23), GigaGAN (CVPR'23), Parti (TMLR'22)**.\n\n**Q2:The novelty is limited.**\n\nAs highlighted by reviewers PszJ, **\"The space-MoE technique, specifically in the context of text-to-image generation, appears to be new and well-motivated, leading to a meaningful performance boost.\" and \"The main technical contribution is well-motivated and novel.\"** Reviewer NLKN also claims \"There are **two novel ideas** explored in the work: using spatial MoEs (for attended features) and using edge supervision for attention weights. Both ideas make sense and should be easily extendable to other setups. \" Both reviewers acknowledge that RAPHAEL presents novelty.\n\nThe space-moe and edge-supervised learning in RAPHAEL are our original contributions, which conduct region-level refinement during the denoising process and significantly improve image quality. We have also implemented a gating function to enhance the performance of time-moe, as noted by reviewer PszJ: \"I particularly appreciate the gating mechanisms, where, in the case of time-MoE, they automatically assign different timesteps to various time experts. Previous work manually assigned experts to different diffusion time intervals (e.g., eDiff-I).\" Our experts are assembled on-the-fly during inference, which is more flexible and parameter-efficient.\n\n**Q3: Working mechanism of time-MoE**\n\nThe space-moe and time-moe are disjoint. We experimented with different placements of space-moe and time-moe and obtained similar performances.\n\nThe feature processed by time-moe is passed on to space-moe. This feature serves as a better representation of latent features than before and is utilized in space-moe. The gating function takes text tokens as input to determine which experts should process the features, as output by the output of time-moe. Notably, the space-moe does not have any temporal dimension; it leverages the improved feature representation from time-moe.\n\n**Q4: Time-moe in inference**\n\nDuring the training process, the gating function is trained to automatically select the appropriate time expert. Once convergence is achieved, for example, at time step 1, expert 1 assists the UNet in fitting the score function. At time step 500, expert 2 is assigned to fit the score function. Therefore, during the sampling process, each time step is associated with a specific time expert to fit the score function. Although skipping of some time steps may occur during sampling when using DDIM or DPM solvers, the score functions for the selected time steps are accurately fitted with the assistance of these experts.\n\n**Q5: Inference cost**\n\nWe provide an analysis in Section 4.2, which shows that the inclusion of space-moe results in an additional 24% overhead. This is faster than models with cascaded designs, such as Imagen and DeepFloyd. Furthermore, time-moe and edge-supervised learning do not introduce any extra inference cost.\n\nWe can also compare RAPHAEL with other popular models. All the models provided here are highly optimized, so we will choose our optimized version to compare (a bit faster than the results in Fig 6c because of the update of our infrastructure). We also use this same environment to conduct a fair comparison between RAPHAEL and Stable Diffusion XL. **Please refer to the pdf file in the general response for the results.** So we think the inference cost is not the bottleneck for RAPHAEL.\n\n**Q6: The impact of internal datasets**\n\nWe incorporated internal datasets to improve the aesthetics of the generated images, a practice common among prestigious text-to-image models, including Imagen, Stable Diffusion XL, DeepFloyd, MUSE, Parti, ERNIE-ViLG 2.0, DALL-E 2, eDiff-I, etc.\n\nThe internal data primarily impacts the aesthetics of the generated images. Due to the limited time of the rebuttal period, we resume the checkpoint of RAPHAEL trained for two months, and continue to train it with LAION-5B for ~7 days instead of training from scratch. It's also reasonable because of the catastrophic forgetting properties of deep neural networks. Based on this model, we conduct a human evaluation using ViLG-300. The results indicate that most people (72.6%) prefer the model trained on internal datasets in terms of image quality. This preference is attributed to the poor image quality of LAION. However, we don't observe significant difference in image-text alignment.\n\nAdditionally, we measure the FID based on this model, resulting in 6.79. So the internal data doesn't have much influence on the coco-30k FID."}]

Author rebuttal:
General response:
We express our sincere appreciation to all the reviewers for their valuable time and efforts in reviewing our paper. We are delighted to learn that the reviewers have generally recognized and appreciated the contributions made in our work, which include:

1. State-of-the-art performance (PszJ, 8jJJ, NLKN, and fnVX).
2. Introducing two novel ideas, namely, space-moe (PszJ, NLKN) and edge-supervised learning (NLKN).
3. Extensive ablation study (PszJ, 8jJJ, fnVX) and visualization results (8jJJ, NLKN).
4. Thorough comparison to other methods (NLKN).

We extend our gratitude to all the reviewers for their insightful and constructive suggestions. The main concerns raised by the reviewers revolve around our experimental settings. To address these concerns, we provide the following summaries and discuss them in rebuttal:
1. Internal datasets (PszJ, 8jJJ, NLKN, fnVX).
2. Multi-scale training implementation (PszJ, NLKN).
3. VAE and U-Net configurations (PszJ, 8jJJ, NLKN).
4. Ablation study settings (NLKN).
5. Choice of Focal Loss (NLKN).
6. Additional properties of diffusion paths. (PszJ, 8jJJ, NLKN)
7. Inference cost. (fnVX)

We are committed to incorporating these improvements and addressing all the raised concerns in our revised paper once we can edit it in openreview.

We will also make a global response for the Q1,Q2,Q3 above:

**Q1: Information about internal dataset**

The data consists of approximately 440 million entries filtered from LAION-5B and approximately 290 million entries from internal datasets, which is less than what Imagen's dataset contains. These datasets possess special characteristics, notably high-quality and aesthetics.

To collect our internal datasets, we follow the methodology of DALL-E. We curate a dataset on a scale similar to JFT-300M by sourcing images from the Internet. We remove instances with aspect ratios outside the range of [1/2,2], and we follow Stable Diffusion v1.4 to filter out images with low aesthetics scores. For captioning these images, we utilized BLIP-2. The main reason for constructing such an internal dataset with **high aesthetics** is to compensate for the poor quality of LAION. We don't add additional limits on categories but want to collect datasets with high aesthetics.

The resolution distribution is [52,24,161,470,52,81,56,70,34] if we adopt the buckets in Q2 and 1k GPUs. It indicates, for example, there are 52 GPUs for [448,832] and each GPU shares a similar number of images.

**Q2: Implementation of multi-scale training**

As outlined in our research paper, we employ a system comprising 9 buckets, each representing a distinct image scale. The initial step involves resizing an image to its nearest size within these predefined buckets. Subsequently, the allocation of GPU resources to each bucket is automated, based on the number of images they contain. Here is a step-by-step breakdown of the process:

1. Aspect Ratio List ($R$): We maintain a list ($R$) containing aspect ratios for all the images in our dataset. The length of this list corresponds to the total number of images.
2. Bucket List ($L$): We establish nine buckets ($L$) that encompass various image sizes, including **[448, 832], [512, 768], [512, 704], [640, 640], [576, 640], [640, 576], [704, 512], [768, 512], and [832, 448]**. For a given image ratio in $R$, we identify the nearest bucket size in $L$ through a matching process. For example, images with an aspect ratio of 1.0 will be associated with the bucket of size [640, 640].
3. Mapping Aspect Ratios to Buckets ($R1$): As a result of the previous step, we generate another list ($R1$) of the same size as $R$. Each element in $R1$ indicates the bucket to which the corresponding element from $R$ is assigned.
4. GPU Allocation: We proceed to compute the GPU allocation based on the information provided by $R1$. Firstly, we calculate the total number of images each bucket contains. Using this information, we create another list ($L1$) of the same size as $L$, and L1.sum() denotes the total images we have. Then, we utilize a simple trick to distribute these buckets across different GPUs. The following code snippet demonstrates this process: We provide codes below:

   bk_gpu_nums = np.clip((L1 / L1.sum() * world_size).astype(int), 0, world_size),

   bk_gpu_nums[bk_gpu_nums.argmax()] = bk_gpu_nums[bk_gpu_nums.argmax()] - (bk_gpu_nums.sum() -world_size),

   Here, "world_size" denotes the total number of GPUs available, and "bk_gpu_nums" signifies the number of GPUs required for each bucket. Then we assign different GPUs to different buckets.
5. Image Selection: Each GPU within a specific bucket will select images from the dataset according to the mapping provided by $R1$, and this GPU will train the model with this bucket scale.

While we acknowledge that the "astype(int)" operation may involve approximation, it becomes negligible when dealing with a large number of GPUs and datasets.

Finally, it is essential to note that each GPU employs the same batch size during the training process. This comprehensive approach enables effective multi-scale training, resulting in enhanced performance and robustness of the model.

Given 1000 GPUs and the predefined buckets, the final resolution distribution (including LAION and internal datasets) of these buckets is [28,67,116,419,43,52,76,112,87], which means there are 28 GPUs in bucket [448, 832].

Hence, the different bucket has different images and different number of GPUs. Each GPU will sample from its belonging bucket. The VAE and diffusion models both need multi-scale training.

**Q3: Configs of VAE and U-Net**

For U-Net, the difference between ours and Stable Diffusion v2.1 is we disable the self-attention module for the largest resolution due to its computation complexity. For VAE, the downsampling ratio is 8, z channel size is 12. We also change the ch (hyper-parameter of the LDM's VAE) from 128 to 256. 

Meta-review: This paper received all acceptance scores, with two acceptances, one weakly acceptance, and one borderline acceptance. The paper introduces RAPHAEL, a new text-to-image generative model, based on the latent diffusion model framework, and it is validated on standard benchmarks and achieves state-of-the-art performance. Moreover, RAPHAEL is a technically solid work with its valuable contributions to the community. As a result, AC has decided to accept this paper.
Decision: Accept (poster)