Ensembling Diffusion Models via Adaptive Feature Aggregation
Authors: Cong Wang
Kuan Tian
Yonghang Guan
Fei Shen
Zhiwei Jiang
Qing Gu
Jun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Image Generation
Diffusion Models
Model Ensembling
Primary area: generative models
TL;DR: We propose Adaptive Feature Aggregation (AFA) to ensemble multiple diffusion models dynamically based on different states.

Abstract: The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source community. These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multiple high-quality models to produce stronger generation ability is valuable, but has not been extensively studied. Existing methods primarily adopt parameter merging strategies to produce a new static model. However, they overlook the fact that the divergent denoising capabilities of the models may dynamically change across different states, such as when experiencing different prompts, initial noises, denoising steps, and spatial locations. In this paper, we propose a novel ensembling method, Adaptive Feature Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple diffusion models, while suppressing their disadvantages. Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into a unified one. The core idea lies in dynamically producing an individual attention map for each model's features by comprehensively considering various states. It is worth noting that only SABW is trainable with about 50 million parameters, while other models are frozen. Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed method.

Published as a conference paper at ICLR 2025
ENSEMBLING DIFFUSION MODELS VIA ADAPTIVE FEATURE AGGREGATION
Cong Wang
Kuan Tian
Yonghang Guan?
Fei Shen
Zhiwei Jiang"
Qing Gu
Jun Zhang?:
State Key Laboratory for Novel Software Technology, Nanjing University
2 Tencent AIPD
owe seeemmir. i1, .nju. edu. en, (kuantian, yohnguan, ffeishen//teenent.coo [jzw, eug)enju. jug)enju.edu.cm en, junejzhangeternent.com
ABSTRACT
The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source commu- nity. These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multiple high-quality models to pro- duce stronger generation ability is valuable, but has been extensively studied. Existing methods primarily adopt parameter merging strategies to produce a new static model. However, they overlook the fact that the divergent denoising ca- pabilities of the models may dynamically change across different states, such when experiencing different prompts, initial noises, denoising steps, and spatial locations. In this paper, we propose novel ensembling method, Adaptive Fea- ture Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple design
diffusion models, while suppressing their disadvantages. Specifically, lightweight Spatial-Aware Block-Wits Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into unified one. The core idea lies in dynamically producing an individual at- tention map for each model's features by comprehensively considering various states. It is worth noting that only SABW is trainable with about 50 million pa- rameters, while other models are frozen. Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed method."
INTRODUCTION
Diffusion models (Sohl- -Sohl-Dickstein et al., 2015; Ho et al., 2020) have progressively become the main- stream models for text-guided image generation (Nichol al., 2021: Ramesh al., 2022: Saharia et al., 2022; Rombach et al., 2022; Balaji et al., 2022; Xue et al., 2023; Feng et al., 2023), which treats generation as an iterative denoising task. Recently, the open-source stable diffusion (SD) (Rombach et al., 2022) model has prompted the development and release of numerous powerful diffusion models within the open-source community (c.g., CivitAI-). These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multi- ple high-quality models to dig out better generations is an important research direction, which has not been extensively studied.
Existing methods leverage multiple diffusion models through the weighted merging of model param- eters, which can be called the static method. The weights usually manually set (c.g., Weighted-
"Work done during internship Tencent AIPD.
'Euual contribution.
1 Cooresponding authors.
I 'The code is available at http://hhho.ccmmmeeeeeeeee//aaaa
hhtpps: h/tpv//viiiiiiicoooooo

---

Published as a conference paper at ICLR 2025
fying the golden coins.
a day taking seffie.
Model
Model C
Proportion of twin Monte- Monte-Carlo Carlo sampled noïses
1000
1000
Denoising Step
Denoising Step
Examples illustrate the dynamical change of the denoising capabilities across various states. We conduct experiments on different prompts with various initial noises. We then plot the proportion of wins (i.e., the model with the least error between the predicted noise and the initial noise), for each model in a certain spatial region.
Figure 1:
Merging and MBW*) or automatically searched through enumeration (e.g., autoMBW*). In con- trast, model ensembling, which can be called the dynamic method, often uses dynamic strategies to fuse multiple models at the feature level. Unlike ensembling for classification models (Freund & Schapire, 1995) that usually work after the output, ensembling for diffusion models typically needs to work for each block. However, the denoising capabilities blocks but also different spatial locations. As illustrated in Figure iments on different prompts with various initial noises. Then, we plot the proportion of wins (i.e., the model with the least error between the predicted noise and the initial noise), for each model in certain spatial region, to distinguish the denoising capabilities of the models at different states. In other words, different prompts, initial noises, denoising steps, and spatial locations can all have significant impact on the denoising capabilities of diffusion models. This implies that an adaptive method is needed to ensure that each diffusion model dominates the generation its strongest states.
models vary not only at different
conduct denoising exper-
In this paper, we propose novel Adaptive Feature Aggregation (AFA) method, which dynamically adjusts the contributions of multiple models at the feature level by taking into account various states, such as prompts, initial noises, denoising steps, and spatial locations. Specifically, we design lightweight Spatial- Block-Wiss (SABW) feature aggregator that adaptively aggregates the block-level intermediate features from multiple U-Net denoisers into unified one. The core idea of adaptive aggregation lies in dynamically producing an individual attention map for each model's features by comprehensively taking into account the various states. A noteworthy aspect of AFA is that only SABW is trainable with about 50 million parameters, while other models are frozen.
Our main contributions summarized as follows:
We propose an ensembling- based AFA method dynamically adjust the contributions of multiple models at the feature level.
We design the SABW feature aggregator that can produce attention maps according to various states adaptively aggregate the block-level intermediate features from multiple U-Net denoisers.
We conduct both quantitative and qualitative experiments, demonstrating that our AFA outper- forms the base models and the baseline methods in both superior quality and context alignment.
wttp./ccoo/ooooeo///seeeeee--eee ook-we
-ttereee-ggooetiiiiieee------- ghted-gui
Phttps://g thub. http:///w///////eeieieoooe wwt.e//wwweii/wwweeeeeeeeeeeeeee
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Text- Image Synthesis. Early works for text-guided image synthesis leverage Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) conditioned on text (Tao et al., 2022; Xu et al., 2018; Zhang et al., 2021a; Zhu al., 2019). Based on the success of Transformers (Vaswani et al. 2017). many subsequent works reframe text-guided image synthesis as a peruuence-to-sqqeeee task (Zhang al., 2021b; Ramesh et al., 2021; Ding et al., 2021; Gafni et al., 2022; Yu et al., 2022). Recently, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) gradually become mainstream, which treats image generation as an iterative denoising task. By injecting text as a condition into the denoising process, many diffusion- based models achieve significant success in text-guided image symthesis (Nichol al., 2021; Ramesh al., 2022; Saharia al., 2022; Rombach et al., 2022; Balaji et al., 2022; Xue et al., 2023; Feng et al., 2023; Shen et al., 2023a; 2024a;b; Shen & Tang. 2024; Wang et al., 2024; Fu et al., 2024; Shen al., 2025). Among them, the Latent Diffusion Model (LDM) (Rombach et al., 2022) performs the diffusion and reverse process in the latent space, instead of the pixel space, which largely reduces the computational burden. Among all the implementations of the Latent Diffusion Model, the stable diffusion model (SD) is the most famous one. Along with its public availability, the open-source community achieves tremendous success, with the emergence of many excellent fine-tuned models. These high-quality models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. In this paper, we aim leverage multiple such models to achieve stronger texl-to-image generation.
Model Merging. For vision or language understanding tasks, the effectiveness of parameter merg- ing (Frankle al., 2020; Wortsman et al., 2022b; Matena & Raffel, 2022: Ilharco et al., 2022b; Li et al., 2022; Don-Yehiya et al., 2022; Jin et al., 2022) can be interpreted from the perspectives of loss landscape (Wortsman al., 2022a; Ainsworth et al., 2022: Stoica et al.. 2023) and task arith- metic (Iharco et al., 2022a; Ortiz-Jimenez et al., 2023; Zhang et al., 2023). However, merging- -ased methods have not been widely studied for generation tasks, especially diffusion- based generation. Recently, many intuitive merging- merging-based methods have emerged. One of the popular methods is Meiggted-Merginn. which manually determines weights to merge each U-Net (Ronneberger et al., 2015) parameter of multiple SD models. Although the simplicity. Weighted Merging coarsely allo- cates weight to all U-Net blocks. As an improvement, Merge Block Weighted (MBW) allows for manually setting different merging weights for the parameters of distinct U-Net blocks, which pro- vides a more fine-grained merging strategy. Furthermore, to reduce the reliance on manual weights, autoMBW attempts to automate the merging process, which enumeratively selects the optimal com- bination of MBW weights by an aesthetic classifier. However, autoMBW is constrained by the performance bottleneck of the aesthetic classifier, and the enumerative selection leads huge time consumption to find the optimal settings, Such merging methods can be called the static methods. In this paper, we aim to leverage multiple diffusion models by the dynamic methods, i.e., ensembling.
Model Ensembling, Model ensembling is an effective method to achieve better performance (Zhou, 2012), which has been widely applied in various vision understanding tasks, such as classification (Zhao et al., 2005; Rokach, 2010; Yang et al., 2010), regression (Mendes-Moreira et al., 2012; Ren et al., 2015), clustering (Vega-Pons & Ruiz-Shulcloper, 20.1). While fewer works focus on the ensembling of generative models, because of the complexity of image space. Vision-Aide (Kumari et al., 2022) focuses on GANs (Goodfellow et al., 2014), which guides the optimization of a target generator by ensembling pretrained models as a loss. MagicFusion (Zhao et al., 2023) focuses on the diffusion models, which fuses the predicted noises of two expert U-Net denoisers to implement specific applications, such as style transferring and object binding. In this paper, we aim to efficiently ensemble multiple diffusion models to achieve general generation improvements".
GAN
METHOD
3.1 PRELIMINARY
As a type of generative model, the diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) consists of two processes, which are the diffusion process and the reverse process, respectively. In
'Noe that combining the styles/concepts of multiple diffusion models is also a common goal of ensem- bling/merging, which is not the interest of our work.
3

---

Published as a conference paper at ICLR 2025
Diffusion Model
Nolay Latent
Predicted Notte
Gunerated Latrel
SARM
Textaad Prompt fying golan caler
Timestop Testual Promutt
Treesfrrnc
Timeelep
Diffusion Model
Fewwre.tggregenn Spatlal-trare Block. Wlse (SABW) Feature Aggregator for the juth Block
Figure 2: Framework of ensembling multiple diffusion models by our AFA method.
the diffusion process (i.e., the forward process), the Gaussian noises are iteratively added to degrade the images over T' steps until the images become completely random noise. In the reverse process, trained denoiser is used to iteratively generate images from the sampled Gaussian noise.
When training, given an input image Xy and the additional condition (e.g., encoded textual prompts etc.), the denoising loss is defined as
denoise ((0,))) ||co (x,, c) cl/
Among them, X2 predefined scalar from the noise scheduler. is the added noise. Eo is the denoiser with the learnable parameters 0, which predicts the noise to be removed from the noisy image.
VaxXo + vI- OfE is the noisy image at timestep t E [1, where ag is a
one of the most popular diffusion models. It performs the diffusion and reverse process in the latent space which is encoded by a Variational Auto-Encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014). In LDM. the U-Net structure (Ronneberger et al., 2015) is used for denoiser, which contains three parts of blocks, which are the down-sampling blocks, the middle block, and the up-sampling blocks, respectively. Each down- sampling block skip-connected with corresponding samsanni block. These blocks are composed of Transformer (Vaswani et al., 2017) layers and ResNet (He et al., 2016) layers. In the Transformer layers, the cross-attention mechanism is employed to incorporate conditional textual prompts, which plays crucial role in text-guided image generation.
LDM stands
3.2 ADAPTIVE FEATURE AGGREGATION
Our proposed Adaptive Feature Aggregation (AFA) ensembles multiple diffusion models that share the same architecture but different parameters. Specifically, SABW is designed to integrate interme- diate features from multiple U-Net denoisers at the block level. Figure 2 demonstrates an example of ensembling two diffusion models, and each U-Net denoiser contains seven blocks by SABW. The forward process of AFA can be seen in Appendix A.
Given N diffusion models to be ensembled, whose U-Net denoisers contain K blocks, let c0, be the (j) i-th U-Net denoiser, and c, be its j-th block. For the timestep t, it transforms the input features
x into the output feature by
(2) Note that because of the same architecture of all EB,, ()) has the same shape, i.c., y E AR xwyxes where hy, Wj, and ej are the height, width, and channels, respectively. To achieve the output feature for the next block, a block (8) of SABW is employed to aggregate ( Jim
1 (x(()x)))))) .x,) yl = cg.
y( = ((v 1 R x x
(3)
4

---

Published as a conference paper at ICLR 2025
where yo is the parameters of the SABW block. As a whole, the j-th ensembled block of these U-Net denoisers can be formulated as
= = (x(") .,,t).
(4)
where θ the parameter collection of ip and all the 0,. It can be deduced that an ensembled block operates with the same functionality as a single block.
3.3 SPATIAL-AWARE BL.OC-- WISE FEATURE AGGREGATOR
The Spatial-Aware Block- Wise (SABW) feature aggregator is learned to aggregate output features of each block from multiple U-Net denoisers, We design SABW to learn spatial attention to reweight the contributions of these U-Net denoisers at each spatial position. Specifically, the spatial attention
map A for N U-Net denoisers is achieved by
F 00 (1y (Ly Ji c,t) R
(5)
()) softmax (F E [0, 1|
(6)
where, F(.) the output of the learnable network of SABW Jr. The ensembled output feature is
N y = c, = Σ
A
sy
1=1
where & is the element-wiss multiplication. A (1) E R x1 is the spatial attention map for the
i-th U-Net denoiser.
Note that the spatial- -attial-ttention mechanism of SABW differs to the typical multi-head self-attention mechanism (Vaswani et al., 2017), which simply uses three linear layers to project the input features into three separate spaces (i.e., query, key, and value), and then computes the attention map across all the projected features. Our spatial-attention mechanism based on an important experimental insight shown in Figure 1. The divergent denoising capabilities of multiple diffusion models are influenced by the prompts and the denoising steps, and they dynamically vary across different spatial locations. Thus, the projection method needs to account for the input prompt and the current denois- ing step. And it is sufficient to compute the attention map on each spatial location individually. As result, better generation performance can be achieved by enabling multiple models to collaborate at different spatial locations under the conditions of the input prompt and the current denoising step.
Specifically, based on the denoising loss (i.e., Eq. 1), the denoising capability? of co for (xp, c) at timestep & can be defined as -E,-N((o.) || - <| where é = cx c, t). For the spatial location (sr, v). the denoising capability is formulated as - (F..) ((2,)) - 6(a.))) 112, which considers the noise prediction errors at the spatial location (x,
For example, as shown in Figure 1, we employ three SD models to present their positional denoising capability on the specific image and textual prompt by Monte-Carlo sampled noises. We demon- strate the results four patches which indicate that the divergent denoising capabilities of the three models vary at different spatial locations. SABW is learned to incorporate these denoisers to achieve stronger denoising capability at all spatial locations the block level.
To implement 9bp in Eq. 5, SABW applies ResNet (He et al., 2016) layer to introduce timesteps by adding the time embedding into the feature map, and a Transformer (Vaswani et al., 2017) layer to introduce textual prompts by the cross-attention.
Y + concat (x Mi) (4
t
(8)
())
+ ResLayer ( + +t)) e g
(9)
O TransformerLayer(H! ,c) xlyxmyxd
(10)
P() ZemlnitConv(O.")) Xm; x
(11)
"Compared with the denoising loss in Fq. 1, the denoising capability is defined by adding the negative sign, because of negative correlation between the denoising capability and the denoising loss.
5

---

Published as a conference paper at ICLR 2025
COCO 2017
Draw Bench Prompts
COCO 2017
Draw Dench Frompts
HD 4.
CLIP4 CLIPT
AES
PS
HPSV2
IK
HDD &
IS
CLIP4 CLIPT
AES
PS
HPSV2
1X
.6724
2609
5,4002
21 27 5007
Model 12.12 5666
5,5641
21,8033
4238
Base Model B
13.45
5.43
.8775
2652
5,5013
21,624
21.72/6
2835
Dase Model B 12.41
5.59
.6818
2530
5,5027
21,7249
28,0343
4202
Base Model
12.32
6.32
.6800
2366
5,4881
21,8031
27.9652
3922
Bave Model C 12,05
585
.8638
2642
5,5712
21,4966
27,8089
3367
Wid. Mergang
10.65
6.93
.6861
2026
5,4815
21,7272
27:9066
30009
Wid. Mergang
11.55 6.56
.6824
.2031
5,5756
21,7516
28,0014
4387
MBW
11.03
6.51
.6870
2634
5,4812
21,7201 27,9080 3022
MBW
12.06
6.42
16826
2633
5,5772
21,7487 28,0029
4306
5,5056
21,4785
27.8192
5.62
(258
21,5135
.3513
MagicFusson
10.53 6.85
.6751
2620
5.343]
21,3840
27.8105
.3317
MaggieFusion
11.63 7,13
.6790
2610
5,4674
21,4270
27.9608
.1941
AEA (Ours)
.0926
5,5201
21.8363
27.9734
(Ours)
10.27
7,42
21.9059
28.0371
Quantitative comparison for the base models in Group (i.c., ER, MMR, and RV).
Table 1:
Table 2: Quantitative comparison for the base models in Group (i.c., AR, CR, and RCR).
where y is used to map the timestep into the time embedding with the same shape of Y d is the hidden state dimension. ZerolnitConv(()) is the convolution layer initialized by zero, which leads to equal attention weights for all denoisers before training. We aspire for the model to commence training from a completely equitable state.
3.4 TRAINING
Given the ensembled denoiser Fe, the training object is same as the denoising loss in Eq. 1,
= -xx. FIFe (x,, c, t) - e||
(12)
Note that the parameters Q contains both the parameters of SABW yp and the parameters of all U-Net denoisers [ where yp are learnable, and [0,
NN are frozen.
A well-trained AFA ensembles multiple U-Net denoisers to enhance the denoising capability in ev- ery block when experiencing different prompts, initial noises, denoising steps, and spatial locations. It leads to smaller denoising errors and stronger generation performance.
3.5 INFERENCE
The inference of AFA starts from the sampled Gaussian noise. Then, the diffusion scheduler (e.g., DDIM (Song et al., 2020), PNDM (Liu et al., 2022), DPM-Solver (Lu et al., 2022a;b), etc.) is applicd to generate images with multiple denoising steps.
For each inference step, the noise prediction relies the technique of Classifier-Fres Guidance (CFG) (Ho & Salimans, 2022), which is formulated as é.te हte + Acra é 2 Among them, anree ete EG and fCrG are the predicted noise, the predicted noise with condition, the predicted noise without condition, and guidance scale, respectively. The latent for the next step is X,-1
VOE X₁ vT-ai apred Finally, the generated image achieved from the latent by a VAE decoder.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Base Models. We select six popular models with the same architecture from $Dv1.5 in CivitAI. There are 12 down- samp-san blocks, middle block, and 12 up-sampling blocks for each model. These models are randomly divided into two groups, with three models in each group. The models from the same group will be ensembled.
Group contains EpicRealism (ER), MagicMixRRellitie 10 (MMR), and Realistic " (RV);
Pitteee//diiitaa.ccc///oeeee/// 2mod PoooeeVerrsiii Version ld-176225
hhtrpss//diittall.cc//noooag/// 2mode1 Version
I dd-330072

---

Published as a conference paper at ICLR 2025
Model II
Image Reward (IR)
0.-
0.-
Model
(Dun))
Imugee Roward (12)
0.3
0.4
0.0
011
0.3
0,4
MMR
CR
0.6
0.5
-0.0%
-0.
0.2
03
Infermnce Steps
H Infermnce Steps
(a) Group I
(b) Group II
(a) Group I
(b) Group II
Figure 3: Quantitative comparison between
Figure 4: Effect of varying inference steps.
AFA with the two base models.
Group II contains AbsoluteReality" (AR), CyberRealistic (CR), and RealCartoonReslistic (RCR).
Training. AdamW (Loshchilov & Hutter, 2017) is used as the optimizer with a learning rate of 0.0001 and weight decay of 0.01. Note that only the parameters of SABW are optimized, while those of U-Net denoisers are frozen. Our AFA framework is trained on 10,000 samples from the dataset JourneyDB (Pan et al., 2023) for 10 epochs with batch size To enable CFG, we use a probability of 0.1 to drop textual prompts.
Evaluation Protocols. For fair comparison, all the methods generate 4 images by DDIM Ho al. (2020) for 50 inference steps. The CFG weight fcra is set to 7.5. We evaluate the models with two datasets, which are COCO 2017 (Lin et al., 2014) and Draw Bench Prompts (Saharia et al., 2022), respectively. For COCO 2017, we apply four metrics, which are Fréchet Inception Distance (FID), Inception Score (IS), CLIP-I, and CLIP-T, respectively. For Draw Bench Prompts,we apply eval- uation metrics, which are AES Pick Score (PS) (Kirstain al., 2023), HPSv2 (Wu et al., 2023), and Image Reward (IR) (Xu et al., 2023), respectively. More details can be found in Appendix C.
Baselines. We compare AFA with several methods, including three merging- based methods (i.e., Beighted-Merginng MBW, and autoMBW) and one ensembling- based method (i.e., MagicFusion (Zhao et al., 2023)). The evaluation of these baselines follows the protocols.
4.2 QUANTITATIVE COMPARISON
The results of the two evaluation protocols are summarized in Tables 1 and 2. Generally, compared with base models, performance improvements are achieved by our AFA. For COCO 2017, after be- ing ensembled by AFA, FID, IS, CLIP-I, and CLIP-T exhibit enhancements under both Group and Group II. This indicates that the quality of our generated images is more closely with the dataset im- ages. Additionally, the contexts of our generated images align more closely with those of the dataset images and corresponding captions. For Draw Bench Prompts, all four metrics show improvements under both Group I and Group II. The improvement generated images surpass those generated by the base models. The improvements on PS, HPSv2, and IR suggest that the general generation capability be boosted by applying AFA. Overall, the quantitative results validate the effectiveness of our AFA in ensembling diffusion models.
AES suggests that the aesthetics of our
While MagicFusion attains impressive performance syle-transferring and object-binding by en- sembling two U-Net denoisers, it fails to achieve superior general generation when ensembling multiple U-Net denoisers. The merging-based methods, such as Weighted Merging, MBW, and au- toMBW. do not always produce models with superior generation capabilities. For instance, under Group II, Weighted Merging achieves a better IR compared to the base models, but under Group it yields lower IR. This could be attributed to the fixed contributions of merged models, which are determined by set of predefined merging weights. Consequently, these methods fail to effectively
14 phtrpp://dovvieal.cco/ooddeee/// http:///////////e/eeeoooooooooo
7

---

Published as a conference paper at ICLR 2025
Hose Model A
Base Model
Bose Model
Weighted Merging
MBW
Magichusion
APA (Ours)
Group (ER MMR 10)
(4) Prompt: VA artronauut is riding a horse.
Prompt: single clook sitting tabíc.
Group IAR RCRO
(c) Prompt: colored
(6) Prompt: Tho doys on the street.
Figure Qualitative comparison between AFA with the base models and the baselines.
facilitate collaboration between models to compensate for each other's weaknesses and improve their generative capabilities. Compared with these baselines, our AFA consistently achieves better performance, demonstrating its superiority.
As depicted Figure 3, we use AFA to ensemble two models from Group I or Group II, and present the IR scores of base models and the ensembled models Draw Bench Prompts. The results show that AFA significantly improves the performance when ensembling two U-Net denoisers. The results presented in Table Table 2, and Figure indicate that AFA be effectively applied to various model combinations and can be extended to accommodate different numbers of models. Meanwhile, consistently achieves superior generation performance compared single base model.
4.3 QUALITATIVE COMPARISON
As shown in Figure 5, we present qualitative comparison of the results from the base models, some of the baseline methods, and our AFA, which allows us to make several insightful observations. Firstly, AFA generate images with better aesthetics. For example, in Figure 5 (a) and (c). AFA can generate images with improved composition and finer details, compared to both the base models and the baseline methods. Secondly, AFA excels in achieving superior context alignment. For example, in Figure (b), all the base models and the baseline methods generate images containing more than one clock, despite the textual prompt specifying a single clock. In contrast, only our AFA generates the image jush just one clock, accurately reflecting the provided context. Thirdly, AFA can focus on the correct context of the base models and drop out the incorrect context. For example, in Figure (d), only the base model B generates the image with two dogs, while the baseline methods do fully trust the base model B, and generate the images with people. Only our AFA fully trusts the base model B, and generates the image with two dogs, which aligns with the textual context.
4.4 MODEL STUDY
Visualization of Attention Maps. As shown in Figure 6, we select to visualize the attention maps of the first and the penultimate (i.e., 24-th) blocks, both of which learned by ensembling models
8

---

Published as a conference paper at ICLR 2025
Prompt: Ans astronaut is rialng fiorse.
(a-1) I-st block
24-th (penuitimate) black
355
(b) Prompt: A pizza being cooked in the oven.
(b-1) J-sz block
(b-2) 24-hh (penuhimate) block
333
35t
203
450
3043
25t
300
25.5
Steps
Steps
Figure 6: Visualization of the learned attention maps. In each attention map, lighter-coloered posi- tions mean higher attention weights, while darker-coloered positions represent lower weights.
in Group II through our AFA. For the first block, the learned attention maps seem to concentrate solely on the feature map of AR at the first inference stage. However, the last few inference steps (i.e.. from the 30th the 50th step). the roles of the feature maps from different models begin to diverge. The attention maps tend to highlight the contextual body (i.c., the astronaut and the horse in Figure 6 (a), and the pizza and the oven Figure 6 (b)) in the feature map of RCR, while focusing on the background elements the feature maps of AR and CR. For the penultimate block, as the inference progresses, the learned attention maps gradually emphasize the contextual body within the feature map of AR, while concentrating on the background within the feature maps of both CR and RCR. Overall, the visualization of the learned attention maps indicates that our AFA can effectively ensemble diffusion models based on the context and the timesteps.
Effect of Fewer Inference Steps. As shown in Figure and Figure 7, we investigate the resilience of our AFA to a reduction in the inference steps. Figure 4 clearly shows a significant drop in the performance of the base models when the number of inference steps is reduced. However, the performance of our AFA remains almost stable when the number of inference steps is reduced from 50 to 20, and only experiences slight decline when the number of inference steps is reduced 10. Additionally, in Figure 7, can find that the quality of the images generated by the base models deteriorates with the reduction inference steps. contrast, our AFA stably generates high-quality images. Overall, it indicates that greater tolerance for fewer inference steps will be achieved after ensembling by AFA. Furthermore, while a single inference step may require a proportional increase in time, the total time consumed by the entire inference process does not escalate in the same manner.
Ablation Study. As shown in Table 3, conduct the ablation study to in- vestigate the necessity of each compo- con-
IR (Group 1)
IR (Group II)
within our AFA. Firstly, duct experiments with the following set- tings. (1) Only Ensembling Last Block: The AFA ensembles only the last block, leaving all other blocks unaltered. (2)
AEA (Full Model)
.4388
.4892
Only Ensembling Last Block
.4176
.4374
Block- Wise Averaging
.4001
.4372
Noise Averaging
.3919
.4355
Sputial Location
.4044
.4600
who Timestep
.4235
.4622
Block- Wise Averaging: Each block is en- sembled by averaging the output fea- ture maps, rather than using SABW. (3) Noise Averaging: Only the last block is ensembled by averaging the output feature maps (i.e., predicted noises). The observed performance
wro Textual Condition
.4163
.4599
Table Ablation study of AFA.
declines in these settings suggest that the design of the AFA effectively contributes to the ensembling of diffusion models, thereby leading to improved generation. Next, conduct an ablation study

---

Published as a conference paper at ICLR 2025
Buse Model A
Base Model
Base Model
AFA (Owrs)
Dase Model A
Base Medel
Dase Model C
AFA (Ours)
Namber Inference Sopa
30
10
(2) Prompt: single clock is sitting tabl.
(b) Prompt: Tho days on the street.
Figure 7: Qualitative comparison under different inference steps.
the spatial location from AFA, which learn only a single attention score (i.e., a scalar) for the feature of each U-Net denoiser, opposed to an attention map. The observed decrease in performance suggests that learning a spatial attention map to adaptively adjust the weights of various U-Net de- noisers in the level of spatial location is beneficial for the ensemble of different diffusion models. Finally, we individually ablate the introduced timestep and the textual condition. The decreasing performance indicates that both the timestep and textual condition play crucial roles for AFA.
Discussion on Computational Efficiency. (1) Ensembling Methods, Compared with another en- sembling method (i.e., MagicFusion (Zhao et al., 2023)), the computational efficiency of our AFA for a single inference step aligns closely with it. Both AFA and MagicFusion necessitate running all base models in one inference step. Although SABW in AFA introduces additional parameters, which leads to more computations, it has fewer than the base models (about 1/16). To be precise, our AFA may be marginally less efficient than MagicFusion on single inference steps, but the difference is practically negligible. (2) Merging Methods. Because the various models are collapsed into one model, the computational efficiency of the merging methods equals that of single base model. For the base models and the merging methods, single image generation takes about seconds with 50 inference steps. When ensembling three such base models, a single generation will take about 22 seconds with the same inference steps. Compared with ensembling, the computation efficiency of one inference step of the merging methods is significantly higher. However, as we discussed in Effeec of Fewer Inference Steps of Section 4.4, AFA shows remarkable tolerance for fewer inference steps. When limited to 20 inference steps, the performance of AFA does not significantly decline (as shown Figure 4), while the time consumed is reduced about 8.8 seconds. This suggests that AFA has comparable computational efficiency with that of the base models or the merging methods throughout the entire inference process.
CONCLUSION
In this paper, we aim to ensemble multiple diffusion models to achieve better generation perfor- mance. To this end, we propose the Adaptive Feature Aggregation (AFA) method, which dy- namically adjusts the contributions of multiple models at the feature level according to different prompts, initial noises, denoising steps, and spatial locations. Specifically, we design a lightweight Spatial-Aware Block-Wise feature aggregator that produces attention maps to adaptively aggregate the block-level intermediate features from multiple U-Net denoisers. The quantitative and qualitative experiments demonstrate that AFA achieves improvements in image quality and context alignment.
Limitation. A significant limitation of our AFA is that single inference step may necessitate proportional increase inference time, resulting in lower computational efficiency for that step. Fortunately, due to its high tolerance for fewer inference steps, the total time consumed by the entire inference process does not escalate in the same manner. This results in a computational efficiency that is par with that of the base models the merging methods.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China under Grants Nos. 62441225, 61972192, 62172208, 61906085. This work is partially supported by Collaborative In- novation Center of Novel Software Technology and Industrialization. This work is supported by the Fundamental Research Funds for the Central Universities under Grant No. 14380001
REFERENCES
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209. 04836, 2022.
Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint a 1.01324, 2022.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. NeurIPS, 2022.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering axt-to-image generation via transformers. NeurlPS, 2021.
Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In /CML, 2022.
David Eigen, Mare Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep 2013.
mixture of experts. arXiv preprint
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 23(120): 2022.
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mf -fdeoooii----iooiiinn---nng denoising-eepertss experts. In CVPR, 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In ICML, 2020.
Yoav Freund and Robert E Schapire. A application to boosting. In European conference computational learning theory, pp. 23-37. Springer, 1995.
generalization on-line learning and an
Yuchen Fu, Zhiwei Jiang, Yuliang Liu, Cong Wang, Zexuan Deng, Zhaoling Chen, and Qing Gu. Ap-adapter: Improving generalization of automatic prompts on unseen text-to-image diffusion models. In NeurlPS, 2024.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene- based text-to-image generation with human priors. In ECCV, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
11

---

Published as a conference paper at ICLR 2025
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In EMNLP, 2021.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
vrXiv:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprini arXiv:2212.04089, 2022a.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Si- mon Kornblith, Ali Farhadi. and Ludwig Schmidt. Patching open- ppcnbbocayylary models by interpo- lating weights. NeurlPS, 2022b.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09844, 2022.
Diederik Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes. arXiv preprint
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569 2023.
Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun- Zhu. Ensembling off-the-shelf models for gan training. In CVPR, 2022.
Yunsung Lee, Jin Young Kim, Hyojun Go, Myeongho Jeong, Shinhyeok Oh, and Seungtaek Choi. auchitarc multi-expert diffusion models. In AAAI, 2024.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Brench-rrain-merge:: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.0333000 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++ Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprini arXiv:2310.04378 2023.
Michael S Matena and Colin A Raffel. Merging models with fisher-weighte averaging. NeurIPS, 2022.
Joao Mendes-Moreira, Carlos Soares, Alípio Mário Jorge, and Jorge Freire De Sousa. Ensemble approaches for regression: A survey, Acm computing surveys (csur), 45(1):1-40, 2012.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10774, 2021.
12

---

Published as a conference paper at ICLR 2025
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. arXiv preprint arXiv:2305. 120.2 2023.
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, al. Journeydb: A benchmark for generative image understanding. arXiv preprint arXiv:2307.00711 2023.
William Peebles and Saining Xie. Scalable diffuuson models with transformers. In ICCV, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (:Xii:2224400122, 2022.
Ye Ren, PN Suganthan, and N Srikanth. Ensemble methods for wind and solar power forecasting- state-of-the-art review. Renewable and Sustainable Energy Reviews, 50:82-91, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, 2014.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. NeurIPS, 2021.
Lior Rokach. Ensemble-based classifiers. Artificial intelligence review, 33:1-39, 2010.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. In MICCAI, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar
Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022.
Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation. NeurIPS, 2024.
Fei Shen, Hu Ye, Jun Zhang, Cong Wang. Xiao Han, and Wei Yang. Advancing pose- geided image synthesis with progressive conditional diffusion models. arXiv preprint 30.00:23/0006311, 2023a.
Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du Zechao Li, and Jinhui Tang- Imagdressing-vl: Customizable virtual dressing. arXiv preprint arXiv:2407.12700 2024a.
Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang. Xiao Han, and Wei Yang. Boosting sistency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.02482. 2024b.
Fei Shen, Cong Wang, Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua. Long-term talkingface generation via motion-prior conditional diffusion model. arXiv preprint arXiv:2502.09533, 2025.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.077 07.2622 2023b.
13

---

Published as a conference paper at ICLR 2025
Jascha Sohl-Dicksteinn Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In JCML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffmann Zipit! merging models from different tasks without training. arXiv preprint wrXiv:2305.03055 2023.
Ming Tao, Hao Tang, Fei Wu, Xiao- Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. CVPR, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Sandro Vega-Pons and José Ruiz- Suiz-Shulcloper. A survey of clustering ensemble algorithms. Interna- tional Journal of Pattern Recognition and Artificial Intelligence, 25(03):337-372, 2011.
Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:24000 02001, 2024.
Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint (...i::2210.14800 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint XrXhv:2044.05977 2023.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In CVPR, 2018
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint arXiv:2305.18225, 2023.
Pengyi Yang, Yee Hwa Yang, Bing B Zhou, and Albert Y Zomaya. A review of ensemble methods in bioinformatics. Current Bioinformatics, 544):296-308, 2010.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In CVPR, 202
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emii-vilg: Unified generative pre-training for bidirectional vision- language generation. arXiv preprint arXiv:2 2021b.
14

---

Published as a conference paper at ICLR 2025
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306. 14870, 2023.
Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, and Wenjing Yang. Magicfusion: Boosting text-to-image generation performance by fusing diffusion models. arXiv preprint arXiv:2303. 13126, 2023.
Ying Zhao, Jun Gao, and Xuezhi Yang. A survey of neural network ensembles. In 2005 international conference on neural networks and brain, volume 1, pp. 438-442. IEEE, 2005.
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
Minfeng Zhu, Pingho Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adver- sarial networks for text-to-image synthesis. In CVPR, 2019.
A ALGORITHM OF AFA
The forward process of AFA can be seen in Alg. 1.
Algorithm One forward process of Adaptive Feature Aggregation (AFA).
1: Input: Timestep t, Condition Input latent Xx, U-Net denoisers {Ee, Jin each of which contains K blocks, and SABW f!)) for j-th U-Net block.
2: Output: Predicted noise é,
3: Initialize an empty stack store the features from the skip connections;
4: (2) +- Xx;
5: for block index j from K do
6: D Ensembling the j-th U-Ner block by SABW feature aggregator f
7: for U-Net denoiser index i from to N do
8: Get the output feature ()) base on Eq. 2;
9:
end for
10:
Get y! by aggregating all yt, base on Eq. for the next block;
(
11:
if (j + 1)-th block is down-sampling block then
x ((1-1) + (3) and push ( into S:
12:
else if (j 1)-th block is up-sampling block then
Pop y from and x (i+1) +- concat(y(" (1) y?):
14:
else x(-1)) + (1).
16:
end if
18: end for
é, + xi (K) 20: return predicted noise ét-
B DETAILS ABOUT PARAMETERS
The parameters of AFA are concentrated in the SABW module. The SABW module comprises three components, which are ResNet layer, Transformer layer, and convolution layer, respectively. Among them, the number of parameters in both the ResNet layer and the convolution layer varies depending on the number of ensembled models. For instance, when AFA ensembles two diffusion models, SABW contains approximately 42.425 million parameters. With each additional diffusion model ensembled, the parameter count of SABW increases by about 207 million. Therefore, when three diffusion models ensembled, the total number of the trainable parameters is 45.132 million, and with four models, it rises to approximately 47.839 million. In conclusion, we estimate that AFA encompasses close to 50 million trainable parameters.
15

---

Published as a conference paper at ICLR 2025
C DETAILS ABOUT EVALUATION PROTOCOLS
COCO 2017 comprises 118,287 and 5,000 image-caption pairs in the test and validation sets. All the models generate images with resolution of 256x256. We apply four metrics evaluate the generation performance, which are Fréchet Inception Distance (FID), Inception Score (IS), CLIP- and CLIP-T, respectively. FID and IS applied to the test set, while CLIP-I and CLIP-T applied to the validation set. Both FID and IS assess the quality of generated images. Note that lower FID means better quality. CLIP-I is the similarity between the CLIP (Radford al.. 2021) images embeddings of generated images and that of images from the image-caption pairs. CLIP-T the CLIPScore (Hessel et al., 2021) between the generated images with the captions.
Draw Bench Prompts contains 200 evaluation prompts. All the models generate images with a res- olution of x12. We apply 4 evaluation metrics, which are AES, Pick Score (PS) (Kirstain al., 2023), HPSv2 (Wu et al., 2023), and Image Reward (IR) (Xu et al., 2023), respectively. All three metries evaluate the performance by model that simulates human preferences. The evaluations are conducted 20 times ensure statistical significance. And the averaged metrics are reported.
D EFFECT OF MORE TRAINING SAMPLES
We conduct an experiment to assess the influence of an increased number of training samples on the performance of AFA. For this purpose, vary the number of training samples from 18,000, increasing the increment of 2,000. As depicted in Figure 8, the trend of the lines initially rises and then stabilizes after reaching 10,000 training samples. This suggests that approximately 10,000 training samples are sufficient for AFA. More training samples do not bring greater performance. It may be due the inherent limitations in the generative capabilities of the base models.
Group
Image Reward (IR)
Group
.475
.45
.425
14k
Training Samples
Figure 8: Effect of varying training samples.
E MORE QUALITATIVE COMPARISONS
More qualitative comparisons between our AFA and the base models can be found in Figure 9.
F ENSEMBLING BY MIXTURE-OF-EXPERTS
Another intuitive approach to ensembling multiple models involves the use of the Mixture-of- Experts (MoE) method Shen et al. (2023b); Eigen et al. (2013); Riquelme et al. (2021): Fedus et al. (2022); Du al. (2022); Chi al. (2022). The MoE framework incorporates multiple models, each designated as an expert. Given input, MoE employs a routing mechanism to determine the most suitable expert. The input then processed by the selected expert, and its output considered as the output of the entire model. The number of parameters in MoE proportionally increases with the number of ensembled experts. However, the parameters of MoE are sparse, indicating that not all parameters utilized during the inference process. Despite ensembling multiple models, only one of them is activated during inference, which makes MoE efficient.
Several text-to-image generation methods leverage MoE-based diffusion models, where multiple denoisers are ensembled along the timesteps. In these methods, only one denoising expert acti- vated each timestep. For instance, ERNIE--ViLG 2.0 (Feng al., 2023) and eDiff-I (Balaji al.,
16

---

Published as a conference paper at ICLR 2025
Bose Model A
Base Model
Base Model
AFA (Ours)
Base Model A
Dase Model
Base Model
APA (Ours)
Prompt: A. sunset.
(1) Prompt: Aniioe illustrationn the Great Pyraaii string next Parttienen blut nightt sky of reiling energy.
(b) Prompt: The Starry Night.
(g) Prompt: a car natting a crystal kall wirli the number written on it black marker
Prompt:7 Kadper naxt tree small waterfall the background.
Prompt: woman long orange hair over background that is a skutch of city skyfine.
TOP STATPIO
STOP
STP
Prompt: A wearing formal clothes and wearing topiaat. Oil puinting in the style of abstract cubism.
Prompt: stop sign with large befiind
Feel
(c) Prompt: & ruccoon wearing formal cloties and waariny uphhtt. Oif patnning the style Vitceet Van Goyh.
(j) Prompt: A puinting of fox in the style of The Sturry Niyht
Figure 9: More qualitative comparisons between our AFA and the base models.
2022) divide all the timesteps into blocks, each of which consists of consecutive timesteps and is assigned to a denoising expert. Building on this idea, MEME (Lee et al., 2024) introduces denoisers with different architectures, tailored distinct timestep intervals. In these MoE-based methods, the denoising capability of each denoiser is restricted into some specific consecutive timesteps through large-scale full-parane parameeers training, which is not suitable to the task of model ensembling.
Inspired by the MoE-based diffusion models discussed above, an intuitive MoE-based ensembling method is employ a trained router to select base denoiser for the current timesteps. We refer to this as the Denoiser-Level MoE method. Furthermore, to mirror the concept of our AFA, we propose another strategy of Block-Level MoE method, which uses router to select the next block. The key difference between the Block-Level MoE method and our AFA lies in their operations: AFA manipulates the outputs of all blocks, whereas MoE directly acts on the input. We will introduce the details of Block-Level MoE method, as the Denoiser- Level MoE method be considered special case of Block- Level MoE method, where the selection occurs only before the first block.
Given N diffusion models, we employ router ((() to determine which model's block will be used, where ( denotes the learnable parameters. Specifically, r output the logits of N models.
(6)
10 -, -000000 ., c) N
(13)
17

---

Published as a conference paper at ICLR 2025
During training, the probability for the i-th model is defined by the Gumbel softmax,
exp((1 + gg/T) Pt Σ K= exp((11f +g//T)
(14)
where, (0) is the i-th element of ( is the Gumbel noise. T is the temperature coefficient. To address the issue of bacbbapkpooooo pack-propagation incapability, we apply the reparameterization trick. Given the probability distribution p = pill- the reparameterized distribution is
(15) Among them, one the one-hot function, which sets the maximum probability to 1 and the other to 0. sg(+) is the stop gradient function. During the forward process. the distribution is defined as p' = onchot(p). However, during the backward process, the distribution p' = p is used to compute the gradients. The output of the j-th block is
p' = onehot(p) + p sg(p)
Ep' @ concat( ((y 1N M
(16)
From Eq. 16, we can infer that the forward time is proportional to the number of ensembled models during training. The training loss is the denoising loss.
During inference, only the model that achieves maximum logits will be used to process the block input,
y = ( (x( c,
= max ( iE[M]
(17)
The forward time of the ensembled model the same as that of a single model during inference.
To implement the router , we apply a ResNet layer to introduce timesteps by adding the time embedding into the feature, and a Transformer layer to introduce textual prompts by the cross- attention. Specifically, given the input for the j-th block (i.e., x(") E 12a X()x)))), the output logits for models (i.e., 1()) are achieved by
h(6) ResLayer(x; (5) + (t)) Ruxxxnx
(18)
0 ()
t TransformerLayer((1" (1) c) E plyxxyydd
(19)
(6 + Linear(AvgPool(o))) E ER.
(20)
Although the time required for a single forward pass in the MoE methods approximately equiva- lent to that of a single model, they face challenges in performing parallel computations when gener- ating multiple images simultaneously. This limitation becomes evident during classifier- free guid- ance. If the router selects two different models to process the conditioned and unconditioned inputs, MoE can only execute two sequential forward passes, rather than concurrently processing these two scenarios within single forward pass. Moreover, when generating K images once, the MoE method has a high probability of necessitating 2K forward passes. While these 2K inputs can be divided among N models, this only reduces the complexity associated with the number of forward passes from OKK) to O(N). This reduction still represents a substantial computational demand.
The quantitative comparison among the base models, the MoE methods, and our AFA is summarized in Table 4. AFA demonstrates superior performance compared to the MoE methods. This may be because the base denoisers have vell-balanced denoising capabilities across all timesteps, and selecting specific denoiser for a given timestep does not significantly enhance the overall denoising capability of the ensembled model. Furthermore, we compare AFA and the MoE methods using fewer inference steps (i.e., 20 steps). As shown in Table 5, unlike our AFA, the MoE methods do not demonstrate robustness with fewer inference steps, performing similarity to the base models.
The efficiency comparison among the base models, the MoE methods, and our AFA is summarized in Table 6. Firstly, the MoE methods have a similar number of parameters and training TFLOPs as our AFA. This is because the MoE methods still require all base model be available for selection, and during training, each base model must perform a forward pass. Secondly, during inference, the MoE methods exhibit lower TFLOPs compared to AFA, only one base model is activated each
18

---

Published as a conference paper at ICLR 2025
timestep. However, despite this advantage in TFLOPs, the inference time of the MoE methods is comparable to that AFA due to their inability to leverage parallel inference effectively. Moreover, the MoE methods do not support fewer inference steps. In contrast, when AFA reduces the number of inference steps to 20, it still outperforms the MoE. methods, even when they use 50 inference steps, while also achieving lower TFLOPs and faster inference. This highlights the efficiency and effectiveness of our AFA.
Group
Group
ABS
PS
IIPSv2
HPPSv2
Base Model A
5,4102
21.6279
27.8007
.3544
5,5641
21.8013
28.0183
4238
Base Modell B
5,5013
21.4624
27.7246
.2835
5.5027
21.7249
28.0343
.4202
Base Model C
5,4881
21.8031
27.9652
.3922
5,5712
21 21.4
27.8089
.3367
Denoiser-Leeve MoF.
5,4727
21 21.6893
27.5345
4019
5,5638
21,5531
27.7843
4380
Block-Level MoB
5.5037
21.7690
27.8749
.037
5,5718
21,7636
27,9842
4463
(Ours)
5.5201
21.8263
27.9734
5,5798
21.8059
28.0371
Table 4: Quantitative comparison between AFA with the base models and the MoE methods.
Group
Group
AES
HPSv2
HPSv2
Demoiser-LLew Level MoF!
1,2310
2.9435
4.1739 5,3583
.0184 .0988
2,4556
3.6723 4.2368
5.3415 4,7252
.0377
Block-Level MoB
1,6854
3,4573
2.5734
0344
(Ours)
5,4951
21.8093
27.7347
5.5322
21.7830
27.9775
Table 5: Quantitative comparison of AFA and the MoE methods with 20 inference steps.
image
images
images
# params.
T-T1.PPs
1-TFL.OPs
Times (6)
1-TFL.OPs
Times (s)
1-TF1.OPs
Times (s)
Base Model!
859.52M
70.24
2.94
140.45
5.61
280.88
6,87
Denoiser-Level MoE"
2581.01M
6,36
72,37
5.36
144.75
10.37
289.48
20.81
Block-Lovel MoE"
2621 1621
6.43
74.41
5.39
148.82
10.43
297.65
21.00
AFA (50 inf. steps)
2621 3621
6.42
218.74
8.98
437.48
15.57
874.96
21.33
AFA (20 steps)
2621.03M
6.42
86.61
173.22
Table 6: Efficiency Comparison between AFA with the base models and the MoE methods when ensembling three base models to generate images with a resolution of 512x512. T-TFLOPs and TFLOPs denote TFL.Ps for training and inference, respectively. Note that only the parameters of denoisers are considered. Note that all the inference times are evaluated in our A6000 environment over 20 runs. *oote that both the base models and MoE methods are evaluated with 50 inference steps, because they cannot achieve comparable performance with fewer inference steps.
G ENSEMBLING MORE MODELS
To evaluate the scalability of our AFA, we apply it to ensemble larger number of base models. Specifically, we select and ensemble several base models with best performance, and test IR metrics in Draw Bench Prompts. As shown in Table 7, ensembling more base models leads more perfor- mance improvements. Additionally, increasing the number the ensembled base models enhances the tolerance for fewer inference steps.
H COMPARISON ON FEWER INFERENCE STEPS
We compare our AFA with the baseline methods under fewer inference steps. As shown in Table 8, AFA not only outperforms the baseline methods higher inference steps but also demonstrates significantly better performance under fewer inference steps. This highlights AFA's superior toler- ance the fewer inference steps compared to the baselines.
19

---

Published as a conference paper at ICLR 2025
base model
2 base models
3 base models
4 base models
5 base models
6 base models
50 inference steps
0.4238
0.5003
0,4892
0.4967
0.5042
0.5347
40 inference steps
0.4065
0.4834
0.4874
0.4906
0.49884
0,5310
inference steps
20 inference steps
0.2103
0,3641
0,4803
0,4840
0.4975 0,4423
0.5304
inference stops
-0.5255
0.1844
0.4013
0.4285
0.5135
Table 7: Performance (IR) comparison across different numbers of ensembled base models and varying inference steps.
Group 1
Group II
inference sleps
50
40
30
20
10
50
40
30
20
10
Wid. Merging
0.3909
0.3104
0.2451
-0.0535
-0,7536
0.4387
0.4123
0.2942
0.1046
-0,5105
MBW
0.3922
0.3041
0.2345
0.0341
-0.7731
0.4396
0.4094
0.2893
-0.0031
-0.5524
autoMBW
0.3672
0.3158
0,2510
0.4173
-0.044
0.3513
0.3545
0,2502
0.0841
-0.5841
MugicFusion AFA (Ours)
0,3104
0.2349
0.0175
0.4194
0.3408
0.0818
0.4388
0.4389
0,4238
0,4191
0.3575
0.4892
0,4874
0,4846
0,4803
0.4013
Table Quantitative comparison (IR) on fewer inference steps.
I EFFICIENCY COMPARISON
We compare the efficiency of merging methods and ensembling methods. As shown in Table 9, when using the same 50 inference steps, ensembling methods (e.g.. MagicFusion and AFA) do not have efficiency advantage parameters, TFLOPs, and inference time all increase linearly with the number of base models.
single base models merging methods. The number of model
However, thanks AFA's high tolerance for fewer inference steps, it can achieve similar perfor- mance with reduced steps, as demonstrated in Table 7, while the baseline methods fail to maintain performance, as shown Table Therefore, under fewer inference steps, AFA achieves compara- ble efficiency (TFLOPs and inference time) to that of single base models and merging methods.
2 base mudels
base models
base models
params.
TFLOPs
Times (s)*
params.
TELOPs
Times (s)
params.
TFLOPs
Times (8)
Merging Method
859. 52M
70.24
2.94
52M
70.24
2.94
859.52M
70.24
2.94
MagicFusion
1719.04M
138.66
2578.56M
206.88
9.13
5157.12M
411.53
22.41
AFA inf, steps)
1761.47M
148.64
2621.03M
8.98
5210.37M
23.15
AFA (30 inf. steps)
176147M
94.24
3.21
AIA inf. steps)
2621.03M
86.61
3.62
AFA (10 inf. steps)
5210.37M
103.68
4.58
Table 9: Efficiency Comparison between the merging methods and the ensembling methods when generating image with resolution of 512x512. NNote that only the parameters of denoisers are considered. Nott that all the inference times are evaluated in our A6000 environment over 20 NNote that both the merging methods and MagicFusion are evaluated with 50 inference steps, because they cannot achieve comparable performance with fewer inference steps.
J GENERALITY ON OTHER ARCHITECTURES
To evaluate the generality of our AFA other diffusion architectures, select three SDXL (Podell et al., 2023) models and two FLUX. II [ models from CivitAl.
SDXL consists two U-Nets: the denoising U-Net and the refining U-Net. Both U-Nets shares similar architecture SDv1.5, but contain larger blocks. Consistent with the method used for ensembling SDv1.5 models, employ the SABW module to aggregate features from each block.
hhttps::/blackkrrre wtt.:/anaaaannniiiaaa nng forest- labs/
20

---

Published as a conference paper at ICLR 2025
FLUX .I [dev] is DiT-based model (Peebles & Xie, 2023). To ensemble models with this archi- tecture, we also utilize the SABW module to aggregate features from each DiT Transformer block.
The selected SDXL models NoobAI-XL", 17 iNiverse-Mix" and epiCRealism-XL" The lected FLUX. [dev] models are PixelWave and Verus VerusVisions Vision"
As shown in Table 10, when ensembling models with the or FLUX .I [dev] architectures, the performance of the ensembled model surpasses that of the individual base models, demonstrating the generality of our AFA different model architectures. However, the observed improvement is relatively small, likely because the base models already exhibit strong performance, and ensembling leads only marginal gains. Additionally, due the large size of the base models, ensembling multiple such models may be less practical.
worth noting that our AFA cannot be applied ensembling models with different architec- tures, as the misalignment of block features makes block-wise aggregation challenging, We view ensembling models with diverse architectures as a promising direction for future research.
[dev]
AES
PS
HPSv2
AES
HPSV2
IR
Base Model A
6,1948
21.7854
28.0452
66314
6.2341 6.2346
21.8843
28.0958
.7341
Model
28.0531
28.1003
Base Model C
6.1852
21.7752
27.9984
.6339
AFA (Ours)
6.2190
21.7959
28.0894
.6342
6.2490
21,8931
28.1194
.7370
Table 10: Quantitative comparison of AFA with different diffusion architectures.
K DISTILLATION TO FEWER INFERENCE STEPS
While our AFA demonstrates robustness fewer inference steps, we aim to further distill the en- sembled model into significantly fewer steps. Specifically, we explore using LCM (Luo et al., 2023) to achieve this. In our experiments, we kept the parameters of the denoisers frozen while training only the parameters of the SABW modules. However, as shown in Table 11, this approach is un- successful. A possible reason for this failure that the frozen parameters may have impeded the distillation process.
Original Ensembled Model
Distilld Ensembled Model
Inference Steps
20
IR
0.4892
0.4803
0.4793
-0.5846
-0.5594
-0.5952
Table 11: Experimental results (IR) for distilling the ensembled model with AFA into fewer infer- ence steps using LCM.
L EVALUATION ON MORE DATASETS
To validate the generality of AFA, we evaluate it additional datasets, which are DiffusionDD (Wang et al., 2022), JourneyDB (Pan et al., 2023), and LAION-COCO" respectively. We randomly selected 50,000 samples from each dataset to compare the performance of AFA against the base models and the baseline 2e
htt.:e/e/..aaaaa..oeeee/e//////p 18 wtttep///hhtaaaooo//ss//////dees
20 httte://hhaaalco//////////////a2
Note that for JoumeyDB, ensured the selected samples distinct from those in the training
21

---

Published as a conference paper at ICLR 2025
The quantitative comparisons the three additional datasets are presented in Table 12, Table 13, and Table 14, respectively. The results demonstrate that our AFA consistently outperforms both the base models and the baseline methods, highlighting the generality and effectiveness of our approach.
Group
Group
FID J.
1S
CLIP-1
CLIPTT
AES
PS
HPSV2
IR
FID
15
CL.IP-1
CLIP-TT
AFS
PS
HPSv2
1R
Base Model A
14,78
6.88
/643
.2957
5.557
23.4113
29.9126
.4544
14.31
7.02
.678
3124
5,612
23.5527
30.1521
A413
Base Model
15,01
6.53
.5133
.3100
5.456
23.1125
29,131
.423
14.98
3128
5,502
23.3128
30,0035
.307
Model
15.13
2245
5.478
23.3123
29.9441
.4399
15.02
2983
5.481
23.3420
29,9113
.4291
Wid. Marging
17.41
6.37
6214
.776
5.213
23 29.2
28 9919
4007
16.98
6.81
.6231
2084
5.501
23.2139
28.7310
A147
MBW
16.67
6.81
6678
23.
2333333
.415
5,493
23,4412
20.0012
sWMMBW
16.66
6.76
/632
.663
5.377
23. 3674
5432
.4233
15.17
7.00
.381
2891
5,551
23.3584
29:8413
4298
MagicFusion
6.98
.5611
5,674
29.9241
.4460
7.02
3125
5.611
23.5049
29,9941
AFA (Ours)
12.32
7.39
.6763
.3691
5.773
23.5132
30,0946
.4763
13.98
7.11
.6122
.211
5.698
23.6139
30.1931
.4712
Table 12: Quantitative comparison in Group I and II on DiffusionDB.
Group
Group
FID J
1S
CLIP-
CLIP-T
AES
PS
1IPSx2
IR
FID
1S
CLIIP-1
CLIP-T
AES
P5
EIPS=2
IR
Base Model A
17.80
5.63
.5324
.2484
4.520
19,2066
24. 24.5997
.3605
17:21
5.25
.5378
2213
4,577
19.1089
24.5196
.3633
Model
18.17
.5579
4.466
18
.3488
5.72
2588
4.482
18,9009
24.3972
Model
17.98
5.62
.5270
.2434
4,507
19.1876
24.5743
.3554
17.87
5.61
.5169
2398
4.445
18,9523
24.2945
.343
Wid. Merging
20.98
5.26
.5170
.2114
4.288
19.1237
23,8558
.3327
20.55
5.50
.5108
2452
4.455
18,8782
23.3782
.3399
MBW
19.74
5.58
-5218
4.399
19.1381
23,7068
.3371
18:35
5.24
.5221
2378
4.484
19,0700
23,5806
W
20.07
5.54
.5004
.2277
4.429
19.1960
23.6664
.3508
15.04
5.70
.3182
2325
4.501
18.9612
24.2598
.3500
MagicFusion
17.76
5.76
.5422
.2591
4.672
19:2147
5909
.3666
5.72
2573
4.550
19.0955
24.3816
AFA (Ours)
14.86
6.84
5587
.3063
4.763
19.3324
24.7316
.3649
16.72
5.78
.5482
.2646
4.651
19.1986
24.5480
.3990
Table 13: Quantitative comparison in Group I and II on JourneyDB.
Group1
Group
FID 4
CLIP-1
CLIP-T
AES
PS
HPSx2
UR
FID
1S
CLIP-1
CLIP-T
ABS
HPPS22
Base Model A
13.82
7,82
.7328
.3432
6,260
26.5012
33,8765
.1962
13:26
8.06
.75.9
3359
6.402
26.8726
34,4362
.5080
Model
14.12
.7397
3367
26
.4554
.7583
.617
6.286
34.2635
Model C
14.08
7.73
.7780
.3571
6.197
76 4257
33,5805
4886
13.84
7.90
.7784
3379
6.254
26.6009
34.1549
.4799
Wid. Merging
16.46
.7101
.3000
26
32.8551
.4533
15:79
3415
6.271
26.5130
32.8277
A760
MRW
15.45
7.68
.7343
.3275
6.071
26.
32,6951
.668%
14.01
8.07
7344
3327
6.311
26.7870
33.1185
15.61
7.67
.7218
.3101
6,092
26. 4808
32 34.6466
.852
13.98
8.00
.7328
3380
6.343
26,6503
34.07944
.9466
Magickusion
13.80
7:52
.1477
.3594
6,425
2694
33,8799
.5048
13:13
8.05
.2632
3632
6.421
26.8431
34.2797
.5024
AFA (Ours)
11.66
S.36
.7716
.241
6,582
26.6673
34,1098
.5331
12.84
8.16
.7717
.335
6.552
26,9677
34,4844
.5405
Table 14: Quantitative comparison in Group and II LAION-COCO.
M ENSEMBLING MODELS WITH HIGHLY CORRELATED FEATURES
As the number of the base models increases, reliance on multiple base models may leads to overfit- ting to correlated features. To evaluate the robustness of our AFA against highly correlated features, we design experiment. Specifically, we selected one high-quality base model and one low-quality base model. Using AFA, we ensemble the high-quality model with several low-quality models to evaluate whether the ensembled model will perform toward that of the low-quality model due the dominance of the correlated features from the low-quality model.
As shown in Table 15, although the performance the ensembled model slightly declines when en- sembling with large number of low-quality models, it still outperforms the high-quality model. This demonstrates that our AFA exhibits strong robustness highly correlated features.
N TRAINING WITH DATASET OF LOWER QUALITY
In this paper. we train our AFA using a high-quality dataset, JourneyDB. This raises the concern of whether AFA's high performance is primarily attributed to the quality of the training dataset.
22

---

Published as a conference paper at ICLR 2025
AES
PS
HPSv2
IR
High-Quality Base Model
5.5641
21.8013
28.0183
.4238
Low Quality Base Model
5.5013
21.4624
27.7246
.2835
AFA High-Quality Model + Low-Quality Model)
5.6013
21.9341
28.3485
.4746
AFA High Quality Model + Low wow-Quality Models)
5.60000
21,9348
28.3398
.4739
AFA High- Quality Madel Low Quality Models)
5.5974
21.8248
28.3399
.4730
AFA (1 High-Quality Model + Low-Quality Models)
5,5831
21.8019
28,2033
.4993
Table 15: Quantitative comparison of ensembling models with high-correlated features.
To address this, we also train AFA using LAION-COCO, a dataset of lower quality compared to JourneyDB..
As shown in Table 16, the model trained on LAION-COCO performs similarly to the one trained on JourneDB,, indicating that AFA is not sensitive the quality of the training dataset. Its consistent performance across datasets of varying quality highlights its robustness and generalization ability. Regardless of whether the training dataset is high-quality low-quality, AFA effectively leverages the features the base models to deliver stable and reliable results.
Group
Group
CLIIPT
HPSv2
FID
JB
.626
2675
5.5201
218863
27,9734
10.27
.2177
5,5798
8039999
28,0371
LAION-COCO
2664
21,8257
27.9881
10.39
7.39
6861
5,5789
8077
28.0363
Table 16: Quantitative comparison in Group I and II trained on JourneyD and LAION-COCO.
O MORE EXPERIMENTAL DETAILS
We train our AFA in an environment equipped with 8 NVIDIA V100 GPUs, each with 32GB of memory. When ensembling two base models, the GPU memory consumption is about IIGB under FP16 precision and batch size When ensembling three base models, the memory consumption increases to 14GB under the same settings. For six base models, the memory consumption reaches about 30GB. Due to the frozen parameters of the base models, training our AFA demands relatively modest GPU memory, even when ensembling larger number of base models.
23


Reviews:
[{"review_id": "eY3rA01hEP", "reviewer": "Reviewer_W7j2", "summary": "This paper introduces Adaptive Feature Aggregation (AFA), a novel ensembling method for text-to-image diffusion models. Unlike conventional static parameter merging approaches, AFA dynamically aggregates features from multiple diffusion models using a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator. The method adaptively adjusts model contributions based on various states including prompts, initial noise, denoising steps, and spatial locations. The key technical innovations lie in block-wise feature aggregation and spatial attention mechanisms, enabling effective ensemble that maximizes strengths while minimizing weaknesses of individual models.", "questions": "- Given that recent work in diffusion models increasingly focuses on efficiency, have you explored knowledge distillation or other compression techniques to reduce the inference overhead of AFA?\n- How does the spatial attention mechanism in SABW interact with different types of prompts? Are there certain prompt patterns where timestep-specific expert selection might be more appropriate than spatial aggregation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel perspective on model ensembling that goes beyond both conventional parameter merging and timestep-specific expert selection strategies\n- Comprehensive consideration of multiple states (prompts, noise levels, timesteps, spatial locations) for adaptive feature aggregation, offering more fine-grained control than existing expert-based approaches\n- Well-designed empirical validation with thorough ablation studies and clear visualization of spatial attention patterns\n- Practical advantages in terms of implementation, requiring only the training of SABW while keeping base models frozen\n- Demonstrates robust performance with fewer inference steps, suggesting potential for real-world applications\nStrong quantitative improvements across multiple metrics (FID, IS, CLIP scores) compared to both single models and existing ensemble methods", "weaknesses": "- While the paper introduces a novel method, it lacks direct comparison with recent mixture-of-expert diffusion approaches such as ERNIE-ViLG 2.0 [a], eDiff-I [b], and MEME [c], which also address the dynamic nature of the denoising process with multiple models. Basically, I think that the reader should be informed about the ideas that can be thought of in the direction of improving DPMs using multiple models, such as using different expert models along the time axis, and how this study proposes a different concept from those studies. Specifically, comparing performance metrics (e.g., FID, IS, CLIP scores), computational efficiency (inference times, resource utilization), and qualitative differences in generated images would provide a clearer understanding of how AFA stands relative to these methods. Including such comparisons or ablation studies would strengthen the claim that spatial-aware aggregation offers advantages over or complements timestep-specific expert selection. In the process of improving DPMs by utilizing multiple models, I believe that research directions along two axes, spatial and temporal, should be compared.\n- Missing theoretical analysis on when and why spatial-aware feature aggregation outperforms timestep-specific expert selection. Incorporating theoretical insights—such as analyzing the information flow, gradient propagation, or the capacity of spatial attention mechanisms within the diffusion process—could help elucidate the conditions under which AFA is most effective. A comparative theoretical framework would enhance the understanding of the benefits and limitations of spatial versus temporal adaptation strategies.\n- Computational overhead during inference could be problematic for practical applications, yet alternatives or optimizations are not thoroughly explored. I think the limitations associated with these computational costs should be clearly explained and guidelines for practical application should be provided.\n- Experiments limited to SDv1.5-based models, raising questions about generalizability across different model architectures\n- Lack of analysis on the trade-off between the granularity of adaptation (spatial/temporal) and computational cost\n\n---\n[a] Feng, Zhida et al. “ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 10135-10145.\n\n[b] Balaji, Yogesh et al. “eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers.” ArXiv abs/2211.01324 (2022): n. pag.\n\n[c] Lee, Y., J. Kim, H. Go, M. Jeong, S. Oh, and S. Choi. “Multi-Architecture Multi-Expert Diffusion Models”. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, Mar. 2024, pp. 13427-36, doi:10.1609/aaai.v38i12.29245.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJmtSDCLuh", "reviewer": "Reviewer_WgNL", "summary": "This paper introduces a novel ensembling method called Adaptive Feature Aggregation (AFA) for leveraging multiple high-quality diffusion models to enhance text-to-image generation capabilities. The key innovation of AFA is its ability to dynamically adjust the contributions of multiple models at the feature level based on various states, such as prompts, initial noises, denoising steps, and spatial locations. This is achieved through a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptively aggregates block-wise intermediate features from multiple U-Net denoisers into a unified one.\n\nThe paper's main contributions are:\n1. The proposal of an ensembling-based AFA method that dynamically adjusts the contributions of multiple models at the feature level.\n2. The design of the SABW feature aggregator, which can produce attention maps according to various states to adaptively aggregate the block-level intermediate features from multiple U-Net denoisers.\n3. The demonstration, through both quantitative and qualitative experiments, that the proposed AFA outperforms the base models and the baseline methods in terms of superior image quality and context alignment.", "questions": "Has any research optimizing the SABW module to reduce its computational overhead been done? potentially through model pruning, quantization, or more efficient network architectures.\n\nOther ensemble strategies can be compared, such as model distillation or more sophisticated attention mechanisms.\n\nCould the authors extend their evaluations to diverse datasets to testify its generality?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel Adaptive Feature Aggregation (AFA) method for ensembling multiple diffusion models in text-guided image generation. \n\n1. The paper introduces a new approach to ensembling diffusion models by dynamically adjusting the contributions of multiple models at the feature level based on various states such as prompts, initial noises, denoising steps, and spatial locations. This is a departure from existing methods that primarily adopt parameter merging strategies to produce a new static model.\n\n2. The proposed AFA method is evaluated quantitatively and qualitatively, demonstrating improvements in image quality and context alignment. The method outperforms base models and baseline methods.\n\n3. The paper is well-structured, with clear explanations of the proposed method, including the Spatial-Aware Block-Wise (SABW) feature aggregator. The authors provide a detailed description of the AFA framework, its components, and the training and inference processes.\n\n4. The paper addresses an important research direction in leveraging multiple high-quality diffusion models to improve text-to-image generation. The proposed AFA method has the potential to significantly enhance the generation capabilities of diffusion models, making it a valuable contribution to the field.", "weaknesses": "1. AFA's computational efficiency is lower for individual inference steps compared to merging-based methods, primarily due to the additional parameters introduced by the Spatial-Aware Block-Wise (SABW) feature aggregator. However, the paper notes that AFA's tolerance for fewer inference steps can offset this initial inefficiency, making the overall computational cost comparable to that of base models or merging methods.\n\n2. The performance of AFA is contingent on the quality of the base models. If the base models have inherent limitations in their generative capabilities, increasing the number of training samples for AFA does not necessarily lead to improved performance.\n\n3. While the paper proposes a dynamic feature-level ensembling method, it does not extensively explore alternative ensemble strategies that could potentially offer further improvements.\n\n4. The paper does not address the scalability of the AFA method when ensembling a larger number of models, which could become computationally prohibitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N1b3wAKQc5", "reviewer": "Reviewer_x8ta", "summary": "This work introduces a novel method called Adaptive Feature Aggregation (AFA) to enhance the performance of text-guided diffusion models by dynamically adjusting the contributions of multiple models at the feature level. Different from the existing static model which utilizes merging strategies, AFA can recognize the performance of diffusion models depending on different states including text prompts, initial noises, denoising steps, and spatial locations. The method employs a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator, which produces attention maps to realize adaptively aggregation of block-level intermediate features from multiple U-Net denoisers.", "questions": "1. How does the method handle the situation that base models have correlated features or that base model bias is generated by imbalanced data? \n2. How is the method’s performance when facing particularly noisy input data?\n3. What modifications should be considered when applying AFA to integrate models with different architectures?\n4. Are there any potential optimization strategies to reduce the extra computational complexity brought by AFA to every single inference?\n5. What if the contributions of different models have conflict impacts on the result significantly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. the method can adjust the contributions of multiple diffusion models based on various states.\n2. the generated attention maps can give a straightforward view of the context and timestamps in the ensembled diffusion model.\n3. AFA demonstrates its tolerance to reductions in inference steps.", "weaknesses": "1. the method integrates intermediate features from U-Net denoisers with the same architecture, posing doubts on its ability to ensemble a wider range of model types or architectures.\n2. As the number of base models increases, the method's reliance on multiple base models may result in overfitting if, say, the base models have correlated features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7MEAw0MWLl", "reviewer": "Reviewer_TNJb", "summary": "The paper investigates an ensemble of multiple diffusion models with identical architectures. Rather than simply blending the network parameters of multiple diffusion models through weighted averaging, it introduces a trainable \"glue\" module, referred to as the AFA method, to integrate them into a cohesive system. This module dynamically adjusts each model's contribution at the feature level, adapting to various conditions such as prompts, initial noise, denoising steps, and spatial locations. Specifically, the paper proposes a U-Net-specific \"glue\" module, called SABW. Through extensive experiments and ablation studies, the effectiveness of SABW in integrating three diffusion models with U-Net architectures is thoroughly demonstrated.", "questions": "1. Could the authors provide further explanation as to why you chose to randomly split the six diffusion models into two groups of three, rather than integrating all six models directly? Additionally, could you provide more insights into the rationale behind selecting three diffusion models for integration?\n2. During the training process of AFA, it appears that intermediate inference features of all base diffusion models at different timesteps are required. Wouldn’t this process be quite time-consuming? Could the authors provide more information on the training time efficiency of AFA compared to other baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and well-structured, with clear and informative figures and tables that effectively support the content.\n2. The motivation is interesting and novel: using a trainable \"glue\" module to integrate multiple pre-trained diffusion model experts at the feature level, aiming to improve generation quality. This approach is instructive and easy to follow.\n3. The experimental design is comprehensive and thorough. The paper conducts extensive experiments on two datasets, COCO 2017 and Draw Bench Prompts, utilizing four evaluation metrics for each. These experiments provide robust evidence of SABW's effectiveness in integrating three diffusion models with U-Net architectures, demonstrating the model's enhanced performance across diverse scenarios.", "weaknesses": "1. This paper aims to effectively integrate multiple diffusion models to achieve improved generation quality. However, based on the framework and experimental setup of SABW, it appears that SABW is only suitable for integrating diffusion models with identical U-Net architectures, which may limit its flexibility and applicability.\n2. According to the training algorithm presented, SABW can theoretically be used to combine any N diffusion models. However, the experiments seem to only consider the integration of three diffusion models, without sufficient discussion or explanation of this choice. A study on the impact of the number of integrated diffusion models would better validate the effectiveness of SABW.\n3. During inference, AFA essentially combines all base models with the glue module SABW, which results in relatively low inference efficiency. The authors provided results for AFA with a reduced number of inference steps compared to the base models. However, comparing AFA's results to those of the baselines under the same reduced-step inference would better validate AFA's practical time efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0cKu9BxZ1", "reviewer": "Reviewer_DhvC", "summary": "This study focuses on dynamically ensembling multiple diffusion models with the same architecture for stronger general generation capability, previously done mostly by static merging. The authors propose to train a plug-in module to generate spatial attention maps for each U-Net block to aggregate the intermediate features. Experiments demonstrate consistent performance improvement.", "questions": "1.\tMore recent diffusion models, such as Stable Diffusion XL and DiT-based models, have different architectures from Stable Diffusion v1.5. For example, if we view DiT models in the same way as used in this study, the entire model will be a single block. SDXL models also have significantly larger inner structures within each block. How may the proposed method generalize to these new models?\n2.\tAs far as I know, some diffusion models also fine-tune the text encoder alongside U-Net. How may the proposed method apply to such scenarios?\n3.\tIn line 346, it is stated that all the methods generate 4 images. This means 4 images per what? Only 4 images per dataset will cause serious insufficiency in evaluation, so the authors should clarify if this is another presentation issue.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe studied task is very useful in practice. The technical contribution of this study should be acknowledged.\n2.\tThe proposed method is intuitive and reasonable, with inner workings and results that match the motivation.\n3.\tExperiments show consistent performance improvement.", "weaknesses": "1.\tThe presentation really needs polishing. For example, “various states” are repeatedly mentioned in both line 80 and line 85. There is a missing reference in line 352. Some of the fonts in Figure 2 are also too small. The following two weaknesses are also related to presentation.\n2.\tSABW implements spatial attention using the composition of several modules, instead of a variant of typical multi-head self-attention in diffusion models. This makes the presentation quite confusing for readers who are familiar with diffusion model architecture. I think the authors should clearly remind readers of this difference, as well as explain why such a special architecture is necessary in comparison to some variant of self-attention.\n3.\tWhile this study aims to enhance the general generation capability, it is also a common goal of ensembling/merging to combine the styles/concepts of multiple models, which is not the interest of this study. The authors should help readers notice this difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SndRDnJD0j", "reviewer": "Reviewer_RkPx", "summary": "The paper introduces Adaptive Feature Aggregation (AFA), a new feature layer ensemble method for generating high-quality images from text. In particular, the paper proposes a Spatial-Aware Block-Wise (SABW) aggregator, which seeks to aggregate intermediate features of various pre-trained models dynamically at the block level. Furthermore, the paper includes a wide range of quantitative & qualitative validations showing strong empirical evidence of AFA's improvements in a range of image generation tasks.", "questions": "Where there any cases or prompts where the authors find that AFA underperforms expectations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n- The paper is well-written and provides a clear outline of the underlying motivation for AFA.\n- The paper provides a timely analysis of an underexplored field of feature layer ensemble methods for diffusion models.\n- SABW component addresses a novel spatial attention variation in denoising capabilities across states.\n- The resulting AFA shows strong empirical results across a wide range of tasks.", "weaknesses": "Some weaknesses of the paper are as follows:\n- From the reviewer's perspective, the AFA method, especially the SABW component, introduces a large degree of additional complexity.\n- The review would also like to see some empirical comparisons with other methods when evaluating the computational efficiency of AFA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kGIvWkUiB6", "reviewer": "Reviewer_Ynbz", "summary": "This paper introduces a novel method called Adaptive Feature Aggregation (AFA) for ensembling multiple diffusion models to improve the quality of text-guided image generation. By adding a spatial-aware block-wise feature aggregator in every unet block of two aggregated models, this work ensembles multiple diffusion models into a united one.", "questions": "1. The contribution you mentioned in this paper is the sampling efficiency, but in the limitation phase you’re aware that “A significant limitation of our AFA is that a single inference step may necessitate a proportional increase in inference time, resulting in lower computational efficiency for that step”. As for comparison experiment Figure 4, I don’t think it fair and meaningful to only compare inference steps without detailed time consumption.\n\n2. Some of the results are not reproducible. For instance, in Figure 5(b), the paper claims ”in Fig. 5 (b), all the base models and the baseline methods generate images containing more than one clock, despite the textual prompt specifying a single clock”, but my tests with SD 1.5 produced correct results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is novel to aggregate multiple diffusion models with a light weighted structure, showing an evident improvement on inference quality especially in small inference steps. The structure of Spatial-aware block-wise feature aggregator is well-designed, which has some enlightening reference value on latent map feature learning.", "weaknesses": "1. Considering the model version, the paper lacks comparisons with newer models like SD 3.0 or Flux. Considering the model architecture, the paper doesn’t use diffusion with different frameworks like DiT, but only ensembles multiple SDv1.5 with the same unet structure, which is not convincing. Besides, the application of this work is limited. The method fails when considering ensembling different diffusion model architectures.\n\n2. The experimental details are incomplete, with no information on the hardware used, detailed memory consumption or quantitative analysis on time consumption.\n\n3. The experiment is not rigorous enough. Aggregators are trained on JourneyDB which is a high-quality dataset, but baselines are not finetuned on the same dataset. The comparison between baselines and your ensembled model is not fair. Besides, in the paper it’s claimed that SDv1.5 failed to understand the number in the prompt, but this phenomenon is hard to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)