Ensembling Diffusion Models via Adaptive Feature Aggregation

## Abstract

Abstract The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source community.
These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities.
Leveraging multiple high-quality models to produce stronger generation ability is valuable, but has not been extensively studied.
Existing methods primarily adopt parameter merging strategies to produce a new static model.
However, they overlook the fact that the divergent denoising capabilities of the models may dynamically change across different states, such as when experiencing different prompts, initial noises, denoising steps, and spatial locations.
In this paper, we propose a novel ensembling method, Adaptive Feature Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple diffusion models, while suppressing their disadvantages.
Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into a unified one.
The core idea lies in dynamically producing an individual attention map for each model’s features by comprehensively considering various states.
It is worth noting that only SABW is trainable with about 50 million parameters, while other models are frozen.
Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed Adaptive Feature Aggregation method.
The code is available at https://github.com/tenvence/afa/ .

###### Keywords:

## 1 Introduction

Diffusion models [, ] have progressively become the mainstream models for text-guided image generation [, , , , , , ], which treats generation as an iterative denoising task.
Recently, the open-source stable diffusion (SD) [] model has prompted the development and release of numerous powerful diffusion models within the open-source community (*e.g*., CivitAI(^1^11[https://civitai.com/](https://civitai.com/))).
These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities.
Leveraging multiple high-quality models to dig out better generations is an important research direction, which has not been extensively studied.

Figure: Figure 1: Examples to illustrate the dynamical change of the denoising capabilities across various states. We conduct experiments on different prompts with various initial noises. We then plot the proportion of wins (*i.e*., the model with the least error between the predicted noise and the initial noise), for each model in a certain spatial region.
Refer to caption: x1.png

Existing methods leverage multiple diffusion models through the weighted merging of model parameters, which can be called the static method.
The weights are usually manually set [, ] or automatically searched through enumeration [].
In contrast, model ensembling, which can be called the dynamic method, often uses dynamic strategies to fuse multiple models at the feature level. Unlike ensembling for classification models [] that usually work after the output, ensembling for diffusion models typically needs to work for each block.
However, the denoising capabilities of models vary not only at different blocks but also at different spatial locations.
As illustrated in Fig. 1,
we conduct denoising experiments on different prompts with various initial noises. Then, we plot the proportion of wins (*i.e*., the model with the least error between the predicted noise and the initial noise), for each model in a certain spatial region, to distinguish the denoising capabilities of the models at different states.
In other words, different prompts, initial noises, denoising steps, and spatial locations can all have a significant impact on the denoising capabilities of diffusion models.
This implies that an adaptive method is needed to ensure that each diffusion model can dominate the generation at its strongest states.

In this paper, we propose a novel Adaptive Feature Aggregation (AFA) method, which dynamically adjusts the contributions of multiple models at the feature level by taking into account various states, such as prompts, initial noises, denoising steps, and spatial locations.
Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptively aggregates the block-level intermediate features from multiple U-Net denoisers into a unified one.
The core idea of adaptive aggregation lies in dynamically producing an individual attention map for each model’s features by comprehensively taking into account the various states such as text prompts, denoising steps, and spatial locations.
A noteworthy aspect of AFA is that only SABW is trainable with about 50 million parameters, while all other models are frozen.

Our main contributions are summarized as follows:

- •
We propose an ensembling-based AFA method to dynamically adjust the contributions of multiple models at the feature level.
- •
We design the SABW feature aggregator that can produce attention maps according to various states to adaptively aggregate the block-level intermediate features from multiple U-Net denoisers.
- •
We conduct both quantitative and qualitative experiments, demonstrating that our AFA outperforms the base models and the baseline methods in both superior quality and context alignment.

## 2 Related Work

#### 2.0.1 Text-Guided Image Synthesis.

Early works for text-guided image synthesis leverage Generative Adversarial Networks (GAN) [] conditioned on text [, , , ].
Based on the success of Transformers [], many subsequent works reframe text-guided image synthesis as a sequence-to-sequence task [, , , , ].
Recently, diffusion models [, ] gradually become mainstream, which treats image generation as an iterative denoising task.
By injecting text as a condition into the denoising process, many diffusion-based models achieve significant success in text-guided image synthesis [, , , , , , ].
Among them, the Latent Diffusion Model (LDM) [] performs the diffusion and reverse process in the latent space, instead of the pixel space, which largely reduces the computational burden.
Among all the implementations of the Latent Diffusion Model, the stable diffusion model (SD) is the most famous one.
Along with its public availability, the open-source community achieves tremendous success, with the emergence of many excellent fine-tuned models.
These high-quality models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities.
In this paper, we aim to leverage multiple such models to achieve stronger text-to-image generation.

#### 2.0.2 Model Merging.

For vision or language understanding tasks, the effectiveness of parameter merging [, , , , , , ] can be interpreted from the perspectives of loss landscape [, , ] and task arithmetic [, , ].
However, merging-based methods have not been widely studied for generation tasks, especially diffusion-based generation.
Recently, many intuitive merging-based methods have emerged.
One of the popular methods is Weighted Merging [], which manually determines weights to merge each U-Net [] parameter of multiple SD models.
Although the simplicity, Weighted Merging coarsely allocates weight to all U-Net blocks.
As an improvement, Merge Block Weighted (MBW) [] allows for manually setting different merging weights for the parameters of distinct U-Net blocks, which provides a more fine-grained merging strategy.
Furthermore, to reduce the reliance on manual weights, autoMBW [] attempts to automate the merging process, which enumeratively selects the optimal combination of MBW weights by an aesthetic classifier.
However, autoMBW is constrained by the performance bottleneck of the aesthetic classifier, and the enumerative selection leads to a huge time consumption to find the optimal settings.
Such merging methods can be called the static methods.
In this paper, we aim to leverage multiple diffusion models by the dynamic methods, *i.e*., ensembling.

#### 2.0.3 Model Ensembling.

Model ensembling is an effective method to achieve better performance [], which has been widely applied in various vision understanding tasks, such as classification [, , ], regression [, ], clustering [].
While fewer works focus on the ensembling of generative models, because of the complexity of image space.
Vision-Aided GAN [] focuses on GANs [], which guides the optimization of a target generator by ensembling pretrained models as a loss.
MagicFusion [] focuses on the diffusion models, which fuses the predicted noises of two expert U-Net denoisers to implement specific applications, such as style transferring and object binding.
In this paper, we aim to efficiently ensemble multiple diffusion models to achieve generation improvements.

## 3 Method

### 3.1 Preliminary

As a type of generative model, the diffusion model [, ] consists of two processes, which are the diffusion process and the reverse process, respectively.
In the diffusion process (*i.e*., the forward process), the Gaussian noises are iteratively added to degrade the images over $T$ steps until the images become completely random noise.
In the reverse process, a trained denoiser is used to iteratively generate images from the sampled Gaussian noise.

When training, given an input image $\mathbf{x}_{0}$ and the additional condition $\mathbf{c}$ (*e.g*., encoded textual prompts *etc*.), the denoising loss is defined as

$$ $\mathcal{L}_{\text{denoise}}=\mathbb{E}_{\mathbf{x}_{0},\boldsymbol{\epsilon} \sim\mathcal{N}(\mathbf{0},\mathbf{I}),\mathbf{c},t}\left\|\boldsymbol{ \epsilon}_{\theta}\left(\mathbf{x}_{t},\mathbf{c},t\right)-\boldsymbol{ \epsilon}\right\|^{2}\ .$ (1) $$

Among them, $\mathbf{x}_{t}=\sqrt{\alpha_{t}}\mathbf{x}_{0}+\sqrt{1-\alpha_{t}}\boldsymbol{\epsilon}$ is the noisy image at timestep $t\in[1,T]$, where $\alpha_{t}$ is a predefined scalar from the noise scheduler.
$\boldsymbol{\epsilon}$ is the added noise.
$\boldsymbol{\epsilon}_{\theta}$ is the denoiser with the learnable parameters $\theta$, which predicts the noise to be removed from the noisy image.

LDM stands out as one of the most popular diffusion models. It performs the diffusion and reverse process in the latent space which is encoded by a Variational Auto-Encoder (VAE) [, ].
In LDM, the U-Net structure [] is used for denoiser, which contains three parts of blocks, which are the down-sampling blocks, the middle block, and the up-sampling blocks, respectively.
Each down-sampling block is skip-connected with a corresponding up-sampling block.
These blocks are composed of Transformer [] layers and ResNet [] layers.
In the Transformer layers, the cross-attention mechanism is employed to incorporate conditional textual prompts, which plays a crucial role in text-guided image generation.

Figure: Figure 2: Framework of ensembling multiple diffusion models by our Adaptive Feature Aggregation (AFA) method.
Refer to caption: x2.png

### 3.2 Adaptive Feature Aggregation

Our proposed Adaptive Feature Aggregation (AFA) ensembles multiple diffusion models that share the same architecture but different parameters.
Specifically, SABW is designed to integrate intermediate features from multiple U-Net denoisers at the block level.
Fig. 2 demonstrates an example of ensembling two diffusion models, and each U-Net denoiser contains seven blocks by SABW.
The forward process of AFA can be seen in Algorithm 1.

Given $N$ diffusion models to be ensembled, whose U-Net denoisers contain $K$ blocks,
let $\boldsymbol{\epsilon}_{\theta_{i}}$ be the $i$-th U-Net denoiser, and $\boldsymbol{\epsilon}_{\theta_{i}}^{(j)}$ be its $j$-th block.
For the timestep $t$, it transforms the input features $\mathbf{x}_{t}^{(j)}$ into the output feature by

$$ $\mathbf{y}_{t,i}^{(j)}=\boldsymbol{\epsilon}_{\theta_{i}}^{(j)}(\mathbf{x}_{t} ^{(j)},\mathbf{c},t)\ .$ (2) $$

Note that because of the same architecture of all $\boldsymbol{\epsilon}_{\theta_{i}}$, $\mathbf{y}_{t,i}^{(j)}$ has the same shape, *i.e*., $\mathbf{y}_{t,i}^{(j)}\in\mathbb{R}^{h_{j}\times w_{j}\times c_{j}}$, where $h_{j}$, $w_{j}$, and $c_{j}$ are the height, width, and channels, respectively.
To achieve the output feature for the next block, a block $f_{\varphi}^{(j)}$ of SABW is employed to aggregate $\{\mathbf{y}_{t,i}^{(j)}\}_{i=1}^{N}$,

$$ $\mathbf{y}_{t}^{(j)}=f_{\varphi}^{(j)}(\{\mathbf{y}_{t,i}^{(j)}\}_{i=1}^{N}, \mathbf{c},t)\in\mathbb{R}^{h_{j}\times w_{j}\times c_{j}}\ ,$ (3) $$

where $\varphi$ is the parameters of the SABW block.
As a whole, the $j$-th ensembled block of these U-Net denoisers can be formulated as

$$ $\mathbf{y}_{t}^{(j)}=\mathcal{F}_{\Theta}^{(j)}(\mathbf{x}_{t}^{(j)},\mathbf{c },t)\ ,$ (4) $$

where $\Theta$ is the parameter collection of $\varphi$ and all the $\theta_{i}$.
It can be deduced that an ensembled block operates with the same functionality as a single block.

Figure: Algorithm 1 One forward process of Adaptive Feature Aggregation (AFA).

### 3.3 Spatial-Aware Block-Wise Feature Aggregator

The Spatial-Aware Block-Wise (SABW) feature aggregator is learned to aggregate output features of each block from multiple U-Net denoisers.
We design SABW to learn spatial attention to reweight the contributions of these U-Net denoisers at each spatial position.
Specifically, the spatial attention map $\mathbf{A}_{t}^{(j)}$ for $N$ U-Net denoisers is achieved by

$$ $\displaystyle\mathbf{F}_{t}^{(j)}$ $\displaystyle=g_{\varphi}^{(j)}(\{\mathbf{y}_{t,i}^{(j)}\}_{i=1}^{N},\mathbf{c },t)\in\mathbb{R}^{h_{j}\times w_{j}\times N}\ ,$ (5) $\displaystyle\mathbf{A}_{t}^{(j)}$ $\displaystyle=\text{softmax}(\mathbf{F}_{t}^{(j)})\in[0,1]^{h_{j}\times w_{j} \times N}\ ,$ (6) $$

where, $\mathbf{F}_{t}^{(j)}$ is the output of the learnable network of SABW $g_{\varphi}$.
Note that all the parameters of SABW (*i.e*., ${\varphi}$) are used to learn spatial attention.
The ensembled output feature is achieved by

$$ $\mathbf{y}_{t}^{(j)}=\mathcal{F}_{\Theta}^{(j)}(\mathbf{x}_{t}^{(j)},\mathbf{c },t)=\sum_{i=1}^{N}\mathbf{A}_{t,i}^{(j)}\otimes\mathbf{y}_{t,i}^{(j)}\ ,$ (7) $$

where $\otimes$ is the element-wise multiplication.
$\mathbf{A}_{t,i}^{(j)}\in\mathbb{R}^{h_{j}\times w_{j}\times 1}$ is the spatial attention map for the $i$-th U-Net denoiser.

This spatial attention operation is based on an important experimental insight as shown in Fig. 1.
The divergent denoising capabilities of multiple diffusion models will dynamically change in different spatial locations.
Thus, better generation performance can be achieved by enabling multiple models to collaborate at different spatial locations.

Specifically, based on the denoising loss (*i.e*., Eq. 1), the denoising capability(^2^22Compared with the denoising loss in Eq. 1, the denoising capability is defined by adding the negative sign, because of a negative correlation between the denoising capability and the denoising loss.) of $\boldsymbol{\epsilon}_{\theta}$ for $(\mathbf{x}_{0},\mathbf{c})$ at timestep $t$ can be defined as $-\mathbb{E}_{\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\|
\hat{\boldsymbol{\epsilon}}_{t}-\boldsymbol{\epsilon}\|^{2}$, where $\hat{\boldsymbol{\epsilon}}_{t}=\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},
\mathbf{c},t)$.
For the spatial location $(x,y)$, the denoising capability is formulated as $-\mathbb{E}_{\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\|
\hat{\boldsymbol{\epsilon}}_{(x,y)}-\boldsymbol{\epsilon}_{(x,y)}\|^{2}$, which considers the noise prediction errors at the spatial location $(x,y)$.

For example, as shown in Fig. 1, we employ three SD models to present their positional denoising capability on the specific image and textual prompt by Monte-Carlo sampled noises.
We demonstrate the results of four patches which indicate that the divergent denoising capabilities of the three models vary at different spatial locations.
SABW is learned to incorporate these denoisers to achieve stronger denoising capability at all spatial locations at the block level.

To implement $g_{\varphi}$ in Eq. 5, SABW applies a ResNet [] layer to introduce timesteps by adding the time embedding into the feature map, and a Transformer [] layer to introduce textual prompts by the cross-attention,

$$ $\displaystyle\mathbf{Y}_{t}^{(j)}$ $\displaystyle\leftarrow\text{concat}(\{\mathbf{y}_{t,i}^{(j)}\}_{i=1}^{N})\in \mathbb{R}^{h_{j}\times w_{j}\times(N\cdot c_{j})}\ ,$ (8) $\displaystyle\mathbf{H}_{t}^{(j)}$ $\displaystyle\leftarrow\text{ResLayer}(\mathbf{Y}_{t}^{(j)}+\gamma(t))\in \mathbb{R}^{h_{j}\times w_{j}\times d}\ ,$ (9) $\displaystyle\mathbf{O}_{t}^{(j)}$ $\displaystyle\leftarrow\text{TransformerLayer}(\mathbf{H}_{t}^{(j)},\mathbf{c} )\in\mathbb{R}^{h_{j}\times w_{j}\times d}\ ,$ (10) $\displaystyle\mathbf{F}_{t}^{(j)}$ $\displaystyle\leftarrow\text{ZeroInitConv}(\mathbf{O}_{t}^{(j)})\in\mathbb{R}^ {h_{j}\times w_{j}\times N}\ ,$ (11) $$

where $\gamma$ is used to map the timestep into the time embedding with the same shape of $\mathbf{Y}_{t}^{(j)}$.
$d$ is the hidden state dimension. $\text{ZeroInitConv}(\cdot)$ is the convolution layer initialized by zero, which leads to equal attention weights for all denoisers before training. We aspire for the model to commence training from a completely equitable state.

### 3.4 Training

Given the ensembled denoiser with AFA $\mathcal{F}_{\Theta}$, the training object is the same as the denoising loss in Eq. 1,

$$ $\mathcal{L}=\mathbb{E}_{\mathbf{x}_{0},\boldsymbol{\epsilon}\sim\mathcal{N}( \mathbf{0},\mathbf{I}),\mathbf{c},t}\left\|\mathcal{F}_{\Theta}\left(\mathbf{x }_{t},\mathbf{c},t\right)-\boldsymbol{\epsilon}\right\|^{2}\ .$ (12) $$

Note that the parameters $\Theta$ contains both the parameters of SABW $\varphi$ and the parameters of all U-Net denoisers $\{\theta_{i}\}_{i=1}^{N}$, where $\varphi$ are learnable, and $\{\theta_{i}\}_{i=1}^{N}$ are frozen.

A well-trained AFA ensembles multiple U-Net denoisers to enhance the denoising capability in every block when experiencing different prompts, initial noises, denoising steps, and spatial locations.
It leads to smaller denoising errors and stronger generation performance.

**Table 1: Quantitative comparison between our AFA with the three base models in Group I (*i.e*., ER, MMR, and RV) and the baseline methods.**
|  | COCO 2017 | Draw Bench Prompts |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FID $\downarrow$ | IS | CLIP-I | CLIP-T | AES | PS | HPSv2 | IR |
| Base Model A | 13.01 | 5.65 | .6724 | .2609 | 5.4102 | 21.6279 | 27.8007 | .3544 |
| Base Model B | 13.45 | 5.43 | .6775 | .2652 | 5.5013 | 21.4624 | 27.7246 | .2835 |
| Base Model C | 12.32 | 6.32 | .6890 | .2566 | 5.4881 | 21.8031 | 27.9652 | .3922 |
| Weighted Merging [] | 10.65 | 6.93 | .6861 | .2626 | 5.4815 | 21.7272 | 27.9086 | .3909 |
| MBW [] | 11.03 | 6.51 | .6870 | .2624 | 5.4812 | 21.7201 | 27.9080 | .3922 |
| autoMBW [] | 13.35 | 5.51 | .6772 | .2577 | 5.5056 | 21.4785 | 27.8192 | .3672 |
| MagicFusion [] | 10.53 | 6.85 | .6751 | .2620 | 5.3431 | 21.3840 | 27.8105 | .3317 |
| AFA (Ours) | 9.76 | 7.14 | .6926 | .2675 | 5.5201 | 21.8263 | 27.9734 | .4388 |

### 3.5 Inference

The inference of AFA starts from the sampled Gaussian noise.
Then, the diffusion scheduler (*e.g*., DDIM [], PNDM [], DPM-Solver [, ],
*etc*.) is applied to generate images with multiple denoising steps.

For each inference step, the noise prediction relies on the technique of Classifier-Free Guidance (CFG) [], which is formulated as

$$ $\hat{\boldsymbol{\epsilon}}_{t}^{\text{pred}}=\hat{\boldsymbol{\epsilon}}_{t}^ {\text{uc}}+\beta_{\text{CFG}}(\hat{\boldsymbol{\epsilon}}_{t}^{\text{c}}-\hat {\boldsymbol{\epsilon}}_{t}^{\text{uc}})\ .$ (13) $$

Among them, $\hat{\boldsymbol{\epsilon}}_{t}^{\text{pred}}$, $\hat{\boldsymbol{\epsilon}}_{t}^{\text{uc}}$, $\hat{\boldsymbol{\epsilon}}_{t}^{\text{c}}$, and $\beta_{\text{CFG}}$ are the final predicted noise, the predicted noise with condition, the predicted noise without condition, and a guidance scale, respectively.
The latent for the next step is

$$ $\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\mathbf{x}_{t}-\frac{\sqrt{1- \alpha_{t}}}{\sqrt{\alpha_{t}}}\hat{\boldsymbol{\epsilon}}_{t}^{\text{pred}}\ .$ (14) $$

Finally, the generated image is achieved from the latent by a VAE decoder.

## 4 Experiments

### 4.1 Experiment Settings

#### 4.1.1 Base Models.

We select six popular models with the same architecture from SDv1.5 [] in CivitAI.
There are twelve down-sampling blocks, one middle block, and twelve up-sampling blocks for each model.
These models are randomly divided into two groups, with three models in each group.
The models from the same group will be ensembled.

- •
Group I contains EpicRealism (ER) [], MajicMixRealistic (MMR) [], and RealisticVision (RV) [];
- •
Group II contains AbsoluteReality (AR) [], CyberRealistic (CR) [], and RealCartoonReslistic (RCR) [].

**Table 2: Quantitative comparison between our AFA with the three base models in Group II (*i.e*., AR, CR, and RCR) and the baseline methods.**
|  | COCO 2017 | Draw Bench Prompts |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FID $\downarrow$ | IS | CLIP-I | CLIP-T | AES | PS | HPSv2 | IR |
| Base Model A | 12.12 | 5.66 | .6849 | .2623 | 5.5641 | 21.8013 | 28.0183 | .4238 |
| Base Model B | 12.41 | 5.59 | .6818 | .2580 | 5.5027 | 21.7249 | 28.0343 | .4202 |
| Base Model C | 12.05 | 5.95 | .6638 | .2642 | 5.5712 | 21.4936 | 27.8089 | .3367 |
| Weighted Merging [] | 11.53 | 6.56 | .6824 | .2631 | 5.5756 | 21.7516 | 28.0014 | .4387 |
| MBW [] | 12.06 | 6.42 | .6826 | .2632 | 5.5772 | 21.7487 | 28.0029 | .4396 |
| autoMBW [] | 12.39 | 5.62 | .6774 | .2588 | 5.5478 | 21.5135 | 27.9873 | .3513 |
| MagicFusion [] | 11.63 | 7.13 | .6790 | .2640 | 5.4674 | 21.4270 | 27.9608 | .4194 |
| AFA (Ours) | 10.27 | 7.42 | .6855 | .2717 | 5.5798 | 21.8059 | 28.0371 | .4892 |

#### 4.1.2 Training.

AdamW [] is used as the optimizer with a learning rate of 0.0001 and a weight decay of 0.01.
Note that only the parameters of SABW are optimized, while those of U-Net denoisers are frozen.
Our AFA framework is trained on 10,000 samples from the dataset JourneyDB [] for 10 epochs with batch size 8.
To enable CFG, we use a probability of 0.1 to drop textual prompts.

#### 4.1.3 Evaluation Protocols.

For a fair comparison, all the methods generate 4 images by DDIM [] for 50 inference steps.
The CFG weight $\beta_{\text{CFG}}$ in Eq. 13 is set to 7.5.
We evaluate the models with two datasets, which are COCO 2017 [] and Draw Bench Prompts [], respectively.

- •
COCO 2017 comprises 118,287 and 5,000 image-caption pairs in the test and validation sets.
All the models generate images with a resolution of 256$\times$256.
We apply four metrics to evaluate the generation performance, which are Fréchet Inception Distance (FID), Inception Score (IS), CLIP-I, and CLIP-T, respectively.
FID and IS are applied to the test set, while CLIP-I and CLIP-T are applied to the validation set.
Both FID and IS are metrics to assess the quality of generated images.
Note that lower FID means better quality.
CLIP-I is the similarity between the CLIP [] images embeddings of generated images and that of images from the image-caption pairs.
CLIP-T is the CLIPScore [] between the generated images with the captions.
- •
Draw Bench Prompts contains 200 evaluation prompts.
All the models generate images with a resolution of 512$\times$512.
We apply 4 evaluation metrics, which are AES [], Pick Score (PS) [], HPSv2 [], and Image Reward (IR) [], respectively.
All three metrics evaluate the performance by a model that simulates human preferences.
The evaluations are conducted 20 times to ensure statistical significance.

#### 4.1.4 Baselines.

We compare our AFA with several methods, including three merging-based methods (*i.e*., Weighted Merging [], MBW [], and autoMBW []) and one ensembling-based method (*i.e*., MagicFusion []).
These baselines are also evaluated on two evaluation protocols under two model groups.

Figure: Figure 3: Quantitative comparison between AFA with the two base models.
Refer to caption: x3.png

### 4.2 Quantitative Comparison

The results of the two evaluation protocols are summarized in Tab. 1 and Tab. 2.
Generally, compared with base models, performance improvements are achieved by our AFA.
For COCO 2017, after being ensembled by AFA, FID, IS, CLIP-I, and CLIP-T exhibit enhancements under both Group I and Group II.
This indicates that the quality of our generated images is more closely with the dataset images.
Additionally, the contexts of our generated images align more closely with those of the dataset images and corresponding captions.
For Draw Bench Prompts, all four metrics show improvements under both Group I and Group II.
The improvement on AES suggests that the aesthetics of our generated images surpass those generated by the base models.
The improvements on PS, HPSv2, and IR suggest that the general generation capability can be boosted by applying AFA.
Overall, the quantitative results validate the effectiveness of our AFA in ensembling text-to-image diffusion models.

While MagicFusion attains impressive performance in style-transferring and object-binding by ensembling two U-Net denoisers, it fails to achieve superior general generation when ensembling multiple U-Net denoisers.
The merging-based methods, such as Weighted Merging, MBW, and autoMBW, do not always produce models with superior generation capabilities.
For instance, under Group II, Weighted Merging achieves a better IR compared to the base models, but under Group I, it yields a lower IR.
This could be attributed to the fixed contributions of merged models, which are determined by a set of predefined merging weights.
Consequently, these methods fail to effectively facilitate collaboration between models to compensate for each other’s weaknesses and improve their generative capabilities.
Compared with these baselines, our AFA consistently achieves better performance, demonstrating its superiority.

As depicted in Fig. 4, we use AFA to ensemble two models from Group I or Group II, and present the IR scores of base models and the ensembled models on Draw Bench Prompts.
The results show that AFA significantly improves the performance when ensembling two U-Net denoisers.
The results presented in Tab. 1, Tab. 2, and Fig. 4 indicate that AFA can be effectively applied to various model combinations and can be extended to accommodate different numbers of models.
Simultaneously, it consistently achieves superior generation performance compared to a single base model.

Figure: Figure 5: Qualitative comparison between AFA with the base models and the baselines.
Refer to caption: x5.png

### 4.3 Qualitative Comparison

As shown in Fig. 5, we present a qualitative comparison of the results from the base models, some of the baseline methods, and our AFA, which allows us to make several insightful observations.
Firstly, AFA can generate images with better aesthetics.
For example, in Fig. 5 (a) and (c), AFA can generate images with improved composition and finer details, compared to both the base models and the baseline methods.
Secondly, AFA excels in achieving superior context alignment.
For example, in Fig. 5 (b), all the base models and the baseline methods generate images containing more than one clock, despite the textual prompt specifying a single clock.
In contrast, only our AFA generates the image with just one clock, accurately reflecting the provided context.
Thirdly, AFA can focus on the correct context of the base models and drop out the incorrect context.
For example, in Fig. 5 (d), only the base model B generates the image with two dogs, while the baseline methods do not fully trust the base model B, and generate the images with people.
Only our AFA fully trusts the base model B, and generates the image with two dogs, which aligns with the textual context.

Figure: Figure 6: Visualization of the learned attention maps. In each attention map, the position with a lighter color means a higher attention weight. Conversely, the position with a darker color means a lower attention weight.
Refer to caption: x6.png

### 4.4 Model Study

#### 4.4.1 Visualization of Attention Maps.

As shown in Fig. 6, we select to visualize the attention maps of the first and the penultimate (*i.e*., 24-th) blocks, both of which are learned by ensembling models in Group II through our AFA.
For the first block, the learned attention maps seem to concentrate solely on the feature map of AR at the first inference stage.
However, at the last few inference steps (*i.e*., from the 30th to the 50th step), the roles of the feature maps from different models begin to diverge.
The attention maps tend to highlight the contextual body (*i.e*., the astronaut and the horse in Fig. 6 (a), and the pizza and the oven in Fig. 6 (b)) in the feature map of RCR, while focusing on the background elements in the feature maps of AR and CR.
For the penultimate block, as the inference progresses, the learned attention maps gradually emphasize the contextual body within the feature map of AR, while concentrating on the background within the feature maps of both CR and RCR.
Overall, the visualization of the learned attention maps indicates that our AFA can effectively ensemble diffusion models based on the context and the timesteps.

#### 4.4.2 Effect of Fewer Inference Steps.

As shown in Fig. 4 and Fig. 7, we investigate the resilience of our AFA to a reduction in the inference steps.
Fig. 4 clearly shows a significant drop in the performance of the base models when the number of inference steps is reduced.
However, the performance of our AFA remains almost stable when the number of inference steps is reduced from 50 to 20, and only experiences a slight decline when the number of inference steps is reduced to 10.
Additionally, in Fig. 7, we can find that the quality of the images generated by the base models deteriorates with the reduction of inference steps.
In contrast, our AFA stably generates high-quality images.
Overall, it indicates that greater tolerance for fewer inference steps will be achieved after ensembling by AFA.
Furthermore, while a single inference step may require a proportional increase in time, the total time consumed by the entire inference process does not escalate in the same manner.

Figure: Figure 7: Qualitative comparison under different inference steps.
Refer to caption: x7.png

#### 4.4.3 Ablation Study.

As shown in Tab. 3, we conduct the ablation study to investigate the necessity of each component within our AFA.
Firstly, we conduct experiments with the following settings.
(1) Only Ensembling Last Block: The AFA ensembles only the last block, leaving all other blocks unaltered.
(2) Block-Wise Averaging: Each block is ensembled by averaging the output feature maps, rather than using SABW.
(3) Noise Averaging: Only the last block is ensembled by averaging the output feature maps (*i.e*., predicted noises).
The observed performance declines in these settings suggest that the design of the AFA effectively contributes to the ensembling of diffusion models, thereby leading to improved generation.
Next, we conduct an ablation study on the spatial location from AFA, which learn only a single attention score (*i.e*., a scalar) for the feature of each U-Net denoiser, as opposed to an attention map.
The observed decrease in performance suggests that learning a spatial attention map to adaptively adjust the weights of various U-Net denoisers in the level of spatial location is beneficial for the ensemble of different diffusion models.
Finally, we individually ablate the timestep and the textual condition which are introduced to AFA.
The decreasing performance indicates that both the introduced timestep and textual condition play crucial roles in ensembling diffusion models by AFA.

#### 4.4.4 Discussion on Computational Efficiency.

(1) Ensembling Methods. Compared with another ensembling method (*i.e*., MagicFusion), the computational efficiency of our AFA for a single inference step aligns closely with it.
Both AFA and MagicFusion necessitate running all base models in one inference step.
Although SABW in AFA introduces additional parameters, which leads to more computations, it has fewer than the base models (about 1/16).
To be precise, our AFA may be marginally less efficient than MagicFusion on single inference steps, but the difference is practically negligible.
(2) Merging Methods. Because the various models are collapsed into one model, the computational efficiency of the merging methods equals that of a single base model.
For the base models and the merging methods, a single image generation takes about 7 seconds with 50 inference steps.
When ensembling three such base models, a single generation will take about 22 seconds with the same inference steps.
Compared with ensembling, the computation efficiency of one inference step of the merging methods is significantly higher.
However, as we discussed in Effect of Fewer Inference Steps of Sec. 4.4, AFA shows remarkable tolerance for fewer inference steps.
When limited to 20 inference steps, the performance of AFA does not significantly decline (as shown in Fig. 4), while the time consumed is reduced to about 8.8 seconds.
This suggests that AFA has comparable computational efficiency with that of the base models or the merging methods throughout the entire inference process.

## 5 Conclusion

**Table 3: Ablation study of AFA.**
|  | IR (Group I) | IR (Group II) |
| --- | --- | --- |
| AFA (Full Model) | .4388 | .4892 |
| Only Ensembling Last Block | .4176 | .4374 |
| Block-Wise Averaging | .4001 | .4372 |
| Noise Averaging | .3919 | .4355 |
| w/o Spatial Location | .4044 | .4610 |
| w/o Timestep | .4235 | .4622 |
| w/o Textual Condition | .4163 | .4559 |

In this paper, we aim to ensemble multiple diffusion models to achieve better generation performance.
To this end, we propose the Adaptive Feature Aggregation (AFA) method, which dynamically adjusts the contributions of multiple models at the feature level according to different prompts, initial noises, denoising steps, and spatial locations.
Specifically, we design a lightweight Spatial-Aware Block-Wise feature aggregator that produces attention maps to adaptively aggregate the block-level intermediate features from multiple U-Net denoisers.
Both the quantitative and qualitative experiments demonstrate that AFA achieves improvements in image quality and context alignment.

#### 5.0.1 Limitation.

A significant limitation of our AFA is that a single inference step may necessitate a proportional increase in inference time, resulting in lower computational efficiency for that step.
Fortunately, due to its high tolerance for fewer inference steps, the total time consumed by the entire inference process does not escalate in the same manner.
This results in a computational efficiency that is on par with that of the base models or the merging methods.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Given that recent work in diffusion models increasingly focuses on efficiency, have you explored knowledge distillation or other compression techniques to reduce the inference overhead of AFA?
- How does the spatial attention mechanism in SABW interact with different types of prompts? Are there certain prompt patterns where timestep-specific expert selection might be more appropriate than spatial aggregation?
soundness: 3
strengths: - Novel perspective on model ensembling that goes beyond both conventional parameter merging and timestep-specific expert selection strategies
- Comprehensive consideration of multiple states (prompts, noise levels, timesteps, spatial locations) for adaptive feature aggregation, offering more fine-grained control than existing expert-based approaches
- Well-designed empirical validation with thorough ablation studies and clear visualization of spatial attention patterns
- Practical advantages in terms of implementation, requiring only the training of SABW while keeping base models frozen
- Demonstrates robust performance with fewer inference steps, suggesting potential for real-world applications
Strong quantitative improvements across multiple metrics (FID, IS, CLIP scores) compared to both single models and existing ensemble methods
summary: This paper introduces Adaptive Feature Aggregation (AFA), a novel ensembling method for text-to-image diffusion models. Unlike conventional static parameter merging approaches, AFA dynamically aggregates features from multiple diffusion models using a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator. The method adaptively adjusts model contributions based on various states including prompts, initial noise, denoising steps, and spatial locations. The key technical innovations lie in block-wise feature aggregation and spatial attention mechanisms, enabling effective ensemble that maximizes strengths while minimizing weaknesses of individual models.
weaknesses: - While the paper introduces a novel method, it lacks direct comparison with recent mixture-of-expert diffusion approaches such as ERNIE-ViLG 2.0 [a], eDiff-I [b], and MEME [c], which also address the dynamic nature of the denoising process with multiple models. Basically, I think that the reader should be informed about the ideas that can be thought of in the direction of improving DPMs using multiple models, such as using different expert models along the time axis, and how this study proposes a different concept from those studies. Specifically, comparing performance metrics (e.g., FID, IS, CLIP scores), computational efficiency (inference times, resource utilization), and qualitative differences in generated images would provide a clearer understanding of how AFA stands relative to these methods. Including such comparisons or ablation studies would strengthen the claim that spatial-aware aggregation offers advantages over or complements timestep-specific expert selection. In the process of improving DPMs by utilizing multiple models, I believe that research directions along two axes, spatial and temporal, should be compared.
- Missing theoretical analysis on when and why spatial-aware feature aggregation outperforms timestep-specific expert selection. Incorporating theoretical insights—such as analyzing the information flow, gradient propagation, or the capacity of spatial attention mechanisms within the diffusion process—could help elucidate the conditions under which AFA is most effective. A comparative theoretical framework would enhance the understanding of the benefits and limitations of spatial versus temporal adaptation strategies.
- Computational overhead during inference could be problematic for practical applications, yet alternatives or optimizations are not thoroughly explored. I think the limitations associated with these computational costs should be clearly explained and guidelines for practical application should be provided.
- Experiments limited to SDv1.5-based models, raising questions about generalizability across different model architectures
- Lack of analysis on the trade-off between the granularity of adaptation (spatial/temporal) and computational cost

---
[a] Feng, Zhida et al. “ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 10135-10145.

[b] Balaji, Yogesh et al. “eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers.” ArXiv abs/2211.01324 (2022): n. pag.

[c] Lee, Y., J. Kim, H. Go, M. Jeong, S. Oh, and S. Choi. “Multi-Architecture Multi-Expert Diffusion Models”. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, Mar. 2024, pp. 13427-36, doi:10.1609/aaai.v38i12.29245.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Has any research optimizing the SABW module to reduce its computational overhead been done? potentially through model pruning, quantization, or more efficient network architectures.

Other ensemble strategies can be compared, such as model distillation or more sophisticated attention mechanisms.

Could the authors extend their evaluations to diverse datasets to testify its generality?
soundness: 2
strengths: The paper presents a novel Adaptive Feature Aggregation (AFA) method for ensembling multiple diffusion models in text-guided image generation. 

1. The paper introduces a new approach to ensembling diffusion models by dynamically adjusting the contributions of multiple models at the feature level based on various states such as prompts, initial noises, denoising steps, and spatial locations. This is a departure from existing methods that primarily adopt parameter merging strategies to produce a new static model.

2. The proposed AFA method is evaluated quantitatively and qualitatively, demonstrating improvements in image quality and context alignment. The method outperforms base models and baseline methods.

3. The paper is well-structured, with clear explanations of the proposed method, including the Spatial-Aware Block-Wise (SABW) feature aggregator. The authors provide a detailed description of the AFA framework, its components, and the training and inference processes.

4. The paper addresses an important research direction in leveraging multiple high-quality diffusion models to improve text-to-image generation. The proposed AFA method has the potential to significantly enhance the generation capabilities of diffusion models, making it a valuable contribution to the field.
summary: This paper introduces a novel ensembling method called Adaptive Feature Aggregation (AFA) for leveraging multiple high-quality diffusion models to enhance text-to-image generation capabilities. The key innovation of AFA is its ability to dynamically adjust the contributions of multiple models at the feature level based on various states, such as prompts, initial noises, denoising steps, and spatial locations. This is achieved through a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptively aggregates block-wise intermediate features from multiple U-Net denoisers into a unified one.

The paper's main contributions are:
1. The proposal of an ensembling-based AFA method that dynamically adjusts the contributions of multiple models at the feature level.
2. The design of the SABW feature aggregator, which can produce attention maps according to various states to adaptively aggregate the block-level intermediate features from multiple U-Net denoisers.
3. The demonstration, through both quantitative and qualitative experiments, that the proposed AFA outperforms the base models and the baseline methods in terms of superior image quality and context alignment.
weaknesses: 1. AFA's computational efficiency is lower for individual inference steps compared to merging-based methods, primarily due to the additional parameters introduced by the Spatial-Aware Block-Wise (SABW) feature aggregator. However, the paper notes that AFA's tolerance for fewer inference steps can offset this initial inefficiency, making the overall computational cost comparable to that of base models or merging methods.

2. The performance of AFA is contingent on the quality of the base models. If the base models have inherent limitations in their generative capabilities, increasing the number of training samples for AFA does not necessarily lead to improved performance.

3. While the paper proposes a dynamic feature-level ensembling method, it does not extensively explore alternative ensemble strategies that could potentially offer further improvements.

4. The paper does not address the scalability of the AFA method when ensembling a larger number of models, which could become computationally prohibitive.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How does the method handle the situation that base models have correlated features or that base model bias is generated by imbalanced data? 
2. How is the method’s performance when facing particularly noisy input data?
3. What modifications should be considered when applying AFA to integrate models with different architectures?
4. Are there any potential optimization strategies to reduce the extra computational complexity brought by AFA to every single inference?
5. What if the contributions of different models have conflict impacts on the result significantly?
soundness: 3
strengths: 1. the method can adjust the contributions of multiple diffusion models based on various states.
2. the generated attention maps can give a straightforward view of the context and timestamps in the ensembled diffusion model.
3. AFA demonstrates its tolerance to reductions in inference steps.
summary: This work introduces a novel method called Adaptive Feature Aggregation (AFA) to enhance the performance of text-guided diffusion models by dynamically adjusting the contributions of multiple models at the feature level. Different from the existing static model which utilizes merging strategies, AFA can recognize the performance of diffusion models depending on different states including text prompts, initial noises, denoising steps, and spatial locations. The method employs a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator, which produces attention maps to realize adaptively aggregation of block-level intermediate features from multiple U-Net denoisers.
weaknesses: 1. the method integrates intermediate features from U-Net denoisers with the same architecture, posing doubts on its ability to ensemble a wider range of model types or architectures.
2. As the number of base models increases, the method's reliance on multiple base models may result in overfitting if, say, the base models have correlated features.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could the authors provide further explanation as to why you chose to randomly split the six diffusion models into two groups of three, rather than integrating all six models directly? Additionally, could you provide more insights into the rationale behind selecting three diffusion models for integration?
2. During the training process of AFA, it appears that intermediate inference features of all base diffusion models at different timesteps are required. Wouldn’t this process be quite time-consuming? Could the authors provide more information on the training time efficiency of AFA compared to other baselines?
soundness: 2
strengths: 1. The paper is well-written and well-structured, with clear and informative figures and tables that effectively support the content.
2. The motivation is interesting and novel: using a trainable "glue" module to integrate multiple pre-trained diffusion model experts at the feature level, aiming to improve generation quality. This approach is instructive and easy to follow.
3. The experimental design is comprehensive and thorough. The paper conducts extensive experiments on two datasets, COCO 2017 and Draw Bench Prompts, utilizing four evaluation metrics for each. These experiments provide robust evidence of SABW's effectiveness in integrating three diffusion models with U-Net architectures, demonstrating the model's enhanced performance across diverse scenarios.
summary: The paper investigates an ensemble of multiple diffusion models with identical architectures. Rather than simply blending the network parameters of multiple diffusion models through weighted averaging, it introduces a trainable "glue" module, referred to as the AFA method, to integrate them into a cohesive system. This module dynamically adjusts each model's contribution at the feature level, adapting to various conditions such as prompts, initial noise, denoising steps, and spatial locations. Specifically, the paper proposes a U-Net-specific "glue" module, called SABW. Through extensive experiments and ablation studies, the effectiveness of SABW in integrating three diffusion models with U-Net architectures is thoroughly demonstrated.
weaknesses: 1. This paper aims to effectively integrate multiple diffusion models to achieve improved generation quality. However, based on the framework and experimental setup of SABW, it appears that SABW is only suitable for integrating diffusion models with identical U-Net architectures, which may limit its flexibility and applicability.
2. According to the training algorithm presented, SABW can theoretically be used to combine any N diffusion models. However, the experiments seem to only consider the integration of three diffusion models, without sufficient discussion or explanation of this choice. A study on the impact of the number of integrated diffusion models would better validate the effectiveness of SABW.
3. During inference, AFA essentially combines all base models with the glue module SABW, which results in relatively low inference efficiency. The authors provided results for AFA with a reduced number of inference steps compared to the base models. However, comparing AFA's results to those of the baselines under the same reduced-step inference would better validate AFA's practical time efficiency.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	More recent diffusion models, such as Stable Diffusion XL and DiT-based models, have different architectures from Stable Diffusion v1.5. For example, if we view DiT models in the same way as used in this study, the entire model will be a single block. SDXL models also have significantly larger inner structures within each block. How may the proposed method generalize to these new models?
2.	As far as I know, some diffusion models also fine-tune the text encoder alongside U-Net. How may the proposed method apply to such scenarios?
3.	In line 346, it is stated that all the methods generate 4 images. This means 4 images per what? Only 4 images per dataset will cause serious insufficiency in evaluation, so the authors should clarify if this is another presentation issue.
soundness: 3
strengths: 1.	The studied task is very useful in practice. The technical contribution of this study should be acknowledged.
2.	The proposed method is intuitive and reasonable, with inner workings and results that match the motivation.
3.	Experiments show consistent performance improvement.
summary: This study focuses on dynamically ensembling multiple diffusion models with the same architecture for stronger general generation capability, previously done mostly by static merging. The authors propose to train a plug-in module to generate spatial attention maps for each U-Net block to aggregate the intermediate features. Experiments demonstrate consistent performance improvement.
weaknesses: 1.	The presentation really needs polishing. For example, “various states” are repeatedly mentioned in both line 80 and line 85. There is a missing reference in line 352. Some of the fonts in Figure 2 are also too small. The following two weaknesses are also related to presentation.
2.	SABW implements spatial attention using the composition of several modules, instead of a variant of typical multi-head self-attention in diffusion models. This makes the presentation quite confusing for readers who are familiar with diffusion model architecture. I think the authors should clearly remind readers of this difference, as well as explain why such a special architecture is necessary in comparison to some variant of self-attention.
3.	While this study aims to enhance the general generation capability, it is also a common goal of ensembling/merging to combine the styles/concepts of multiple models, which is not the interest of this study. The authors should help readers notice this difference.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Where there any cases or prompts where the authors find that AFA underperforms expectations?
soundness: 4
strengths: The strengths of the paper are as follows:
- The paper is well-written and provides a clear outline of the underlying motivation for AFA.
- The paper provides a timely analysis of an underexplored field of feature layer ensemble methods for diffusion models.
- SABW component addresses a novel spatial attention variation in denoising capabilities across states.
- The resulting AFA shows strong empirical results across a wide range of tasks.
summary: The paper introduces Adaptive Feature Aggregation (AFA), a new feature layer ensemble method for generating high-quality images from text. In particular, the paper proposes a Spatial-Aware Block-Wise (SABW) aggregator, which seeks to aggregate intermediate features of various pre-trained models dynamically at the block level. Furthermore, the paper includes a wide range of quantitative & qualitative validations showing strong empirical evidence of AFA's improvements in a range of image generation tasks.
weaknesses: Some weaknesses of the paper are as follows:
- From the reviewer's perspective, the AFA method, especially the SABW component, introduces a large degree of additional complexity.
- The review would also like to see some empirical comparisons with other methods when evaluating the computational efficiency of AFA.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The contribution you mentioned in this paper is the sampling efficiency, but in the limitation phase you’re aware that “A significant limitation of our AFA is that a single inference step may necessitate a proportional increase in inference time, resulting in lower computational efficiency for that step”. As for comparison experiment Figure 4, I don’t think it fair and meaningful to only compare inference steps without detailed time consumption.

2. Some of the results are not reproducible. For instance, in Figure 5(b), the paper claims ”in Fig. 5 (b), all the base models and the baseline methods generate images containing more than one clock, despite the textual prompt specifying a single clock”, but my tests with SD 1.5 produced correct results.
soundness: 3
strengths: This work is novel to aggregate multiple diffusion models with a light weighted structure, showing an evident improvement on inference quality especially in small inference steps. The structure of Spatial-aware block-wise feature aggregator is well-designed, which has some enlightening reference value on latent map feature learning.
summary: This paper introduces a novel method called Adaptive Feature Aggregation (AFA) for ensembling multiple diffusion models to improve the quality of text-guided image generation. By adding a spatial-aware block-wise feature aggregator in every unet block of two aggregated models, this work ensembles multiple diffusion models into a united one.
weaknesses: 1. Considering the model version, the paper lacks comparisons with newer models like SD 3.0 or Flux. Considering the model architecture, the paper doesn’t use diffusion with different frameworks like DiT, but only ensembles multiple SDv1.5 with the same unet structure, which is not convincing. Besides, the application of this work is limited. The method fails when considering ensembling different diffusion model architectures.

2. The experimental details are incomplete, with no information on the hardware used, detailed memory consumption or quantitative analysis on time consumption.

3. The experiment is not rigorous enough. Aggregators are trained on JourneyDB which is a high-quality dataset, but baselines are not finetuned on the same dataset. The comparison between baselines and your ensembled model is not fair. Besides, in the paper it’s claimed that SDv1.5 failed to understand the number in the prompt, but this phenomenon is hard to reproduce.

Meta-review: This paper explores adaptive feature aggregation strategies for effectively ensembling diffusion models. It initially receives borderline scores on average. The authors did hard work during the rebuttal phase. Most of the reviewers are now positive toward this paper. It is non-trivial to rebuttal with as many as seven reviewers within a short time. Above all, I'm happy to recommend accepting this paper.

Decision: Accept (Poster)