When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models
Authors: Raphaël Razafindralambo
Rémy Sun
Damien Garreau
Frederic Precioso
Pierre-Alexandre Mattei
Venue: Accepted by TMLR ()

Abstract: Diffusion models now generate high-quality, diverse samples, with an increasing focus on
more powerful models. Although ensembling is a well-known way to improve supervised
models, its application to unconditional score-based diffusion models remains largely un-
explored. In this work we investigate whether it provides tangible benefits for generative
modelling. We find that while ensembling the scores generally improves the score-matching
loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as
FID on image datasets. We confirm this observation across a breadth of aggregation rules
using Deep Ensembles, Monte Carlo Dropout, on CIFAR-10 and FFHQ. We attempt to ex-
plain this discrepancy by investigating possible explanations, such as the link between score
estimation and image quality. We also look into tabular data through random forests, and
find that one aggregation strategy outperforms the others. Finally, we provide theoretical
insights into the summing of score models, which shed light not only on ensembling but also
on several model composition techniques (e.g. guidance). Our Python code is available at
https://github.com/rarazafin/score_diffusion_ensemble.

Published in Transactions on Machine Learning Research (02/2026)
When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models
Raphaël Razafndralambo
raphael. www.eeeeaaanddoooooonoorroo...
Université Côte d' A'Auur, Inria, CNRS, 185, Moosoi, Nice, France
Rémy Sun
remay.sun@@inrii....
Université Côte d' d'Azur, Invia, CNRS, 135, Maasai, Nice, France
Frédéric Precioso
frederic. Condvv-oteddeur fr
Université Cóte l'Auur, Inria, CNRS, 185, Moosai, Nice, France
Damien Garreau
daminn.gorreau@uuiivvvrrrror de
Mulbus-Mazimiiaas--- Universittt Würzhbur,
Institute for Computer Science / CAIDAS, Würzburg, Germany
Pierre Alexandre Mattei Université Côte d'A d'Azur, Inria, CNRS, LJAD, Maasai, Nice, France
piere..leaanddn.matttiiiinn.
Reviewed on OpenReview: https: openreview. forum? id= 4d=44R29600A
Abstract
Diffusion models now generate high-quality, diverse samples, with an increasing focus on more powerful models. Although ensembling is a well-known way to improve supervised models, its application to unconditional score-based diffiusion models remains largely un- explored. In this work we investigate whether it provides tangible benefits for generative modelling. We find that while ensembling the scores generally improves the score-matching loss and model likelihood, fails to consistently enhance perceptual quality metrics such as FID image datasets. We confirm this observation across breadth of aggregation rules using Deep Ensembles, Monte Carlo Dropout, CIFAR-10 and FFHQ. We attempt to ex- plain this discrepancy by investigating possible explanations, such as the link between score estimation and image quality. We also look into tabular data through random forests, and find that one aggregation strategy outperforms the others. Finally, we provide theoretical insights into the summing of score models, which shed light. not only on ensembling but also on several model composition techniques (e.g. guidance). Our Python code is available at https://githe: com wow/raaaaaaai//correer ott.aaaaa.aoaeeooooo..oooaooooo cefffusion_ensembee
1 Introduction
Diffusion models have emerged as powerful generative models, demonstrating state-of-the-art performance in applications such as image gencration (Dhariwal & Nichol, 2021), text-to-image synthesis (Saharia al., 2022), video modeling (Ho al., 2022), graph and molecular design (Liu al., 2023), and tabular data generation and imputation (Kim et al., 2023; Palicoeur-Martinee et al,, 2024). These models are inspired by diffusion processes which gradually transform complex data distributions into simple ones, and vice versa.
The key idea behind diffusion models is to define a forward process that progressively adds noise to data over multiple steps (Ho et al., 2020; Song et al., 2021b), reaching a "known" distribution, such as Gaussian noise. A reverse process, typically parameterized by a U-Net (Ronneberger et al., 2015), then trained denoise the data step by step, reconstructing the original data distribution. Sampling is performed by applying this denoising process iteratively given model trained to match score function, starting from the noise.

---

Published in Transactions on Machine Learning Research (02/2026)
Weakest model (M4)
28.5 (1)
Distribution of FID-10k 28.7
24.3
21.722.1
23.5
Strongest model (MI)
21.7 (
M1 M2 M3 M4 Ens.
Average FID = 24.2
Individual models FiDs Ensemble
Ensemble
23.5 (L) x
Figure 1: Visual comparison of samples and FID-10k (7) from two individual models from an ensemble of size K = 4 trained on FFHQ-256, the ensemble using the arithmetic mean and DDIM (Section 2.2.2), and the distribution of FID-10k on the ensemble. The initial noise seed is fixed. Ensembling does not clearly improve results: quantitatively, ensembling does not beat the best model. See Section 5.1 for the evolution of two image quality metries with respect to K.
While diffusion models have demonstrated impressive performance in generating high-quality data, the usual strategy to improve their capabilities is either to train deeper neural networks or to extend the training time (Dhariwal & Nichol, 2021; Song et al., 2021b; Rombach et al., 2022). However, this scaling strategy comes with increased computational costs and may not be accessible to all practitioners.
Ensembling (Dietterich, 2000), which combines multiple models' outputs to improve performance, offers an overlooked opportunity to improve diffusion without relying on extensive training of one single model. The idea behind ensembles is that each model contributes different errors and can correct one another's mistakes, These techniques, widely used in statistics and supervised lcarning, have demonstrated strong empirical results. Ensembling has shown promise for diffusson models as illustrated by the use of Random Forests (an ensemble-based strategy) for unconditional sampling of tabular data (Jolicoeur- -arttnnnn et al., 2024). Forms of ensembling are also used for uncertainty estimation of diffusion models, leveraging entropy over the ensemble's predictions (Jazbec et al., 2025; Kon et al., 2024; De Vita & Belagiannis, 2025), and to improve fidelity when largeting specific distributions (Liu et al., 2022; Du et al., 2023).
Interestingly, attempting directly apply ensembling to diffusson models does not improve the samples as straightforwardly as one might expect. This is precisely what we observe in the preliminary example of Figure 1 where the outputs of the neural networks, which are the ensemble members, are aggregated using a simple mean of score models. Our observations show that an ensemble of five models does not achieve the highest image quality even though it beats the average one. Yet in supervised learning, neural network ensembles yield noticeable gains as soon as K = 2 (Lalshmminarayanan et al., 2017) and generally surpass the best individual model (e.g. Hansen & Salamon, 1990, Figure 4; Lee et al., 2015, Table 4).
In this paper we investigate the impact of ensembling on diffusion models across multiple methods. Our con- tribution is to show that ensembling yields only marginal, if any, improvements in sample quality, despite consistent (but small) reduction in score matching loss. Specifically, we contribute as follows.
1. We look into ensembling for diffusion models (Section 3), using both deep and non-deep base models (Section 3.1), and diverse aggregation strategies (Section 3.2). We also provide theoretical insights on its effect on the training objective error and contextualize ensembling within other contexts (Section 4).
2. We empirically evaluate multiple ensembling methods and show that, despite improving the training objective as predicted by our theoretical analysis, they bring no significant gains on key metrics such as FID and KID on CIFAR-10 and FFHQ, or the Wasserstein distance on tabular data (Section 5.2.1).
3. We highlight two settings in which ensembling can be helpful. First, the Mixture of Experts provides small benefits when the models are sufficiently complementary, allowing their diversity to be effectively
2

---

Published in Transactions on Machine Learning Research (02/2026)
exploited. Second, in the case of decision-tree aggregation, method alleviates an underestimation bias issue, leading to improved performance.
4. We investigate factors affecting ensemble performance, including model diversity (Section 5.2.3) and the mismatch between the score matching loss and perceptual quality metrics that could prevent ensembling from showing clear gains (Section 5.3).
2 Preliminary on ensembles and diffusion models
2.1 Ensemble methods
Ensemble methods trace back to the early days of classical machine learning, where they emerged as a powerful strategy to improve model robustnoss and accuracy (Dictterich, 2000; Yang et al., 2023). Rather than relying on a single predictor, ensembles aggregate the outputs of multiple models to reduce variance and avoid overfitting. A notable example is the Random Forest algorithm (Breiman, 2001), which builds on the principle of Bagging (Breiman, 1996) to create a diverse collection of decision trees and combine their predictions. Formally, let X' and y denote the input and output spaces. An ensemble is defined as collection of K >>> predictors, {f(k) X y. 1 < K}, which are combined using aggregation rule, lypically the mean or majority voting. In classification tasks with L classes, the output space y corresponds to the L-dimensional probability simplex. Over time, ensemble mothods have become standard tools in predictive modelling, often outperforming individual models, both in theory (Mattei & Garreau, 2025) and practice (Dietterich, 2000; Grinsztaju al., 2022). Beyond prediction, ensembles can also be used for generative modelling (Riesselman et al., 2018), clustering (Fleury et al., 2025), or statistical inference (Bach, 2008).
As deep learning became a dominant paradigm in machine learning, ensemble methods were naturally adapted to neural networks. A prominent example is the Deep Ensemble (DE), introduced in the 1990s (Hansen & Salamon, 1990) but popularized decades later thanks to its strong empirical results (Lakshmi- narayanan et al., 2017: Stewart et al., 2023). It involves constructing K models, each sharing the same neural network architecture, which we denote ((k) = fo, for 1 k S K with 0₂ the parameters of the k-th model. A key ingredient for its success is the dicersily between the individual models (Fort et al., 2019). This diversity naturally arises from sources such as random initialization and stochastic gradient descent, even when training on the same datasct. To mitigate the cost of training K separate models, several "train one, get K for free" techniques such as Monte Carlo Dropout (MC Dropout, Srivastava et al., 2014; Gal & Ghahramani, 2016) or Snapshot Ensembles (Huang et al., 2017) have been proposed, which are particularly appealing for diffusion models where training multiple networks is computationally expensive.
Regardless of how the ensemble is constructed, test-tine time inference requires combining the predictions of the K individual models into single output. Given a test input X, this is done via an aggregation rule
[(; K. 0) = AGG (2), .......66000 (w))
(1)
A common approach, particularly in regression and probabilistic settings, the arithmetic mean, in part because it is the combination that minimizes the average squared loss (Wood et al., 2023).
2.2 Diffusion models
In score-based diffusion models, sampling involves learning to estimate the score function. Since we wish to ensemble these estimated scores, we first recall their purpose within the formulation of diffusion models of Song et al. (2021b), based on stochastic differential equations (SDEs).
2.2.1 General framework: score-based SDEs
We introduce three central components that will be used throughout this work: the denoising SDE (Equa- tion (3)), its associated numerical approximation for sampling, and the training loss used to learn the score function (Equation (4)). These components will serve as the basis for explaining how ensembles are trained and how individual models can be combined and evaluated at test time.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Diffusion models aim to generate samples that approximate given target distribution. Let gx(x)) denote this target d-dimensional data distribution. Score-based diffiusion models (Song & Ermon, 2020; Song et al., 2021a) simulate a process in which data Xo qo (xx) is progressively corrupted by noise (the forward process) and then denoised to recover the original data distribution (the backward process). The forward process is defined by an SDE with continuous time from time 0 to T of the form
dx = f(x, t)dt + g())dw
(2)
where f: TRd x [0, 7] + Rd and g: [0. - [0, +000 are pre-assigned such that Xy approximates (O I) Wr E Rd is a standard Wiener process, and Xo 9o Under some regularity conditions, defining (x) as the marginal distribution of x, and Wi a time-reversed Wiener process, corresponding reverse-time (backward) SDE with the same marginals can be written (Anderson, 1982; Haussmann & Pardoux, 1986)
dx = [((x () v(1)²vx x log ( g(()dw.
(3)
To draw samples that approximate go, we follow the procedure of Song et al. (2021a): start from Gaussian noise XT and integrate the backward SDE using typically Euler Maruyama updates. This involves discretizing the interval [0, 2 into N steps 0 = to <... < ft = T and applying the solver from ty down to to. Provided the model sufficiently trained and under additional conditions (De Bortoli, 2022), the final sample Xt, is distributed according to density P0,0 that approximates qo- Equation (3) also admits deterministic ODE whose trajectories share the same marginals, allowing the exact computation of the variables formula (Chen et al., 2018).
model likelihood via the instantaneous change-of vaaiiallf-variables
The Stein score Vx log qe (x) is the unknown component of Equation (3). Accordingly, score-based diffusion models train to estimate it at each & by minimizing the Denoising Diffusion Score Matching (DDSM) objective
(DDSM($8) = E.-10.r (x))
(1s (x,, t) - Vx, log (=)((x((x00)]
(4)
where A A(t)= > 0 is a weighting function, and se(x,,t), t), generally a time-dependent neural network, aims to predict Vx log (x) (Song & Ermon, 2020). Implementations of s0(x,, t) include U-Nets (Ronneberger et al., 2015), Vision Transformers (Peebles & Xie, 2023) and Tree-based models (Jolicoeu--Martinen et al., 2024) such as Random Forests (Breiman, 1996) or Gradient Boosted Trees (Friedman et al., 2000).
2.2.2 Unifying perspectives on denoising generative processes
The neural network (or any chosen model) estimation of time-dependent vector field (e.g., Stein score) is the central element of the denoising process for many generative models. While the continuous SDE formulation estimates the score function (Song et al., 2021b) to go to data from noise, other popular generative models such as DDPM (Ho et al., 2020) and DDIM (Song et al., 2020) estimate the noise associated to cach step. Flow Matching (Lipman et al., 2023) estimates a velocity field to transport one distribution to another, and Albergo et al. (2025) unify this perspective with diffusion models in a general interpolation framework. Here, we illustrate on DDPMs/DDIMS the unifying framework by showing that they can be treated interchangeably with basre-based models when reasoning about the score.
Indeed, DDPMs model the forward and backward process Gaussian Markov chains and learn to predict noise with a network t)) t). This prediction can be roparamcterized into score estimate as sa(x, t) = to(x.1), where oe denotes the standard deviation of the marginal distribution at time t. This relation allows de
to unify DDPMs, DDIMs, and score-based models within shared ensemble framework.
3 How to make an Ensemble of Diffusion models?
The aim of this work is to study the application of ensemble methods to diffusion models. While ensembling in classical machine learning typically involves training multiple models and simply averaging their outputs once, this straightforward strategy does not apply to diffusion models. Their generative process involves a complex iterative procodure, making the aggregation of multiple models non-trivial. We structure this section in two parts: we first explain how to obtain К distinct diffusion models (Section 3.1), and then how to combine their outputs into a single one (Section 3.2). We illustrate the general methodology in Figure 2.
4

---

Published in Transactions on Machine Learning Research (02/2026)
Reverse the noising process with combination
SDE SOLVER
(as,) Nx,
@ (xe, 1) 2 vx. log (xi)
Get K soore models.
(xes, 2)
22) (x).
SDE SOLVER
AGG
.x,e
86 (00 1) Xx log (x²)
Figure Overview of ensembling within the Diffusion Models framework. We build K score models, for example using Deep Ensemble where each model optimize in parallel the same loss LDDSM At inference time, start from noise and generate X using SDE solver update rule at each step in which combine the score models using specific combination rule (e.g. arithmetic mean), instead of using one score model.
3.1 Instantiate K score-based predictors
Building on the fact that difflusion models learn a time-dependent score function, we now explore how to build a diverse diffusion model ensemble. We consider both approaches based on neural networks, including independently trained U-Nets, as well as ensemble of decision trees. That said, we will consider DE as the main focus of our empirical study due to its simplicity and widespread in neural network ensembling.
3.1.1 Ensembles of neural networks
We mainly consider the case of neural network ensembles (Hansen & Salamon, 1990). This typically implemented through Deep Ensemble. To construct DE of diffusion models, we independently train K U-Nets { ===1 each initialized with different random seed. Hence each model is trained separately on the full datasct to optimize the DDSM objective LDDSM (Equation (4)).
At this stage, the procedure identical to that of constructing & standard deep ensemble regression (Lakshminarayanan al., 2017). notable point is that the Monte Carlo approximation of the loss exhibits relatively high entropy, due to both the stochastic sampling of timesteps and the injection of Gaussian noise.
Beyond DE, we investigate alternatives techniques to encourage ensemble diversity (discussed in our experi- ments in Section 5). Indeed Xu et al. (2024) showed that diffusion models with different weight initializations (or architectures) tend to converge to similar functions, despite a natural intuition that stochasticity in the loss computation would lead to diverse solutions. This is explained by the smoothness of the loss landscape, and the result similar outputs inference time for the same input noise.
One can use Monte Carlo Dropout (Srivastava al., 2014; Gal & Ghahramani, 2016; Ashukha al., 2020) (MC-Dropout)
lightweight alternative to training K separate U-Nets, which can be computationally prohibitive in terms of training time or memory during inference in our ensemble framework. Indeed MC- Dropout keeps dropout (Srivastava et al., 2014) active at test time on a single model, enabling multiple stochastic forward passes that approximate ensemble behavior without additional training cost.
3.1.2 Ensembles of decision trees
Random Forests which ensemble individual decision trees can be integrated into diffusion model frame- works to enable efficient training and sampling. Notably, that such tree-based models are particularly effective for tabular data generation, outperforming deep learn- ing-based diffusion mothods (Kim et al., 2023; Kotclnikov et al., 2023). The method replaces neural networks
Martenean et al. (2024) demonstrated
5

---

Published in Transactions on Machine Learning Research (02/2026)
with standard ensemble-based estimators, training a distinct Random Forest at each noise level to approx- imate the score function. We complement their study by focusing on new aspect: while they emphasize that XGBoost is the best method to estimate the score (ahead of Random Forest), we analyze the effect of the number of trees forming a Random Forest on performance and study various aggregation schemes.
3.2 How to aggregate the K score predictors?
Building on the previous section Section 3.1, where we introduced how to construct an ensemble of K time-dependent models (..))....... 8g
(K) (..-) each trained to estimate the score function at every timestep, we now turn to the question: how can we effectively aggregate these diffusion models? In this section, we introduce novel aggregation schemes for combining the individual score outputs in R from K neural networks at each time step, which we later evaluate in our experiments.
We focus on aggregating diffusion models within a "democratic" ensemble framework, where each model contributes equally to the aggregation process. Furthermore, we assume that the models are exchangeable, meaning their order does not influence the outcome of the aggregation. Based on these principles, we examine several types of ensembles. Unlike traditional ensemble methods in machine learning, where models generate single prediction for each instance, our framework has to handle outpuls al every timestep of the sampling process. Indeed, each diffusion model corresponds to a complete sampling trajectory. Consequently, we consider ensemble approaches that merge score predictions at cach timestep. In the following, we adopt the formalism of sctre-matching, consequently, focus on the neural network that predicts the score. However each of the methods can be applied within other frameworks (Section 2.2.2), and in particular DDPM.
The Arithmetic mean of scores, used as first example in Section 1, is the most natural way to combine the score models at each time step. Let us denote
(K) (x, 8
(k) 0 (x.
(5)
K
This is the most common approach for aggregating models to form ensembles, and we provide in Section 3.3 a theoretical analysis that discusses an intuitive though non-rigorous interpretation of this combination rule geometric mean of densities. In the same way, we compute the sum of scores Σx= 8g (k) (x, which also admits a meaningful probabilistic interpretation (see Section 3.3) but likely fails by producing out-of- distribution score outputs at each noise level. The Geometric mean and Median are alternative schemes, which apply coordinate-wise computations to K vector field outputs as the previous method.
In contrast averaging schemes, we also consider methods that perform random selection among score models. The Mixture of experts strategy choses unique score model randomly and uniformly prior to sampling, which corresponds to sampling directly from the mixture of densities (see Appendix C.2). In Alternating sampling, we exploit the iterative nature of the process by rather randomly selecting a different expert (xx,, fn) at each noise level fg. Finally, in the Dominant feature approach, we select at each timestep the component of largest absolute value across all experts, defined for cach E (1....,d}
[ (x, :) where [**0*) (x, f)]d = maxıgжsx ([ (x, t))al- This strategy selects contributions from the models that have the most to say at each coordinate, with the goal of producing the strongest possible noise and inducing updates of maximal magnitude.
3.3
Contextualization of score combinations with other uses
There exists a commonly used connection between step-wise score model aggregation and sampling from composition of models, given that scores correspond to gradients of log-densities. Model composition usually refers to combining models that have been trained to address different tasks.
A popular example is classier-free guidance (Ho & Salimans, 2022), where scores of different models are summed to guide samples toward class under a temperature parameter. Specifically, given a class label y, and 7 2 0, the goal is to target the distribution with density p(xly) x p(xl))p(ylx)*** x "pxx)")((( For this distribution, the corresponding score function is straightforward to compute: Vx (x(y)
6

---

Published in Transactions on Machine Learning Research (02/2026)
Vx logg((x) "pxxl) 1+7 (1 7) (ollooggpxx)) >V VVVoogpie p(x). Consequently, to sample from the target distribution, the reverse SDE uses the score function given for all 1> 0 by the approximation
V (x (1 + 7> log V log
(6)
where both score functions are estimated using two diffusion models so(x, t y) and s0 t). Liu et al. (2022) adopts a generalizing framework of classífier-free guidance to combine multiple features. Alexanderson el al. (2023) apply this approach in another context and refers to as a Product-of Product-of-Epperts. In the context of experimental design, Iollo et al. (2025) combine posterior distributions by averaging their scores.
Interestingly, this mirrors our setting: averaging K > 1 score models across all 1 > O corresponds to combining densities, Specifically, given K probability density functions (x)...... (x)) the average score R Ek= Vologp (x) is the score of the normalized geometric mean of the densities, which we define for the remainder of the paper as the Product-offFapeet (PoR):
(K) (x) = (xxXxx)).pp(x((x))))
ZK :
(x))).x))))))) dx.
The same terminology has been used to describe generalized geometric means with arbitrary exponents in the context of Gaussian Processes (Liu et al., 2018; Cohen et al., 2020). Applying the same logic as for guidance, ensembling in the backward process for every > would estimate
K
1
Σ vx Vx log (K) (x) 2 Vx log (KK)
(x);
(8)
K
where > O refers to distributions obtained by noising the data distribution at t = 0. Alternatively, summing the scores corresponds to the unnormalized product II (x),, also referred to as PoE in Hinton (2002).
Some works combine multiple diffusion models in alternative ways. For instance, Balaji et al. (2022); Feng et al. (2023b) combine expert denoisers specialized for different stages of the generative process to better preserve text-conditioning signals. While these approaches do not perform score averaging, they can still be viewed form of ensembling, similarly to our Alternating sampling method.
4 Does averaging scores make sense? A theoretical insight
In this section, we examine whether averaging scores at each step makes sense from theoretical perspective. We state that while it provides simple motivation for ensembling by reducing the LDDSM objective in expectation (Section 4.1), we have to caution against a natural but misleading interpretation as geometric mean of densities (Section 4.2). We show that in fact diffusion and composing densities in this way do not commute, a relevant point beyond ensembling, as similar compositions arise in guidance-basee frameworks.
4.1
+ models are better than K for score estimation
The following proposition shows that, under mild assumptions, an ensemble of models s0 ,...,,,, 8 (K) leads a better expected loss than K models. Such results, mostly based on Jensen's inequality, are classical and well established in machine learning (e.g., regression) for usual convex losses like MSE (McNees, 1992; Breiman, 1996; Mattei & Garreau, 2025). Here we extend them to the score estimation context.
Proposition 4.1 (Monotonicity for the DDSM loss). Let (1) ......g (K) be K score estimators mapping nd [0, to Rdd If for any pair of random variables (x),t) Rdd TJ. the outputs {s (x,,t) are identically distributed (i.d.) conditional on (x,,t), then
(1)
T.DDsM((s [x+ < [[
See Appendix C.1 for a proof. We also discuss the latter the i.d. assumption of this proposition and provide similar but more general result on classical score-matching (Hyvärinen, 2005). We also provide results based on KT. divergence between probability paths, linking ensembling to improvements in likelihood.
7

---

Published in Transactions on Machine Learning Research (02/2026)
4.2 Diffused PoE is not the PoE of diffused distributions
In this section, we show that diffusion and composition do not commute. More precisely, we exhibit a straightforward counter example demonstrating that contrary to natural intuition, adding noise to distribu- tions (c.g., according to a noise scale o₂) and then combining them (e.g., goometric mcan) does not yield the same result as combining them first and then adding noise. Such result has already been stated in prior work for the product of two densities (Du et al., 2023). Chidambaram et al. (2024) separately provided a counterexample showing that the supports of the resulting distributions generally differ, which consequently implies non- conmmmmmutativity. Here, we complement these works with even simpler and more direct one in Section 4.2.1 centered on the non-commulativityy Then, we contextualize the result in Section 4.2.2.
4.2.1 The simple Gaussian counterexample
Let us now formalize this result. We consider the case where the initial distributions are K centered Gaussians with diagonal covariance matrices, i.e., (xx; og.)) for k 1,...; K with on >> 0. Moreover, we adopt simplified setting detailed in Appendix C.4. In brief, we focus a specific instance of the Variance Preserving SDE framework (Song et al., 2021b). To summarize,, our counterexample demonstrates that noise and PoE generally do not commute, except when all initial densities are identical, actually reducing PoE to one model. c(K) We denote the PoE of these distributions by Po
Proposition 4.2. Let Po (k) = NOO, axf) for all ls S K. PoE(pó (1) ......P (K)). Furthermore, for any density go, we write qe the distribution obtained by diffusing it under the forward VP SDE. Then, for any 0, Pt.
((K)
* PoE(pi
(1) .....Pt (()) unless an =1.... ag.
4.2.2 Implications for PoE and Diffusion Guidance
Having established that diffusion and PoE do not generally commute, we now turn to discussing the impli- cations in two distinct contexts: the ensemble framework studied in this paper and the guidance setting.
To begin with, Proposition 4.2 straightforwardly shows that Equation (8) is generally not an equality for any > 0, it directly follows from applying the score function. More importantly, and more generally, it demonstrates that step-wise score averaging (resp. summing) does not correspond to sampling from PoE (resp. normalized product of densities). If it were true, such result would be beneficial for two reasons we detail in Appendix C.2 (simplifies theory, eliminates low-probability regions).
Following this, Equation (6) is also not an equality, and the guidance method proposed by Ho & Salimans (2022) does not produce the target distribution p(xiy), although it approximates it closely. This approach of combining diffusion models should be viewed as heuristic for approximating the desired distribution. Some previous works have pointed out this subtlety, and proposed methods to targel more closely the true distribution using MCMC-based correctors (Du et al., 2023) the Feynman Kac equation (Skreta et al., 2025).
These results also extend to classical inverse problems as discussed in Appendix C.5, in which we target a conditional distribution p(xly) oc <(x))((|xx) given an observed signal y.
5 Experiments
In our experiments, we show that ensembling generally does not lead to consistent improvements in the sample quality of diffusson models. We adopt as our set of practices the study of simple democratic ensembles, that is, ensembles with equal-weight aggregation of independently trained score (or noise) predictors (Section 3.2)
sharing the same architecture. We first explore the straightforward strategy of averaging the score predictions of independently trained models at each timestep and find that it fails to enhance visual quality (Section 5.1).
We then experiment various ensemble strategies, and aggregation mot ranging from simple arithmetic and geometric means to more elaborate strategies, in an attempt to identify which factors, if any, can mitigate the shortcomings (Section 5.2). We follow the common practice of studying both neural network ensembles and decision tree ensembles (Wood et al., 2023; Theisen et al., 2023). Finally, we seek to explain them by
8

---

Published in Transactions on Machine Learning Research (02/2026)
analyzing the relationship between image quality and the objective the models are actually optimized for within the ensemble (Section 5.3).
We train models on tabular data, on CIFAR-10 (32x32), and on FFHQ (256x 256), using different archi- tectures. We measure the FID and KID on 10k samples', which are two perceptual metrics, and LDDSM. For given maximum ensemble size maxx; we report confidence intervals for the variability across model subsets when evaluating FID for smaller ensemble sizes (e.g. for K = we take all the subsets of size in the whole set of size K). For the LODSM loss, intervals computed from 10 runs. In Section 5.3, FID intervals estimated using bootstrap. KID follows the same procedure as FID. Datasets, model choices, training sctups, evaluation methods, and confidence intervals are detailed in Appendix A.
5.1 When ensemble averaging fails: insights from image quality metrics
We show that applying simple ensembling using arithmetic mcan yields either little no improvements in image quality despite reducing the score matching loss. In Figures 3a to 3e, we evaluate this approach by progressively increasing the ensemble size K up to K = 5 or == 10.
26
17
5.00
9.0
2.89
LODsM
1.6
0.5
25
0 1.5
4.90
22.87
Dev
24
X
1.4
4.85
82.86
23
1.3
4.60
7.5
22
1.2
1.0
2.84
4.70
4 Ensemble size
2
Ensemble size K
Ensemble size K
Ensemble size
Ensemble size
(a) FID (FFHQ)
(b) KID (FFHQ)
(c) FID (CIFAR)
(d) KID (CIFAR)
LDDSM (CIFAR)
Figure 3: Evolution of FID (L) and KID (4) and LDDSM (4) in function of ensemble size. Samples of the models in Figures 3a and 3b are displayed in Figure 1. LDDSM is only evaluated SIFAR-10 since it
corresponds to its training objective. See Appendix A for details on each experiment (uncertainty bounds).
This result is surprising since the main heuristic justification of ensembles that groups are collectively wiser than individuals, aggregated decisions tend to exhibit reduced variance. Moreover, because the matching loss, used as the training objective, consistently decreases with larger ensembles, might expect the models to achieve better approximation of the target distribution and produce improved samples.
Here, however, we find that simple ensembbling yields marginal, any, gains over the average model and never surpasses the best individual model in perceptual metrics such as FID and KID. We stress that surprisingly, as shown on CIFAR-10 Appondix (Figure 8), ensembling can oven perform worse than the weakest. individual model.
Figure 10 (Appendix) shows further unsuccessful attempts to use score averaging to compensate for dis- crotization errors with fewer DDIM steps on FFHQ. We also tried applying in Table 7 (Appendix) only during the early sampling sleps to test whether averaging is mainly harmful near the data space, but it did not outperform individual models. All these findings suggest that selecting an ensembling method like arithmetic mean that reduces the training loss is insufficient, and that further progress will require going beyond this aggregation strategy.
5.2 Investigation on ensemble approaches
We explore a range of ensemble variants identify promising directions and better understand simple averaging failures. First, evaluate alternative aggregation strategies (Section 3.2) beyond the arithmetic mean. Next, we evaluate the MC Dropout approach (Section 5.2.2). We also investigate whether encouraging diversity among models can improve performance (Section 5.2.3). Finally we extend our analysis with the non-deep learning approach described in Section 3.1.2 using Forest-VE VP with Random Forest (Section 5.2.1).
The standard is 50k samples. Due the smaller sample size, FID (often referred as FID-10k this context, but simply refer as FID) tends be overestimated, as it biased upward.

---

Published in Transactions on Machine Learning Research (02/2026)
Comparison different aggregation methods on FID, KID, and LDDSM- The best value for each metric is highlighted in bold. We exclude Alternating Sampling and Mixture Experts, as LDDsM reduces a simple average of individual losses. We focus on aggregations where actual loss reduction expected. Standard deviations (in parentheses) are reported when applicable; Appendix A for computation details.
Table 1:
CIFAR-10 (32 32) and K =
FFHQ (256 256) and K 4
Deep Ensemble
FID 4
KID 4
LDDSM +
FID 4
KID 4.
Individual models
4.79 (0.08)
0.0008 (0.0001)
0.0284 (0.0005)
24.20 (5.70)
0.015 (0.003)
Arithmetic mean Geometric mean Dominant
4.98
0.0007
0.0280 (0.0005)
23.44
0.013
4.97
0.0008
0.0280 (0.0005)
24.32
0.013
7.88
0.0041
0.0287 (0.0004)
46.79
0.040
Median
4.98
0.0009
0.0281 (0.0006)
23.13
0.012
Alternating sampling
5.05
0.0008
23.07
0.012
Mixture of experts
4.93
0.0008
20.36
0.011
Best individual model
4.65
0.0006
0.0282 (0.0006)
21.7
0.012
5.2.1 Comparative study of aggregation techniques
We verify whether the behavior observed by averaging scores holds across all the aggregation methods given in Section 3.2 on both datasets and show that the perceptual improvements are at best marginal (sce Table 1).
The four aggregation schemes that combine models at each timestep show no significant differences. On CIFAR-10, these ensembles fail to outperform the average performance of individual models, even though slight improvements in the score matching objective can be observed. On FFHQ, they generally achieve marginal gains over individual models but still do not surpass the best single model. This supports the notion that reducing the training loss alone insufficient to achieve improvements perceptual quality.
Among the alternative methods, Mixture of Experts appears most effective, outperforming even the best model on FFHQ. This suggests that randomly selecting model prior to each sampling increases diversity across generated samples by leveraging inter-model variability. However, since only one model contributes to each generated image, this approach may not qualify as true ensemble in traditional sense. Additional results for aggregation schemes, including the sum, are reported in Appendix D.2 as they did not lead to improvements, and are discussed theoretically in Appendices C.1 and D.
5.2.2 MC Dropout instead of Deep Ensemble
We also evaluate MC Dropout on CIFAR-10 with to K = 20 models and find that it performs no better-and often worse worse-than Deep Ensembles (see Table 5 in appendix). FID degrades from 4.83 (no dropout) to 5.85 at best, which corresponds to Arithmetic mcan, KID doubles, and LDDSM slightly increases by 0.0002. We do not measure on FFHQ since the associated model is not trained with Dropout. The drop in image quality aligns with deterioration of LDDSM, indicating that MC Dropout's Monte Carlo approximation weakens the network and introduces harmful noise in score estimation. As result, it not only fails to improve over DE but can even degrade performance both perceptual metrics and LDDSM-
5.2.3
strategies
In this scction we chock if encouraging prodictive diversity between models load to improvements. Indood ensemble performance is commonly attributed to two factors: (1) the average performance of the individual models, and (2) the diversity in their predictions (Breiman, 1996; 2001; Brown al., 2005). The idea is that if models make different errors, their combination can reduce overall error through averaging. We focus on the Arithmetic mean and Mixture of experts, as the former serves as our baseline ensemble technique and the latter operates at a different stage of sampling and exhibited the most notable improvements in Table 1.
We first examine the common initialization practice in U-Nets that may hinder diversity in diffusion models (see Figure 4). A near-zero scaling factor À on the terminal weights causes the outputs to be near zero (see details in Appendix G.1). In Table 2, we evaluate how this choice influences the performance of DE, by
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Comparison of different aggregation meth- ods for K 5 on CIFAR-10 in terms of FID, KID, and LDDSM- We upscaled initialization variance to one to enhance diversity.
Ensemble performance for K = 5 models trained on overlapping subsets of CIFAR-10 in terms of FID, KID, and LDDSM- Two classes are excluded per model, shifting every two.
Table 3:
CIFAR-10 (32 32)
CIFAR-10 (32 x 32)
À upscaled K =
FID +
KID 4
LDDSM
Subsets K = 5
FID 4
KID +
LDDSM +
Individual model DE
4.79
0.0008
0.0284
Individual model DE
4.79
0.0008
0.0284
(0.08)
(0.0001)
(0.0005)
(0.08)
(0.0001)
(0.0005)
Individual model
5.07
0.0010
0.0283
Individual model
9.65
0.0028
0.0289
(0.08)
(0.0001)
(0.0004)
(0.92)
(0.0006)
(0.0005)
Arithmetic mean
5.21
0.0011
0.0284
Arithmetic mean
7.13
0.0027
0.0284
(0.0005)
(0.0006)
Mixture of experts
5.38
0.0010
Mixture of experts
5.86
0.0008
leveling up the scale À to one. We find that this approach does not resolve the lack improvement observed with ensembles, despite introducing additional diversity in the network's output space (see Appendix G for a broader analysis, including the impact of scaling on training dynamics and evaluation metrics). This suggests that increasing predictive diversity alone is insufficient to enhance ensemble performance in diffusion models.
Secondly we train ensemble members on different subsets of the dataset as a straightforward way to encourage diversity, and we evaluate this strategy in Tables and 6. For instance, on CIFAR- 10, one can assign each model to a specific subset of classes (c.g.: two or three classes per model), ensuring that each member spe- cializes in different part of the data, distribution. Combining models specialized on different parts of dataset is a well-studied area (McAllister et al., 2025; Skreta et al., 2024).
0.032
0.034
Predictive Diversity
DE scale 1e-10
0.027
0.024
DE scale 1
MC-dropoul
0.016
0.012
0.013
0.008
Since each model is specialized in a specific subset of the dataset, one might expect that taking their arithmetic mean for example would merge their respective expertises. However, we find that both aggregation schemes improve image quality but not suffi- ciently to reach the quality of the models of Table 1. The high FID of Arithmetic mean could result from mass concentration on the intersection of supports, reducing coverage by discarding regions captured by only a subset of models. On Mixture of ex- perts, results are poor in comparison to Table since models are more domain-specifie, which could also harm diversity. We an- alyze the scenario with disjoint supports in appendix (Table 6), revealing an even greater degradation.
0.001
0.002
0.000
2
6
8
10
Ensemble size K
Figure 4: Predictive diversity across three ensemble methods. Increasing X slightly enhances diversity. DE scale denotes Deep Ensemble with models trained on initialization scale X. See Ap- pendix G.5 for details on diversity caleu- lation.
Finally we evaluate models trained with varying numbers of iterations (see Figure 5): fewer iterations yield weaker models with more diverse errors which ensembling can possibly compensate effectively. With this in mind, we analyze how ensembling benefits vary with model strength.
We observe that ensembles built from weaker models do yield bigger improvements in FID, while gains diminish and even turn negative as the base models become stronger. Interestingly, training the models longer consistently reduces the loss, highlighting the disconnect between training objective and perceptual quality.
Overall, these experiments show that more diverse ensembles lead to better ensembling effects, especially when base models are weak or specialized. However, this error compensation alone is not. sufficient. to outperform a single baseline model trained on the full dataset.
11

---

Published in Transactions on Machine Learning Research (02/2026)
5.2.4 Random Forests
We evaluate Random Forests as a score predictor and show Figure 6 that.
16-2
3.20
0.2
FIDx FID
Dominant components aggregation
3.12
k=
k=
0.1
min max
yiclds remarkably stronger performance than other aggregation techniques, that generaly perform poorly. We conduct ex- periments on various number of trees on the Iris dataset. To assess these settings (details Appendix A) we evaluate the Wasserstein distance between generated and test data. See Tables and 10 to 13 for more aggregations larger datasets.
3.04
0.0
2.96
0.1
2.88
-0.2
0.8
1.2
2.0
1.2
1.8
Training Steps
le5
Training Steps
le5
(a) LDDSM Training Steps
(b) AFID Training Steps
Figure 5: Comparison of DE (K-2) and individual model (K-1) in terms of training loss and FID, plotted against the number of training iterations. The difference stays positive for IDDSM (a); while for FID (b), for which we directly plot it, the A varies from positive negative. See Appendix A for details on the CI.
We hypothesize that this particular im- provement of Dominant feature strategy compared to others results from a sys- tematic underestimation of the noise magnitude in simple averaging, rather than from ensembling itself.
To check this, for each perturbed data X, E Raxd, we fit Random Forest regressor to predict the noise that transforms the original data point Xu E Rn into X, This noise is in fact systematically underestimated during generation, as shown in Figure 11. Arithmetic mean exhibits a lower overall standard deviation for all & > 0, supporting the hypothesis a statistical bias, namely that fails to capture the true noise magnitude.
0.8
Arithmetic mean
0.6
Dominant
Jolicoeur- al. (2024)
0.4
u5 Number of trees
Figure 6: Wasserstein distance Wuest (L) for Arithmetic mean and Dominant feature methods as function of the number of trees K. We perform 3 train/ test splits for each experience, and report the mean.
5.3 Correlation between Score Matching and FID
As we have observed across previous experiments, the LDDSM function used during training and perceptual metrics remain poorly aligned in ensemble evaluation. As shown in Section 5.1, score matching can benefit from increasing K when perceptual metrics do not. This suggests that ensembling might be effective to some extent, it enhances score estimation which is the very objective the models are trained to optimize.
In this section, we highlight. the gap between these two metrics. We also show that the gap in behavior between score-based metrics and FID is particularly noticeable when noise is introduced into the score function.
5.3.1 Score matching and FID during training
We show in Figure 7a that already over the training course, the behavior of the optimized objective differs from the one of the perceptual metric of interest. While FID keeps improving steadily, the validation score matching loss drops sharply early on and then stagnates. This shows that the score matching loss alone is not a sufficient indicator on the performance of the model in generation. These results are obtained by evaluating both metrics single score model across checkpoints up to 200k iterations.
5.3.2 Noising the score and its effect FID
We show in Figure 7b that perturbing the score in the regime where ensemble improvements typically affect the loss leads to significant. and chaotic variations in FID At each timestep, we apply perturbations to the score model by adding r||se||a2 to its output, where E [0, is fixed noise level, and is uniformly distributed on the unit sphere. These results highlight that FID does not evolve in predictable manner in comparison to the score, showing that intuitive interpretations of FID variations can be misleading.
12

---

Published in Transactions on Machine Learning Research (02/2020)
Valid LODSM (log)
1.20
1e-2
-0.6
DDSM
FID (log)
1.05
9:84
FID
LoD5
3.0
3.1
LODSM
6.2
6.6
FID
0.90
95% CI
-1.2
5.8
2.9
-1.8
0.75
4.85
2.8
5.4
0.0
0.5
1.0
1.5
2.0
0.2
0.6
1.0
1.4
1.8
10-3
10-2
10
10
Training Steps
Training Steps
Noise level
Noise level
(b)
Figure 7: Evolution of score matching loss and FID (in log scale) during training (a), and scnsitivity of FID and LDDSM to noise levels T (in log scale on the figure) in the score function (b). In both cases, trends suggest that the FID evolution cannot be reliably predicted from the score dynamics.
The disconnect between what diffusion models are trained to optimize (score matching) and how they are typically evaluated explains why ensembling struggles to diminish image quality metries. This is further reflected in the decreasing behavior of the likelihood (see Figure 9), which has close mathematical link (up to first or second-order derivatives (Song et al., 2021a; Lu et al., 2022)) with the score estimation. Echoing our observation in this section, Theis et al. (2016) and van den Oord & Dambre (2015) pointed out generally accepted point in the generative model community, that high likelihood does not necessarily translate into better sample quality. Our experiments are another illustration of this point.
6 Conclusion
We investigated ensemble techniques for diffusion models on unconditional gencration, starting with straight- forward aggregation rules then extending to more heuristic or diversity-proomotingg (but still simple) strategies. Averaging Deep Ensemble, the most natural approach, failed to boost perceptual metries such as FID and KID. Motivated by this, we tried alternative aggregation schemes, dropout-based ensembles, subset train- ing, and initialization tweaks; none consistently closed the gap. We also evaluated Random Forests for the tabular data generation framework, and found that using the component with the highest amplitude across the trees improves sample quality based on Wasserstein distance.
The main observation is that ensembling generally yields only marginal gains in FID. The only consistent improvement observed with Deep Ensembles concerns score matching, which largely a mechanical effect resulting from Jensen's incquality and the models' exchangcability. Nevertheless, we observe a few notable exceptions where specific forms of ensembling can yield more perceptible benefits in particular settings: the Mixture of Experts performs better when the models are truly complementary, suggesting that the diversity captured by independently trained models is valuable. However, averaging their scores tends to blur this diversity, thereby degrading perceptual quality. Overall, these observations reinforce the main disconnect: better score matching does not necessarily translate into higher perceptual quality, which explains the limited impact of standard ensembling.
Regarding the benefit-com cost trade-off, our experiments suggest that standard ensembling offers limited re- turns relative to its computational expense. While a couple of approaches can provide more improvements, these gains remain rare, context-dependent. In addition, the substantial increase in training and sampling cost rarely justifies their use in diffusion models. This stands in contrast with supervised learning, where ensembliin is known to provide consistent and significant performance gains (Hansen & Salamon, 1990; Lee et al., 2015) that clearly outweigh its computational overhead.
We showed that the mean of the scores at each step does not sample the PoE. While Feynman Kac (Skreta et al., 2025) or MCMC-based (Du al., 2023) correctors could in principle adjust the sampling process better target the PoE, we deliberately omitted them to restrict the work to baseline ensembling. Morcover, guidance-based approaches already perform well without the additional theoretical rigor introduced by such corrections. Nonetheless, future work could further explore this direction.
13

---

Published in Transactions on Machine Learning Research (02/2026)
We voluntarily restricted our analysis to "democratic" ensembles, where all models contribute equally and independently. Future work aiming to improve diffusion models through ensembling could explore more advanced strategies, such as train one, get K for free or other low-cost ensemble methods (Ramé et al., 2021; Huang et al., 2017; Ha et al., 2017; Havasi et al., 2021; Wasay et al., 2020). Another promising direction would be to relax the equal or independent constraiit, for instance through weighting or boosting.
Acknowledgments
This work was supported by the French government, through the 31A Côte d' Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference numbers ANR-19-P3IA-000 and ANR-23-LA-- -0001, DG also acknowledges the support of ANR through project NIM-ML (ANR-21- CE23-0005-01). Parts of this work were done as DG was employed at Université Côte d' d'Azur.
We thank the reviewers for their constructive feedback, which helped improve the quality and clarity of this paper. We are also grateful to Florence Forbes for carefully pointing out error in earlier version of the paper.
References
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying frame- work for flows and diffusions. Journal of Machine Learning Research, 26:1-80, 2025.
Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! Audio- driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1 20, 2023.
Shun-ichi Amari. Integration of stochastic models by minimizing a-divergence. Neural computation, 19(10): 2780-22 2796, 2007.
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313 126,, 1982.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncer- tainty estimation and ensembling in deep learning. In International Conference on Learning Representa- tions, 2020.
Francis Bach. Bolasso: model consistent lasso estimation through the bootstrap. In International conference on machine learning, pp. 33-40, 2008.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Milk Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. eDiff-I: Text-to-image diffusson models with an ensemble of expert denoisers. arXiv preprint Xiw:2211.0132 2022.
Mikolaj Binkkwskk, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In Internation Conference on Learning Representations, 2018.
Ali Borji. Pros and cons of GAN evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
Leo Breiman. Bagging predictors. Machine learning, 24:123-144 -120,,10,,,11 1996.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Thomas F Brooks, D Stuart Pope, and Michael A Marcolini. Airfoil self-noise and prediction. Technical
Report PASA-RP-1213 1989.
Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity crcation methods: survey and calegorisation. Information fusion, 6(1):5-20 20, 2005.
Matthew A Chan, Maria J Molina, and Christopher A Metzler. Hyper-diffusion: Estimating epistemic and aleatoric uncertainty with single model. Advances in Neural Information Processing Systems, 38, 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. Advances in Neural Information. Processing Systems, 31, 2018.
Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? A fine-grained analysis in simple setting. Advances in Neural Information Processing Systems, 37: 11968-85005, 2024.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference on Leorning Representations, 2023.
Samuel Cohen, Rendani Mbuvha, Tshilidzi Marwala, and Mare Peter Deisenroth. Healing products of Gaussian processes. International Conference on Machine Learning, 2020.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4):5477 553, 2009.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33:19964-19975, 2020.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions
On Machine Learning Research, 2022.
Michele De Vita and Vasilcios Belagiannis. Diffusion model guided sampling with pixel-wisc alcatoric uncer- lainty estimation. In 2025 IEEE/CVF Winter Conference on Applicutions of Compuler Vision (WACV), pp. 3844 3854. IEEE, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 14:8780-8790) 2021.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple
classifier systems, pp. 1-15. Springer, 2000.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models. Advances in Neural Information Processing Signals, 2020.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dicleman, Rob Fergus, Jascha Sohl- Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: compositional generation with energy-based diffusion models and MCMC. In International conference on machine learning, volume 202, pp. 8489 8510. PMLR, 2023.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine I. Bouman, and William T Freeman. Scoro-based diffusion models as principled priors for inverse imaging. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 10520- 10531, 2023a.
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image dilfusion model with knowledge- enhanced -feeuossio---eooiiinngg. experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10135 10145, 2023b.
Maric-Ange Fleury, Louis Ohl, Lionel Tastet, Mickaöl Loclereq. Frédéric Precioso, Piore-Alexandre Mattei, Romain Capoulade, Kathia Abdoun, Élisabeth Bédard, Marie Arsenault, Jonathan Beaudoin, Mathieu Bernier, Erwan Salaun, Jérémy Bernard, Mylène Shen, Sébastien Hecht, Nancy Côté, Arnaud Droit, and Philippe Pibarot. Unsupervised machine learning analysis to enhance risk stratification in patients with asymptomatic aortic stenosis. European Heart Journal- Journal-Digital Health, pp. ztafl15, 2025.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanann Deep ensembles: A loss landscape perspective. arXiv preprint Xix: 1XX::111120 2, 02757, 2019.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Jerome Friedman, Trevor Hastic, and Robert Tibshirani. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). The annals of slatistics, 28(2):337-401 2000.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In International conference machine learning, volume 48, pp- 1050-1059. PML.R, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249- 256. JMLR Workshop and Conference Proceedings, 2010.
Léo Grinszlaju, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35:507-535 520, 2022.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks International Conference on Learning Represen- tations, 2017.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on Pattern Analysis and Machine Intelligence, 12(10):993-1011, 1990.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, 2009.
Ulrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pp. 1188 1205, 1986.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lakshmi- narayanan, Andrew M. Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two timo-scale update rule converge to local nash equilibrium. Advances in Neural Infor- mation Processing Systems, 30, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation, 14($):1711-1800, 2002.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXic:2207.1550 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In Advances in Newral Information Processing Systems, volume 33, pp. 6840-688 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, Willian Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. International Conference on Learning Representations, 2017.
Michael F Hutthinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Platistics-Simulatiion and Computation, 18(3):1059 1076, 1989.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Jacopo lollo, Christophe Heinkelé, Pierre Alliez, and Florence Forbes. Bayesian experimental design via contrastive diffusions. In The Thirteenth International Conference on Learning Representations, 2025.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Metod Jazbec, Eliot Wong-Toi, Guoxuan Xia, Dan Zhang, Eric Nalisnick, and Stephan Mandt. Generative uncertainty in diffusion models. The 4st Conference on Uncerlainly in Artificial Intelligence, 2025.
Alexia Sallcoeur-Marinennn Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-bootted trees. In International Conference Artificial Intelligence and Statistics, pp. 1296. PMLR, 2024.
Tero Karras. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition workshops, 2019.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 8110-8119, 2020.
Jayoung Kim, Chaejeong Lee, and Noseong Park. STaSy: Score-based tabular data synthesis. International Conference on Learning Representations, 2023.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564-17579. PMLR, 2023.
Siqi Kou, Lei Gan, Dequan Wang, Chongxuan Li, and Zhijie Deng, BayesDiff: Estimating pixel-wise uncer- tainty in diffusion via Bayesian inference. International Conference on Learning Representations, 2024.
Alex Krizhevsky. Learning multiple layers of features from Liny images. Master's thesis, University of Toronto, 2009.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of ImageNet classes in Fréchet inception distance. International Conference on Learning Representations, 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in Newral Information Processing Systems, 30:6402- 6413, 2017.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint orXiw:1511.06311 2015.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations, 2023.
Chengyi Liu, Wenqi Fan, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Genera- tive diffusion models on graphs: Methods and applications. International Joint Conference on Artificial Intelligence, 2023.
Haitao Liu, Jianfei Cai, Yi Wang, and Yew Soon Ong. Generalized robust Bayesian committee machine for largo-scale gaussian process regression. In International Conference Machine Learning, pp. 3131-3140. PMLR, 2018.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423 439. Springer, 2022.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion ODEs by high order denoising score matching. In International Conference on Machine Learning, pp. 14429 144600 PMLR, 2022.
Alexandee Mattei and Damien Garreau. Are ensembles getting better all the time? Journal of Machine Learning Research, pp. 1 1-46, 2025.
17

---

Published in Transactions on Machine Learning Research (02/2026)
David McAllister, Matthew Tancik, Jiaming Song, and Angjoo Kanazawa. Decentralized diffusion models. arXiv preprint or (an:2511005450 2025.
Stephen K McNees. The uses and abuses of 'consensus' forecasts. Journal of Forecasting, 11(():703-77) 710, 1992.
Stanislav Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for gan evaluation. In International Conference on Learning Representations, 2021.
Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International conference on machine learning, pp. 7176 7185. PMLR, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 3162-8177) PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Emile Pierret and Bruno Galerne. Diffusion models for Gaussian distributions: Exact solutions and Wasser- stein errors. International Conference on Machine Learning, 2025.
Alexandre Ramé, Rémy Sun, and Matthieu Cord. Mixmo: Mixing multiple imputs for multiple outputs via deep subnetworks. In Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 823 833, 2021.
Adam J Ricsselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(1)):816 822, 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural net- works. In International Conference on Learning Representations, volume 6. International Conference on Representation Learning, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical im- age segmentation. In Medical image computing and computer-asissed intervention-M -MICCAT 2015: 18th international conference proceedings, pp. 234 241. Springer, 2015.
Walter Rudin. Principles of mathematical analysis. McGraw Hill, 1976.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala. Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-imago diffusion models with deep language understanding. 2022.
Advances in Neural Information Processing Systems,
Marla Skreta, Lazar Atanackovic, Joey Bose, Alexander Tong, and Kirill Neklyudov. The superposition of diffusion models using the Itô density estimator. In International Conference on Learning Representations, 2024.
Marta Skreta, Tara Akhound-Sadegh. Viktor Ohanesian, Roberto Bondesan, Alán Aspuru- Arnand Doucet, Rob Brekelmans, Alexander Tong, and Kirill Neklyudov. Feynman-Kac correctors diffusion: Annealing, guidance, and product of experts. International Conference on Machine Learning, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. International Con- ference on Learning Representations, 2020.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Jiaming Song, Arash Vahdat, Morteza. Mardani, and Jan Kautz. Psendoonverse--uudeee diffusion models for inverse problems. In International Conference on Learning Represenlations, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Newral Information Processing Systems, volume 32, pp. 11895-11907, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-basee diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021a.
Yang Song, Jascha / Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modcling through stochastic differential equations. In International Conference Learning Represenlations, 2021b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15 (1):1929 1958,, 2014.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony Caterini, Eric Taylor, and Gabriel Ganeem Exposing flaws of generative model evalu- ation metries and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36:3732 3784, 2023.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression classification: Influence of task formulation on neural network features. In International Conference on Artificial Intel- ligence and Statistics, pp. 11563-11582 PMLR, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2826, 2016.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. International Conference on Learning Representations, 2016.
Ryan Theisen, Hyunsuk Kim, Yaoqing Yang, Liam Hodgkinson, and Michael W Mahoney. When are en- sembles really effective? Advances in Neural Information Processing Systems, 36:15015-15026, 2023.
Aäron van den Oord and Joni Dambre. Locally-connected transformations for deep GMMs. In International Conference on Machine Learning: Deep learning Workshop, pp. 1-8, 2015.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-1674, 2011.
Abdul Wasay, Brian Hentschel, Yuze Liao, Sanyuan Chen, and Stratos Idreos. MotherNets: Rapid deep ensemble learning. Proceedings of Machine Learning and Systems, 2:199-215, 2020.
Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified thoory of diversity in ensemble learning. Journal of Machine Learning Research, 24(359):1-49, 2023.
Lei Xu, Maria Skoularidou, Alfredo Cuesta- Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. Advances Neural Information Processing Systems, 32, 2019.
Tianshuo Xu, Peng Mi, Ruilin Wang, and Yingeong Chen. Towards faster training of diffussin models: An
inspiration of consistency phenomenon. arXie preprint arXiv:240.007946, 2024.
Yongquan Yang, Haijun Lv, and Ning Chen. A survey on ensemble learning under the era of deep learning. Artificial Intelligence Review, 56(6):5545-5588 2023.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normal- ization. International Conference on Learning Representations, 2019.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 3836-383 3847, 2023.
19

---

Published in Transactions on Machine Learning Research (02/2026)
This appendix supplements main contributions with additional insights, methodological clarifications, and extended empirical results:
Appendix A details the experimental settings used throughout the study.
Appendix B outlines how to assess step-wise aggregation strategies beyond sampling, inchuding evaluations of DDDM and model likelihood.
Appendix C elaborates on the theoretical motivations, formulation, and implications of the proposed
aggregation mechanisms. In particular, we prove Proposition 4.1 and Proposition 4.2.
Appendix D presents exploratory yet ineffective alternatives for combining diffussin models.
Appendix E and Appendix F provide additional plots and tables supporting our experimental find- ings in Section 5.
Appendix G examines and contextualizes the popular yet understudied practice of downscaling final-layer weights at initialization in diffusion models.
Appendix H offers short commentary the limitations of FID and the caution required when interpreting it.
A
Experimental details
We describe in detail both our deep and non-deep approaches.
A.1 Ensemble of U-Nets
We train our models on low-resolution images using CIFAR-10 (32x32) (Krizhevsky, 2009). To assess the scalability to higher-resolution data, we also conduct experiments on FFHQ (256x256) (Karras, 2019).
For experiments on CIFAR-10, our neural network follows the DDPM+ continuous architecture as proposed by Song el al. (2021b). We use the VP-SDE formulation discret ized with 1000 diffusion steps, and a predictor- only sampling procedure by adopting the Euler-Maruyama scheme. Training parameters (learning rate, batch size, etc.) and diffusion parameters (Smin:Pmax) are identical to those in the original paper; in particular, we set the dropout rale 0.1. Each model is trained for 200k iterations. For FFHQ-256, we use the ADM architecture from OpenAT's official repository, which includes several improvements over the previous U-Net design (Dhariwal & Nichol, 2021). The model is trained using the DDPM objective. We follow the repository's 's recommendation for the learning rate (1c-4), while the batch size (4) and total number of training iterations (10M) are chosen based on our computational resources. No dropout is used in this setup. At sampling time, we use the DDIM sampler with 100 steps with an entropy level = 0.5 (sce Song et al. (2020) for a definition) by default. Unless herwise specified, these settings are consistently applied accross all experiments presented in this work.
We evaluate the perceptual quality of generated images using the Fréchet Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Bińkowski et al., 2018). On the one hand, FID measures the distance between the feature distributions of real and generated images using activations from a pre- trained Inception network (Szegedy et al., 2016), and assuming the features are Gaussian-distributed.. On the other hand KID computes the Maximum Mean Discrepancy (MMD) with a polynomial kernel. All metrics are computed on 10k samples instead of the standard 50k. Due to the smaller sample size, FID (often referred to as FID-10k in this context, but we simply refer it as FID) tends be overestimated, as it is biased upward.
A.2 Random Forests
We conduct experiments on Forest-VE VP by varying the number of trees, denoted as K (corresponding estimators in the scikit-learn implementation). The values considered are (1, 10, 25, 50, 100, 500, 1000).
20

---

Published in Transactions on Machine Learning Research (02/2026)
where K 100 is the default setting used by Volicoeur-Martnnann et al. (2024). Moreover, consider various aggregation methods between the trees.
We adopt the same training and sampling procedure as the reference model in the original paper (Jolicoeur- Martineau et al., 2024, Table 6). In particular, we set max_depth following the cited work?, which is sufficient for small datasets such as Iris.
To ensure consistency with Dolicoeur-Mattinee stein distance between the generated and test data, coverage (Naeem et al., 2020), and efficiency (Xu et al., 2019), with results reported in Tables 8, 12 and 13. Coverage assesses the diversity of generated samples relative to the test set. Efficiency, is measured as the average F1 score obtained from training multiple non-deep machine learning models for classification or regression on generated data and evaluated on the test set.
et al. (2024), we evaluate Random Forest using the Wasser-
Alongside the Iris dataset (n 150, = 4), we report additional results on larger datasets (n > 1000) in Appendix F.1.
A.3 Confidence intervals
In this section, we detail how confidence intervals are computed for the different experiments presented in the paper. We distinguish three cases: 1) FID/KII evaluation on ensembles, 2) LDDSM evaluation on ensembles, and 3) FID/KID evaluation in Section 5.3.
A.3.1 Standard deviations on ensembles
For FID and KID evaluations on ensembles of size K = in Figures 3a to 3d and 5b and tables 1 to 3 and similar reports in Appendix, variability assossed across different model subsots and not through repeated metric computation (due to the computational cost of FID on ensembles). For individual models, the interval reflects the variability across the individual models taken separately. For subsets such 2 models, we report the variability obtained by evaluating all possible pairs among the five available models (i.e., (9) combinations). For both configurations, we compute the mean and standard deviation of these measurements. For the full ensemble (K = 5), the metric is computed only once since all models are included.
In the case of Figure 5b, where K corresponds to the full ensemble, we use the minimum and maximum values to represent the uncertainty on K 1, this choice more appropriate than taking the standard deviation.
For the LDDSM evaluation (Figures 3e and 7a and table and similar reports in Appendix), we are able to measure the metric over multiple model combinations for each ensemble size, and repeat each measurement up to 10 times thanks to its low computational cost.
A.3.2 Intervals for a fixed model w/ bootstrap
In Figure 7b, since we rely on single fixed model and require precise estimation of the trend of these perceptual metrics, we assess variability across measurements using the bootstrap method (Hastie et al., 2009, Section 7.11), which offers a cost-cffective alternative to repeated evaluations. The bootstrap method estimates variability by repeatedly resampling (100 times our case), with replacement, from the original set of measurements. Each resample is used recompute the metric, producing an empirical distribution of estimates. The confidence interval can be estimated for example from the standard deviation of the bootstrap distribution by taking its 2.5th and 97.5th percentiles for 95% interval, which is what we do. FID is a biased estimator, as fewer samples tend to increase its value. Standard percentile bootstrapping can therefore introduce additional bias due to resampling with replacement. To ensure that bootstrap samples contain about 10000 unique observations (matching FID-10k), we adjust. the sample size accordingly as follows.
21

---

Published in Transactions on Machine Learning Research (02/2026)
In bootstrap resample of size n, each observation has probability of not being selected given by
" =e- 0.368, -
1
(10)
which means that about 63.2% of observations unique on average. To obtain 10000 unique samples, we solve
0.632 10 000,
(11)
which gives
823.
(12)
Thus, using bootstrap samples of size 12. ensures roughly 10000 distinct observations per resample, aligning the evaluation with FID-10k.
B How to evaluate score matching and model likelihood arithmetic mean?
Given trained model 0, the procedures for computing LDDSM(() and Po.0 ODE (x()) are well established, as explained in Section 2.2.1 and by Song et al. (2021b). Thus, {0 denotes a collection of K set of parameters optimized using DE, we essentially evaluate the metrics for model averaging by plugging (8)
(K) (x, t) = F E | 8g (x, t) into their formula. In the following we provide details for implementation of this particular case, but can be extended to any step-wise aggregation.
B.1 Score matching loss of arithmetic mean
Let Xy E Daaldd- We sample t E [0, and z ~ N(0, I) to compute x2 ~ (x (Xo) using and the explicit formula of the forward transition (see Song & Ermon, 2020, Appendix B). The only difference is that we replace the individual instance of a model by an average. For example, we consider an SDE with zero drift and noise schedule a [0, 27 + Rxo: LDDSM is computed using |ل (K) (x,,)(x,,)) f) + o l12 which is averaged using MC over the distribution of xo and the timesteps t.
B.2 Model likelihood of arithmetic mean
Before discussing ensembles, let us first present a short review of likelihood calculation for diffusson models. Associated to Equation (3), there exists corresponding deterministic process following ODE and whose trajectories [xe [x share the same marginal densities {xx))) as the SDE:
dx di
[(xxt) } ( (x) (00
f(x,
(13)
In this case, we denote Pi, (x,) the probability at each time induced by the parameterized ODE. Un- der this framework, we have particular case of Neural ODE/Continuous Normalizing Flow (Chen et al., 2018). Consequently, the likelihood P6.0 (xo) can be expliciuly derived by using the inslantaneous change- of-variables formula that connects the probability of po.0 (xo) and P1.
ODE
ODE (xn), given by
ODE (x())) = =dd" V F(x()))d) ODE (x())) Po.6
(14)
where (Vx.) denotes the divergence function (trace of Jacobian w.r.t x). For the SDE counterpart, Song et al. (2021a) established that log ( (x(0) admits a tractable lower bound, while Albergo et al. (2025) newly derived a closed-form expression for il.
Now we explain how to compute this for an arithmetic mean of K models. We denote by (x)( the trajectory of the ODE associated with the score model $9 set of models 01,..., 0k. we refer to it using the shorthand for simplicity, following the notation used for the
(K) (x, t). Although {xo(t)}{o depends on the full
22

---

Published in Transactions on Machine Learning Research (02/2026)
average score function. The log-likelhhood can be exactly calculated by numerically solving the concatenated ODFs backward from 7 to O, after initialization with xo(0) ~ 20 (solve_ivp from scipy. integrate),
d xo(() logPr. P (x (xx
(15)
dt
(k) (x)(t) (x) 1 (t)xx $o (K) (xo Vx (t),
f(xo (t),
(16)
The divergence Vx (.) is approximated using the Hutchinson Montc-Carla based estimator of the Trace (Hutchinson, 1989), molivated by the identity
M
Tr(A)= = Ez-N((, Az]
Vx
(K) 80
~ (xo
1
Σ
Zm
V Vx (K)
(xo
z
(17)
M
m=
where 21,..., 2M (O, I). Following Song et al. (2021b), set M = 1. To correctly compute the NLL, dequantization techniques employed; see Theis et al. (2016) for details and justification.
C Theoretical insights on model aggregations
In this section, we complement our thcoretical findings by providing proofs and additional implications of the results. Appendix C.1 addresses the denoising score matching loss whose associaled result is Proposition 4.1; Appendix C.2 expands on Section 3.3, notably by including Mixture of Experts in the analysis; Appendix C.3 develops the misconception raised in Section 4.2; and Appendix C.4 provides the detailed framework and proof of the main result stated in Proposition 4.2, followed by discussion of the contexts implied by this result in Appendix C.5.
C.1 Arithmetic mean and model performance
In this section, provide the proof of Proposition 4.1, which demonstrates the benefit of ensembling on LDDSM- We first establish preliminary result showing that ensemble of K models performs better than single model, before turning to the main result that K+1 models outperform K. We also derive short consequence concerning the KL divergence between probability density paths.
K+1 models are better than for score estimation. The training objective of diffusion models is to minimize the denoising score matching loss in Equation (4), which computes the mean squared error between the estimated score and the true conditional score given clean images, averaged over all timesteps. This formulation has the advantage of roducing the inherently difficult generative modcling task to series of simpler supervised regression problems. Given any positive time-dependent weighting function X(-) and model so(:), we define an equivalent definition to Equation (4),
LDDSM(SO. ((.))))s = 2 1.(" Emoxxojnppoxxxxxxo (x)))] Vx, log (x)((x 12]
(18)
In the following we demonstrate that when models from an ensemble of size are similar, adding one model improves the accuracy of our score estimation in average.
Proposition C.1. Let 5 .......s
.0
(K) be score estimators mapping Rd x [0,T] to Rd. Then, the DDSM
loss satisfies
(K) (K) x(-)) S
1 K
CDDSM
Σ
80 X(-)
(k)
(19)
K-
hyj
Moreover, for any (xr,t) E Rd x [0,T], the outpuls 8 (1) then,
(xex (),..,,
80
(K)
(xx, 1) are identicolly distributed,
E LnnsM(s! (K+1) " .A())] < Lnnsur(3! (K)
,A(-))
(20)
23

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. Let K > 1. As first step, establish the following algebraic identity:
1 K
K
K
9
(x,, t)
K
K
$
(xe,
t),
(21)
k=1
1=1
which is a direct consequence of We now use Jensen's inequality and the convexity of the norm obtain the first inequality.
0
1
0
1
+
(22)
K
-
CDDSM(33 (3) (3) (x-))-
2 (xo(x))) (xo)
1
!
A(1)l
1
(k) (xx;,)
Vx, log (x)
(23)
K
k=1
1
!
TT
X(t)l
1
Σ
K
Σ
$0
(b) (x,, Vx. (xx
2
K
ktj
(24)
1
T
x(1)l
1
Σ
80
(1)
(x1, Vxx log (00
(25)
2K
K
K
DDSM
(h) 80
2(1)
(26)
K
K-
(()),))) Now assumee that for any pair (x, the random variables s 5(0, t) tributed given (x, Then,
are identically dis-
K Σ
E[CDDsM (K) (3) (2,,))))
CDDSM
1
(1)
A(-)
(27)
K
K
0
j=1
koff
K
K
E
CDDSM
Σ
8
(k)
X(-)
(28)
K 1=1
K-1
1=1
E
LDDSM
88
(K-11)
(x-)]
(29)
Similar results are well-established in the case of convex losses (Mattei & Garreau, 2025). Proposition C.I shows that ensembling multiple score estimators systematically improves the DDSM loss. The first incquality holds deterministically: removing an element from the ensemble uniformly at random will, on average, degrade its performance. The sccond incquality strengthens this result in expectation, showing that when ordering does not matter (ie., for i.d. estimators), a. larger ensemble performs better than smaller one in expectation.
(1) We now discuss the i.d. assumption, In our framework, we use DEs, where the models s ..... sg are obtained by training the same neural architecture independently according to the method, on the same
(K)
24

---

Published in Transactions on Machine Learning Research (02/2026)
dataset, but starting from different random seeds of the same initialization distribution. As result, for any fixed input (), the outputs of the individual score models can be regarded as identically distributed samples from an implicit distribution induced by the training randomness. Therefore, our setup satisfies this assumption.
A more general result on score matching can be equivalently deduced. We expect well-trained score models to minimize the following least squares objective:
(s)) = (0 >ee V log - so(x, 1)
(30)
Although this expression is intractable in practice, it is equivalent up an additive constant to the DDSM loss described in Equation (4), as shown by Vineent (2011). Morcover, by applying the same arguments as in the proof of Proposition C.1, we can establish a similar inequality for Equation (30), namely
E CDSM 1)+ s (505)
(31)
K models estimate the path measure better Let qu.r and Po:T be the path measures of the trajec- tories (x(t))²=o and {xo(t))<=o where the former is a stochastic process solution to Equation (3), and the latter is solution
dx = [f(x() (f(x,))-(()" x())2xg (xx) (x, 1) g(()dw.
(32)
Both measures be seen as joint distributions for which qu and poDe are marginals. Lu et al. (2022)
showed that if we consider SDEs with fixed terminal conditions x(T) and xx())) 2,
SDDE Po.T.e .
( (T) z) || ( (- x2
=)) = EEgor
log
LDSMM (5) -g(-))).
(33)
do:T
Combining this with Proposition C.I, we show that the aggregation of a large number of score models leads to more precise approximations, on average, of the true SDE solution given a terminal point z.
C.2 Aggregation rules and their connection to density pooling
In this section, we illustrate how the first three score-level aggregation methods introduced in Section 3.2 can be interpreted through the lens of model composition. The objective is to provide intuition for these aggregation schemes. By model composition, we refer to the combination of multiple probability density functions to form a new distribution, defined up to normalizing constant. Initially studied for Energy Based Models (EBMs), model composition have mostly been applied for conditional generation in the form of classifier or classifier-free guidance (Du et al., 2023; Skreta et al., 2024; 2025).
Arithmetic mean of scores. Let (x), p())) (x) be K probability density functions defined on Rd. Then,
1 K
Vx (x)
= Vo II (x))
(34)
In other words, the arithmetic mean of the individual score functions corresponds to the score of the geometric mean of the densities. This composition yields an unnormalized distributiin, but the unknown normalizing constant vanishes under the gradient. Given the central role that the arithmetic mean of score models will play in our study, we formally introduce the geometric mean of densities Product of Experts in Definition C.1.
Definition C.1 (Product of Experts (PoE)). Let defined on RRd
be K probability density functions
P
(KK) (x) = PXF((
(x)) = (())
(35)
where Zx= = 7 p z (lk)
Using AM-GM inequality one can show without difficulty that Zς is finite and then the normalized compo- sition is well-defined as density. We now state result, that follows as direct consequence.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Sum of scores. In the same way, we can write the sum of scores score since
K
K
Vx (x) = Vx log II p (x).
(36)
It (x)
The operation /IL (x) was originally called Product of Experts by Hinton (2002). This operation
is efficient when the models are different. Indeed it allows each expert to constrain different aspects of the data. As a result, the composed model assigns high probability only to points that satisfy all individual constraints simultaneously, like an AND operator. For example, Du et al. (2020) applies the product rule to model the conjunction of concepts with EBMs, where each density is of the form ( (x) = p(x (&) with Cl:..., CK representing different concepts. More recently, Du al. (2023) applics the product to perform class-conditions and text-to-image generation using diffusion models (e.g. (x)( p(x|"A A sandy beach")).
Mixture of experts. The equivalent of the Product of Experts as defined above but for the union of distributions is the Mixture of Experts defined by
P ( (x) = K
p M
(x).
(37)
This one corresponds to soft union, assigning high probability to regions where at least one of the experts does. In the case where each (x))(x) models conditional distribution p(x (x) for concept Ck, the mixture defines model over samples that belong to any of the concepts {ci,..., CK J. A cheap way to model this given K gencrative models trained for example on different data would be to use method described in Section 3.2, that is randomly selecting a model (thus a distribution) among the K ones before sampling.
In comparison to the former composition rules, there no formula that express the score of the mixture only in term of individual scores. We have
x (kgg(()
(K) (x)
Σ (1)) (x) (ogp (x)
(38)
with (x) = et al. (2024) provides a way to estimate this compositions ions for diffusion models by using Itô density estimators
E (x)) would require to know the individual densities (x))....,,(()))))))) Skreta
(x)
on the fly during sampling.
Operations on probability densities often correspond optimal solutions under divergenco-basen based criteria. Typically; the geometric mean is the density p that minimizes the average KI. divergence to the individual models
(1)
(39)
K
Similarly, the mixture of experts minimizes the average reverse KL divergence. More broadly, Amari (2007) introduces the concept of "a-intcgration" to generalize density combinations, and establishes general op- timality result. for this family of means.
C.3 Pitfalls of these interpretations
Although the equations described in Appendix C.2 are valid out of the diffusion context, composing score models at each timestep in these ways does not actually guide the diffusion process towards the distribution obtained by reverting the gradient operator and the logarithm. The reasons are the following.
1. We do not predict score. Indeed, generally s0 + Vx logggss
SDE (x) and s0(x,t) t) #
Vx log tog (x).
26

---

Published in Transactions on Machine Learning Research (02/2026)
2. Composition and adding noise are not commutative. Typically, by averaging the scores, we don't sample from PoE. because the distribution at time >>> 0 is not the PoE of the trained models.
We detail these two observalions below.
(1) There is no guarantee that the score model has a structure of a score, or even that is predicts at each step t the score associated to the distribution at time t of the trajectory. For example we consider the SDE with linear drift, a necessary condition that PT. pSDB (x) be Gaussian (Lu et al., 2022) which generally not
the case, even though it may be very close in practice. The reason is that qo is not Gaussian, and even if it were, the condition would only hold asymptotically as T oc. In fact in the ODE case in Equation (13), it has been shown that the score output at time t 0, (xx t = 0), is less accurate than computing the gradient of Po.0 (x) with respect to as given by Equation (14) (see Feng et al., 2023a, Figure 3). However, well- trained diffusion models can still generate high-quality samples, as we have convergence guarantees: under suitable assumptions, the Wasserstein distance between SDD and qo is upper bounded and tends to zero with improved learning and finer Euler discretizations (De Bortoli, 2022).
(2) To sample from the product of densities using diffussion models, for example in the guidance framework (Dhariwal & Nichol, 2021; Ho & Salimans, 2022); we operate the product al each noise level of the sampling procedure by replacing the score with what would be the score of the product. For instance, given feature y (e.g. a class), to estimate p(xly), we sample from slighly different distribution
x y
x
p(x)
ply | |x)
(40)
(2)
(x))
(x))(x))
(x)
p(x y)) p(y) 1+:00
= -p(x)
(41)
p(x)
p(x ly) 1+
x x(x)
(42)
p(x)
-p(x)
p(x (y) 1+
(43)
(x)
(x))
where is a temperature parameter. While Dhariwal & Nichol (2021) rely on Equation (40) and train a classifier independently from the diffusion model to approximate the conditional likelihood, Ho & Salimans (2022) adopt Equation (43) and jointly train two diffusion models instead. Hence, if for example we consider the latter, we compute at test time Vx lo y) = -wV, (x + (1 + V) log (xly) for every > O, where (xlx)) and Vx logp((x) are independently learned. However, this stepwise rule assume that the proportionality in Equation (43) hold for noise levels & > 0. It is false when pt and pc(- y) are the diffusion distributions obtained by computing convolution of p and p(- y) with Gaussian noise levels. Fortunately, this pitfall does not compromise the effectiveness of the guidance procedure in practice, showing that this trick is good proxy of the target distribution.
As indicated by Du et al. (2023) and Chidambaram et al. (2024), the misconception in reasoning is also true in the general case of product and geometric mean: the operations of applying noise to the probabilities and computing the product or the PoE in Definition C.1 are not commutative. We may schematically write
(x), (K) (x)), (x)), + PoE(p!") ()) (x))) (x)...., (K) (x))
(44)
as soon as > if (x), ... (k)) (x) are the starting distributions associated to each trained model (in fact # means "not proportional to here). The direct consequence is
K
(K) Vx loggt (x)
1
log
(k) Pi (x).
(45)
K
However, to the best of our knowledge, no simple, closed-form counterexample has been found support this inequality. In the following section, we provide a proof using the case of centered Gaussians.
27

---

Published in Transactions on Machine Learning Research (02/2026)
C.4 Diffused PoE is not the PoE of diffused distributions: proof in the Gaussian case
In this part we show that in general, the PoE of distributions at / = 0 does not lead to the PoE of marginal distributions {pr (p()) Jiso induced by each of the diffusiio trajectories. More precisely, we demonstrate that necessary and sufficient condition is that are all initial distributions are equal (e.g. sample unique distribution). We prove this assuming the target distributions are all centered Gaussians in Rdd This case is particularly interesting, it yields an explicit score and admits analytical solutions for the backward SDE (Pierret & Galerne, 2025).
Following Pierret & Galerne (2025), we consider the Variance Preserving forward SDE (or Orn- stein-Uhlenbeck process)
dz, dt V 23,dw, S 2o ~ qo
(46)
The distribution go noised progressively according to the variance schedule 3r. The equation 46 admits one strong solution written
24 72 n/, OStST
(47)
where E (0, 1) and > 0 are respectively deterministic and gaussian independent of However both depend on Bt. Moreover, if E, denotes the covariance matrix of 2t, we have
E, = 3 + (1 I
(48)
if Σ the covariance matrix of zo.
Assumption 1 (Gaussian assumption). In the following we assume that go is centered Gaussian distri- bution, namely (0,, E), and that Σ is invertible. In this case,
2, ~ = NO,
(49)
and Σ, is invertible for t >> 0.
What is the PoE of (centered) Gaussians? to (0, Ex). Then
Assume .l),..,,(k)) are all Gaussian distributions equal
() (K) N (0. 1 ME
(
Ex )))
(50)
k=1
where the equality holds thanks to the normalization coefficient.
Main proposition. We start with the following lemma.
Lemma C.1. Let a RkK and b R Let (0....,,KK) + ( 2k=1 2 -= denote the harmonic mean function that takes vector in BK Then H is super-additive:
H(a) + H(b) < H(a b)
(51)
and equality holds if and only if there exists > such that a = Xb (we wrile a x b).
Proof. Let a = (ai,..., ax) E R and b = (bi,..., bk) E Rg The inequality corresponds to special case of the reverse Minkowski's Inequality for Sums (Proposition C.3)
K (ax + bb)
1/p
1/p
K
1/P
Σ
>
Σ of
+
Σ
(52)
k=1
k=1
valid for all p 1, and in particular for p = -1
The counter-example is the following.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Proposition C.2. Assume we under Gaussian ussumplion for K distribulions Po .......P Po and
(1) (K).
(K), (k)) is the initial
moreover each initial covariance matriz is equal to ay with og > 0. po Pob(pu ,o. condition of Eq. (46), then for each t > 0, the marginal distributions Pt associated to the strong solution {x:]150 verifies
(
Pt PoE(pt ( .....P.. .!.))
(53)
if and only if on =... =ak.
Proof- Let K > 1. 1<<s T, and o1....,00 (50), Pt (0, )) where
oK > O. Let us derive the distribution Pc. From Eq. (49) and Eq.
PoE
7o(
K
1
Σ
k
0k
1)
+ (1-71) ---]-=
7al
K
Σ
k
k )
+ (1 76) I.
(54)
k=1
k=1
In the other side, PoE(p)) (1)
"...,.".... (k)) = (O, Dqo PoB) where
-1
Enot PoE
(aay + (1- -1)))- Σ
(nay + (1 - 7t)) +1--1))--
(55)
og PoB
Let us compare EP and Enot PoE We establish the following result:
PoE c
(56)
and PoEE cpot PoE + vk (1...., K}. = A (1-)) with A > 0.
To prove this, we write both scalar values in terms of harmonic means.
c
PoB
=7Hl((an...,, ak) + (1 -1)
(57)
poot PoE
H(7::n + (1 7)... ... Max + (1 - %))
where H x (Zi....,, (.k) + ( ΣK 2x - Since everything is positive, the result is straightforward using Lemma C.1 and ((Ax) = (x for all (A,x) E (Rso, RK)).
PoF = Ha 0k) + (1 H1, H(x(()....,kk) + H( - ;), 1)) < H(mai + (1 7). ... max + (1 (1-)
(58)
(59)
(60)
PoE = c
(61)
with equality if and only there cxists X > such that for all k E f1....,KK, K}, a = d (1-w)) which is equivalent all the og being equal.
This proof demonstrates that in the particular Gaussian case, the equality is equivalent to all the individual starting distributions being identical. In this case, and even more generally, for any E (0, 7] the interme- diate probability associated cannot be expected follow PoE structure, and computing the PoE intermediate steps docs not faithfully reflect the PoE of the initial distributions.
C.5 Implications of commutattttt in two contexts
We provide two well-knnown yet simple frameworks that are affected by Proposition 4.2: concept-conditional generative modeling and linear inverse problems.
29

---

Published in Transactions on Machine Learning Research (02/2026)
Concept-conditioaal generative modeling. In this setting (see Appendix C.3), the components of the product are expected to be different. models. Thus, Proposition C.2 hints that relying solely on the guidance approach is unlikely to produce samples from the true target conditional distribution. Some recent works leverage some hacks like MCMC or Feynman-Kac-basee theoretical pitfall and improve sampling accuracy (Du et al., 2023; Skreta et al., 2025). No such correctors are applied our experiments, as we focus on the baseline ensembling behavior without additional guidance refinements.
corrections during sampling to handle this
Linear inverse problems. Let us consider linear inverse problems, that is, observe measurement
y Axo c,
(62)
where A E Rixd is known linear operator (or measurement matrix), Xo I2 the unknown signal to be recovered, and Rh denotes additive noise, typically assumed to be centered Gaussian variable independent of Xo. The goal is to recover Xo from the observation y, possibly under prior assumptions the distribution of or through generative model. We would like to estimate p(xly) x p(x)p(yxx). We can again adopt the "guidance" approach by using a pre-trained model aiming to estimate Vx Pa (x.))))))))))))))>>)) and add to an approximation of the measurement matching term Vx logg (x.) for each noise level of the sampling process. The latter term is estimated using the identity
/ ply xo) p(xo | xx) dxo-
pc(y Xx)
(63)
and numerous approximation methods can be leveraged (Chung et al., 2023; Song et. al., 2023).
Assune x = Xo ~ go = NN Σ). For this part, let us consider e.g. m = and A > 0. The likelihood function p(y/x) taken x has quadratic logarithm and corresponds to
1 (y Ax) Σ -y (y--yxyx)
plylx) x exp
(64)
2
x exp
2 (xT (A E-- - 2x EA--A-_1 yy)
(65)
x exp
2x TS-15 -
2 (x - 2x
(AT
(66)
1 (- (- xx- (n) T M))
x exp
(67)
This yields a Gaussian form with mean EA A) A y and covariance <AT -'A--- provided the latter is positive definite (which is the case here since m = d and A x 0). When y = 0, the distribution is centerod. Bocause p(x) and p(y x) are distinct, particular they have different covariances, the necessary condition in Proposition C.2 is violated, and therefore we cannot expect Equation (53) to hold in this setting.
C.6 Some useful algebra results
Lemma C.2 (Reverse Young's Inequality for Products). Let p, q Byy strictly posilive real numbers satisfying:
1 ==1 Let a E Rzo be positive real number and R50 be a strictly positive real number.
Then
ab > 0"
19
(68)
p
with equality if and only if a² b-9.
30

---

ab a² b-9 Published in Transactions on Machine Learning Research (02/2026)
Proof. We define and such that and = g. By hypothesis, 1 + Inequality for Products (Rudin, 1976). (ab)pb =
thus can apply Young's
((ab)))
(b-v)o/p
(69)
1/p
q/p
b-
a S pab
+ p
(70)
b
p S ab
+
(71)
q
(72)
p q
Equality holds if and only if the equality case in Young's incquality is attained, that is when
ab b- ===bb 0
(73)
Lemma C.3 (Reverse Hölder's Inequality for Suns). Let a E Rso be striclly posilive real rumberrs such that =.
Suppose that the sequences x (InJnEN and y = {yn TR or Red are such that the series
1/p
||x||= =
|xnnl 12
(74)
and
-/9
1brl--- = Σ 1n
(75)
are convergent. Let [|xy]] denote the 1-norm of xy, if xy is in the Lebesque space C. Then,
|xyll 2 ||x||p|y|||--
(76)
Equality holds if and only if there exists a constant > 0 such that for all n E N,
|xn| = c-
c-lumll 1
(77)
Proof. Without loss of generality, assume that x and are non-zero. Let
u {unIneN
x
(78)
||x||
and
y
V = (UnFneN
(79)
Ty -q
Then,
||u| = |V|l-- 11
(80)
Since p 1-1=1, by Reverse Young's Inequality for Products have (Temma C.2),
1n, 2 Un
12
|nn 1cn1-9 19
(81)
q
and summing over all n E N gives
1 2
1 vl
||vv||
=
(82)
p
9
31

---

Published in Transactions on Machine Learning Research (02/2026)
desired.
From Lemma C.2, equality holds if and only if
EnEN, |n |ual"
(83)
and then Hölder's inequality becomes inequality if and only if there exists o, > 0 (namely ||y||
and 3 = |x|| such that
a|z,| 1 = 3.
(84)
Proposition C.3 (Minkowski's Reverse Inequality for Sums: case p 1). Let a (a1, )² E R
b = (61,...,brr 6k)²² E Ry and E R. p < 1, p. O, then
1/p
K
1/p
1/p
Σ (ak + 6p)
d
(85)
N=
and equality holds if and only if a = c- b for some c >> 0.
Proof. Define
P I Then,
1
(86)
p -9 P p
where p > 0 and -q O. Using Lemma C.3 followed by (p- 1)q p, have
K
K
K bax + bk)p-1
akk + bx))
=
(ax + ba
p-1
+
(87)
R=1
k=1
k=1
1/P
K
1/2
1/P
K
1/9
(aax + by
12)
Σ
8%
(lax + ba) 1))
(88)
k=1
K
1/P
K
1/4
K
1/P
K
1/9
Σ
(ax +
bz)P
+
Σ
(ax: + bz)p
(89)
k
A=1
k=1
k=1
K
1/p
K
1/P
K
1/9
K
P
+
Σ
(ak + by )2
(90)
(91)
Then,
K
1-1/9
K
1/p
K
1/p
(ax + 6p)
k
+
k
(92)
kml
km
K (ak + bk))
1/P
1/p
K
1/P
d
b
(93)
From Lemma C.3, equality case holds if and only if there exists Ci C2 > 0 such that ap = (ax + by)
and 6g colax + bb),, which means a b. To demonstrate the latter, suppose ag (a + 62) and by = c2(a ) for all k, with et, 02 0. Then
as
by ak + bk
P
C1
C2-
(94)
0g + bk
32

---

Published in Transactions on Machine Learning Research (02/2026)
Dividing the two equations yields
1/v
ak bk
Gk
Ci
for all k.
bk
Hence, a x b. Conversely, if a = c. b for some > 0, then as + by = (1 + c) bbs so
d cº b/ = c1 c) 2 (ak + bn),
6% (1 + cP (ak + b²)².
Thus, the condition holds with c (1 + p and C2 (1 cc
D Additional aggregation schemes
While Section 3.2 mainly focuses step-wise aggregation schemes, it includes only model that op- crates at higher level. This raises the question of whether ensembling can be performed solely at the final stage of the process, for example. Additionally, all our step-wise aggregators follow the form SDE_SOLLVRRE SOLVER(AGG(,.... the order and apply the AGG function after each step of the SDE SOLVER. We describe two additional aggregation schemes in Appendix D.1 and show in Appendix D.2 that these methods do not perform better in practice Deep Ensemble than our main approaches.
as illustrated in Figure 2, but one may wonder what happens we reverse
D.1 Detailed descriptions
We list two methods that operate differently on the sampling trajectories. Song al. (2021b) provides general framework for reverse diffusson sampling by combining for each noise level predictor step, that drives the sample toward the solution of the SDE, and corrector step that drives the sample to more correct sample given the noise level. We write Predictor(x:, (x) = X,, + (X,,(xx) In. zn) and Corrector(xx,,) = 6,X1, + C'(s (Xt,, tx), zn) where Zn is standard gaussian noise. If we consider Euler-Maruyama step as predictor, Yn = and (X tx), tn. zn) = [f(),.., f..) (tt.)sp(x,,.. tn)] At + ((t,))Atz,, For DDPM, construct sequence of scales (nt,))e( then set Vn VI-A,, and
(x) (n), bn kn) Den so(xt,, (m) v /r,,2n Un. If we consider Langevin dynamics for the corrector, we can set 8n and (x) tn), zn) = (x.s0((1,,) , tn) + 22,
VI-
Average of noises. At each step Intl, we average the noises added to Xt,, to get Xt,-1 in the predictor. Let's define Au+1 Xt, 8t, . It represents the noise injected into the image state to advance to the next step. Here we propose R Ek A & E P(s (xt,, bn ((A)), assuming for simplicity that only the Predictor step is laken into account. Typically, if we consider Euler-Maruyana steps, the update corresponds to
K
[(xx,, (t. (K (x,,, (n) At + gttt)
Σ
(k) n
(95)
K
1=1
which almost equivalent to simply averaging the scores due to the linearity of the updates, though it differs because the Gaussian noise terms are also averaged. Not that the latter vanish asymptotically with K, leaving only the contribution of the averaged scores. Asymptotically it corresponds to the discretization of an ODE
(k) (x, dx [f(x,t) Ea-u. (x) t)]tt + g(t) Eldw
(96)
One can also average the corrections in the corrector steps, similarly to the predictor.
33

---

Published in Transactions on Machine Learning Research (02/2026)
Mean of predictions / Posterior mean approximation. We compute the mean over all samples Xo generated from K sampling paths for given input x. This approach aligus with the objective of Bayesian inference in the generative modeling framework. For diffusion models, the posterior predictive distribution is obtained from an initial noise z following a "prior" distribution (Gaussian noisc). Hence averaging the outputs corresponds to approximating
p(xolz,D)= = ()( EE p(Rolzz 0s)
(97)
The difference from our framework is that represents here the stochasticity introduced in the entire sampling process (not only the starting noise). Some works employ this bayesian inference technique, using different ensemble strategies to sample 0₂ like Laplace approximation (Ritter et al., 2018) or Hyper Networks (Ha et al., 2017), to estimate uncertainty for diffusion models Chan et al. (2024); Jazbec et al. (2025).
D.2 These methods are under-performing
We present additional results complementing Section 5.2.1, by incorporating aggregation schemes that per- form extremely poorly and are described in Appendix D.1. In addition to the sum mentioned Scction 3.2, also evaluate the two trajectory-level aggregation schemes described in Appendix D. We provide results Deep Ensemble using ADM models trained on FFHQ.
Table 4: Comparison of different aggregation methods for on FID, 18, and KID. The parameter defines the level of entropy involved during DDIM generation, see Song et al. (2020) for a clear definition. The result highlighted in orange is the one reported Table 1.
CIFAR-10 10 (32 82)
FFHQ (256 x 256)
Deep Ensemble
FID J
IS
KID
FID
KID
Individual model DE
4.79 (0.08)
9.37 (0.07)
0.001 (0.000)
24.20 (5.70)
0.015 (0.003)
Sum of scores
516.19
1.14
0.591
524.33
0.69
Mean predictions (1)0.0)
58.31
0.04
Mean predictions (1)0.5)
144.89
0.13
Best model
4.65
9.45
0.0006
21.7
0.012
E Tables and plots for U-Nets ensembles
In this section, we first present in Figure 8 the distribution of individual FID-10k values to complement Table 1. We also provide in Tables and and fig. 10 additional tables and plots on ensemble strategies to complement our study in neural network ensemblling for diffusion models (Sections 5.2.1 to 5.2.3). We also report the effect on NLL given by incrcasing K in Figure
E.1
Distribution of FID-10k from DE on CIFAR-10
E.2 Aggregation schemes
E.3 Ensembling on the noise space
We further investigate in Table whether limiting model averaging to the early stages of the reverse diffussin process mitigate the degradation in perceptual quality observed with full-step ensembling. The rationale is that, during the initial timesteps, the model primarily reconstructs coarse structures, while later steps refine high-frequency details that strongly influence perceptual metrics such as FID. Averaging may thus be less harmful if applied only to the early denoising stages. To test. this hypothesis, we perform ensembling over the first third of the diffusion steps only, keeping the remaining steps deterministic.
Partial ensembling does not improve FID compared individual models. Interestingly, the resulting perfor- mance (FID- 4.83) lies between that of the individual models and full-step ensembling, suggesting that the
34

---

Published in Transactions on Machine Learning Research (02/2026)
X
8.98
4.89
4.9
4.83
4.83
4.8
4.76
4.65
M1
M2
M3
M4
M5 Ensemble
Average FID 4.79
Individual models FIDs
X
Ensemble
Figure 8: FID-10k values from all individual models Mi trained with DE where i E (1,2,3,4,00) Ensemble aggregation the arithmetic mean. Score averaging performs worse than the weakest individual model, which suggests an existing blurring effect caused by score averaging all timesteps. It illustrates that ensembling can sometimes be detrimental.
Table 5: Comparison of different aggregation methods for K = 20 CIFAR-10 in terms of FID, KID, and LDDSM- The best value for each metric is highlighted in bold. We do not measure it on FFHQ since the associated model is not trained with Dropout.
CIFAR-10 (32 x 32)
MC Dropout K 20
FID
KID &
LDDSM
w/o dropout
4.83
0.0008
0.0284 (0.0006)
w/ dropout
5.75 (0.07)
0.0014 (0.0001)
0.0286 (0.0003)
Arithmetic mean
5.85
0.0014
0.0286 (0.0005)
Geometric mean
5.87
0.0014
0.0286 (0.0003)
Dominant
8.05
0.0044
0.0285 (0.0004)
Median
5.95
0.0016
0.0285 (0.0002)
Alternating sampling
8.21
0.0017
Mixture of experts 7.47
0.0016
negative effects of ensembling on CIFAR-10 are mitigated when is applied over fewer timesteps. However the differences remain small overall, indicating that the results are essentially similar.
E.4 Arithmetic mean and NLL
The model log-likelihood is computed according to the explicit formula and method given in Appendix B. We use Monte Carlo samples to estimate the trace.
The metric shows marginal downward trend with increasing K. (soe Appendix B).
E.5 Effect of the number of sampling steps on ensembling performance
DDIM enables sampling with a number of diffusion steps that is 10 to 100 times smaller than that used during DDPM training (e.g. 1000), while still maintaining satisfactory image quality (Song et al., 2020) as long as the number of steps is not too small. In Figure 10, we use numbers lower than 100, which initially sclected as it provided the best tradc-off between speed and quality.
35

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: We complement results from Table by evaluating ensemble performance for K models trained on disjoint subsets of CIFAR-10 in terms of FID, KID, and LDDSM- Each model is trained on two classes. The arithmetic mean produces catastrophic results: generated images are extremely noisy. This is because combining models of fundamentally different natures. Moreover, if consider that it approximates PoE, such failure is unsurprising, it essentially attempls to sample from empty support. Mixture of experts performs better, as it avoids this issue, but the models remain too domain-specific to fully exploit diversity and reach high performance.
CIFAR-10 (32 32)
Subsets w/o intersection
FID J
KID
LDDSM &
Individual model DE Individual model
4.79 (0.08)
0.001 (0.00)
0.0284 (0.0005)
65.42 (7.66)
0.037 (0.00)
0.0381 (0.0008)
Arithmetic mean
90.16
0.072
0.0345 (0.0006)
Mixture of experts
12.05
0.007
Table 7: We evaluate FID and KID when using a Deep Ensemble where arithmetic averaging of the scores is applied only during the first one-third of the 1000 diffusion timesteps. After this early aggregation phase, sampling proceeds with single model for the remaining steps. This experiment is repeated five times, each time using different model from the ensemble.
CIFAR-10 (32 x 32)
Early ensembling K
FID
KID +
Individual model DE Arithmetic mean
4.79 (0.08)
0.001 (0.00)
4.98
0.001
Arithmetic mean
4.83 (0.08)
0.001 (0.00)
F Tables and plots for Random Forests
In this section, we provide additional tables on ensemble strategies to complement the part on Random Forest (Section 5.2.4). We vary datasets and metrics. Our conclusion is that Dominant feature dominates in all settings, and Section 5.2.4 and fig. 11 explain why.
F.1 Wasserstein distances on additional datasets
We present detailed results Tris dataset and three additional ones larger than Tris (Tables to 11). We still perform three train/test splits and report averaged results. The Dominant feature aggregation scheme shows in any case the best performance on generated samples. For other methods, increasing K does not significantly improve generation quality.
Table 8: Wasserstein distance Wtest for different aggregation methods as function of the number of trees K. We perform three different train/test splits for each experience.
Wiest Iris dataset Dominant
Number of trees
Arithmetic
Geometric
Median
Alternating
0.96
0.96
0.96
0.96
0.96
25
0.94
0.94
0.50
0.98
0.94
50
0.94
0.94
0.49
0.98
0.96
100
0.94
0.94
0.46
0.99
0.95
500
0.91
0.94
0.16
0.99
0.95
1000
0.91
0.94
0.46
0.99
0.96
VVluee reported by Jolicoeur-Martinean al. (2024).
36

---

Published in Transactions on Machine Learning Research (02/2026)
3.261
Mean NLL
NLL bits/dimm
3.256
3.251
3.246
2
5
Ensemble size K
Figure 9: Mean NLL computed on CIFAR-10 validation set, with the ensemble size K in the x-axis. The uncertainty bands are computed exactly in the same way as FID (Appendix A).
N=10
N=10
N=25
N=25
N-50
N-50
N-100
0005
N-100
FID
A024
3.0%
Ensemble size
Ensemble size
Figure 10: Effect of the number of diffusion steps at inference on perceptual quality FFHQ for different ensemble sizes K. Both FID and KID remain nearly constant with respect to K, showing that ensembling does not mitigate discretization errors.
F.2 Coverage and efficiency
We measure coverage and efficiency (see Appendix for short descriptions) on the Iris dataset. We use the same range for number of trees as above, and aggregation schemes are also unchanged. Dominant feature performs in comparison other methods in terms of diversity (Table 12) and in post-generation classification (Table 13). This result aligns with Table 8 and gives further credence to it.
Table 12: Coverage score coutest for different aggregation methods as function of the number of trees K.
CDUcest on Iris dataset
Number treus K
Arithmetic
Geometric
Dominant
Median
Random select
0.31
0.31
0.31
0.31
0.31
25
0.35
0.32
0.81
0.27
0.36
50
0.35
0.31
0.81
0.27
0.32
100
0.34
0.31
0.82
0.28
0.29
500
0.35
0.35
0.83
0.30
0.33
1000
0.34
0.35
0.85
0.29
0.31
37

---

Published in Transactions on Machine Learning Research (02/2026)
Table 9: Wasserstein distance West for different aggregation methods as a function of the number of trees K the Airfoil Self Noise dataset (n 1503, d 5, Brooks al., 1989).
test on Airfoil Self Noise dataset
Number of trees K
Arithmetic
Geometric
Dominant
Median
Alternating
1
0.88
0.88
0.88
0.88
0.88
25
0.88
0.89
0.58
0.92
0.88
50
0.88
0.89
0.55
0.92
0.88
100
0.88
0.89
0.54
0.92
0.88
500
0.88
0.89
0.53
0.92
0.88
1000
0.88
0.89
0.52
0.92
0.88
Table 10: Wasserstein distance Wtest for different aggregation methods as function of the number of trees K the Wine Quality White dataset (n 4898, d 11, Cortez et al., 2009).
Wtest Geometric
Wine Quality White dataset Dominant
Number trees K
Arithmetic
Median
Alternating
1
3.71
3.71
3.71
3.71
3.71
25
3.69
3.90
2.69
3.60
3.71
50
3.70
3.91
2.62
3.59
3.71
100
3.69
3.90
2.55
3.59
3.71
500
3.69
3.90
2.43
3.58
3.70
1000
3.69
3.90
2.38
3.58
3.70
Table 13: Fl score Fliest for different aggregation methods as a function of the mumber of trees
Fleest on Iris dataset
Number of trees K
Arithmetic
Geometric
Dominant
Median
Random select
0.75
0.75
0.75
0.75
0.75
25
0.78
0.79
0.87
0.76
0.79
50
0.78
0.80
0.86
0.78
0.73
100
0.78
0.78
0.88
0.77
0.75
500
0.77
0.78
0.90
0.77
0.76
1000
0.79
0.78
0.88
0.77
0.74
38

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Wasserstein distance Weett for different aggregation methods as a function of the number of trees K on the Wine Quality Red dataset 1599, 10, Cortez al. (2009)).
West on Wine Quality Red dataset Dominant
Number of trees K
Arithmetic
Geometric
Median
Alternating sampling
3.23
3.23
3.23
3.23
3.23
25
3.23
3.39
2.18
3.11
3.25
50
3.23
3.39
2.09
3.10
3.25
100
3.23
3.39
2.01
3.10
3.25
500
3.23
3.38
1.85
3.09
3.25
1000
3.23
3.38
1.80
3.09
3.25
F.3
Noise underestimation
RF std (Arithmetic mean)
RF std (Dominant feature)
Standard Deviation
XGBoost std
0.6
10
30
Generation Step
Figure 11: We compare the overall standard deviations the two-dimensional predicted scores on the Iris datasot. Specifically, we evaluate the Arithmetic Mcan and Dominant Component methods against XGBoost, which is the best-performing score model according to Jolicoeur-Martiinn Martineau al. (2024). To ensure a fair underfitting analysis, all models were trained with the same maximum tree depth of 7, which is sufficient for this dataset. The Dominant Component method yields higher predicted noise magnitudes than the Arithmetic Mean, closely matching XGBoost's s standard deviation at the final steps.
G Weight scaling at initialization
To the best of our knowledge, prior work has examined the benefits of initializing network with weights scaled close to zero. Yet, will demonstrate, this choice positively influences both training be- havior and final performance. In this section, review related methods and illustrate how this inilialization affects convergence on LDDSM and FID.
G.1 Near-zero output initialization score models
In these architectures, whether the network predicts the score (Song et al,, 2021b) or the noise (Ho et al., 2020: Nichol & Dhariwal, 2021), the final layer's weights are by design initialized near zero. For example, Nichol & Dhariwal (2021) enforce this via their zero module implementation on the final convolutional layer, while Song et al. (2021b) apply Xavier uniform initialization (Glorot & Bengio, 2010) with a very
39

---

Published in Transactions on Machine Learning Research (02/2026)
1.8
RF std (Arithmetic mean)
RF std (Dominant feature)
Standard Deviation
XGBoost std
1.4
1.0
0.6
0
20
40
60
80
100
Generation Step
Figure 12: We compare the overall standard deviations on more diffusion steps (100). Increasing the number of diffusion steps does not allow Random Forest to reach the score estimation quality of the stronger models.
small scale parameter (e.g., 10-10), effectively constraining the network's output to be a zero tensor at the start of training. In this experiment we upscale À to one.
G.2 Related works on general contexts
ControlNet (Zhang et al., 2023) almost applies this principle but in the context of fine-tuning, by initializing newly added convolutional layers to zero allowing the model to incorporate conditioning without immediately altering the behavior of the pre-trained diffusion backbone. similar effect has been studied in the context of deep residual networks, where De Smith (2020) show that Batch Normalization facilitates training by pushing residual blocks loward the identity function. However, subsequent. work demonstrated that BatchNorm is not strictly necessary, as simple modification the initialization (such as setting the residual branch to zero at the start) suffices to achieve similar trainability (Zhang et al., 2019). This suggests that initializing the final layer near zero in diffusion models may play comparable role, ensuring smooth early training dynamics without requiring explicit normalization layers.
G.3 Effect of weight scaling training
We verify how scaling down helps stabilizing training using the DDPM+ model on CIFAR-10 10 (same settings as Section 5).
40

---

Published in Transactions on Machine Learning Research (02/2026)
scale
scale 1
scale le-1
scale 1e-1
Valid Lposit
scale = 1e-2
Valid LODsM
scale 1e-2
1.25
scale 1e-3
scale 1e-3
scale 1e-10
scale 1e-10
10-
0.10
4020
5002
9200
20023
Training Steps
Training Steps
(a) Losses at normal scale.
(b) Losses in log scale.
Figure 13: LDDSM evaluated on the validation set of CIFAR-10 over the course of training up to 200k iterations. We evaluate different scales À (10 for E 2,,1, 10}). We observe that compared to using scale of À = 1, reducing the scale below one brings the average loss closer to 1 at the first step and results in faster convergence during the early stages of training. Despite this, the losses quickly stabilize around similar values regardless of the initial scale.
G.4 Effect of weight scaling FID
We show the positive effect of near-zero scaling of initialization from the DDPM+ architecture sample quality (same settings as Section 5).
228
21.70
Model
220
scale 1e-10
scale 1
175
15.0
12.5
1155
10.0
8.8
0.42
7.5
7.79
6,27
5,86
5.0
5.88
5288
5.29
4.86
4.83
500
20000
6009
60000
R000
18000
200000
Training Steps
Figure 14: FID-10k on CIFAR-10 evaluated over the course of training on the two extreme cases of X up to 200k iterations. While both models improve their FID-10k initialized with scale of 10-10 consistently outperforms the one with scale 1, which fails to reach the same level of performance before 200k steps (the gap consistently stays above 0.2).
over the course of training, the model
G.5 Effect of weight scaling on predictive diversity
We show Figure 4 that increasing the weight scale at initialization enhances post-training diversity.
We measure the predictive (or functional) diversity, which is the diversity in the output space. Since each model is trained by minimizing the MSE of scores noises depending the framework, measure the predictive diversity by adopting the metric associated to the squared loss and arithmetic mean combiner from Wood et al. (2023). Given K models (!1) :....88
(K) taking data point and timestep I as input and
41

---

Published in Transactions on Machine Learning Research (02/2026)
producing output in the same space x, we define predictive diversity as the variance of the predictions and write it as
(s()
(1)
5g ) (x.))
(8) (x.
(K) 8g (xx t))²
(x)²
(98)
K
(b) (x t) is the combination of the predictors, and the average of squared where $ (x, - K E $
differences is calculated per image rather than per pixel.
H Note on FID and related metrics
Automated perceptual metries remain standard and widely used tools for comparing generative models. They provide an essential, reproducible baseline for evaluation, and are often indispensable when large-scale human studies are impractical or unavailable. However, typically we take FID which the most popular one, it frequently used without a thorough assessment of its limitations and its relevance as a proxy for visual quality has been repeatedly questioned in the literature (Stein et al., 2023; Jayasumana al., 2024: Karras et al., 2020; Borji, 2022; Morozov et al., 2021). In particular, Stein et al. (2023) highlight cases where FID (and network reature-based metrics in general) correlates poorly with human judgment, notably showing that FID underestimates the quality of diffusion models on FFHQ compared to human perception. Another point is that FID, IS and KID rely on features extracted from Inception networks pre-trained on ImageNet. Yet several works showed limits of their use on datasets that differ significantly from ImageNet, such as human faces (Borji, 2022; Kynkäänniemi et al., 2023). For instance, Kynkäänniemi et al. (2023) show that FID sensitive to the alignment between the generated and ImageNet class distributions, which can result in misleading evaluations particularly on dalasets like FFHQ, where the label distribution deviates substantially from that of ImageNet.
42


Reviews:
[{"review_id": "kfiQNByiN1", "reviewer": "Reviewer_9hb2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript studies whether ensembling improves generation in unconditional score-based diffusion models. The authors report small but consistent gains in score-matching loss and likelihood, yet marginal to no improvements on perceptual metrics (FID/KID) with Deep Ensemble models on image datasets. They also provide theoretical analysis to provide insights as to why averaging scores reduces the DDSM objective and motivate why ensembling for score estimation is challenging. These findings are complimented with several practical insights and benchmarks on CIFAR-10 and FFHQ, as well as some tabular datasets such as Iris and Wine Color which involve Random Forest models for score estimation.", "claims_and_evidence_yn": "No", "claims_explanation": "The core findings on score loss are accurate and clearly shown: averaging scores can lower the loss and improve likelihood as mathematically proved by the authors. Yet, the generality implied in the abstract that ensembling yields marginal to no improvement for diffusion models is not convincingly established beyond specific choices.\n\nIn particular, the class of Deep Ensemble models considered in this work shows major limitations that are not mentioned nor explored by the authors. Then, while the lack of data types (mostly images) prevents generalizability across domains, the evaluation design remains often unsatisfactory due to its lack of proper statistical validation through multiple runs. Finally, the paper itself shows a counter-argument to its major claim on tabular data, indicating that \"Dominant\" aggregation for Random Forest improves score estimation and substantially helps, ultimately limiting the scope of the original takeaway to Deep Ensemble methods (with same architecture) on image datasets at most.", "audience_yn": "Yes", "audience_explanation": "Ensembling is a key tool to improve ML performances in several areas of research. Investigating its effects in the context of diffusion models bears interest for the ML community in general, both from theoretical and practical perspective.", "requested_changes": "**Major weaknesses:**\n- Empirical evaluation is dominated by image datasets and score-averaging baselines of Deep Ensemble (DE) models with the same architecture. Two tabular experiments use Random Forest-based scores, without intersecting with deep ensembling, despite different behaviors being observed (no gain in the DE ensembling, while RF shows Domination ensembling gains). As a result, conclusions are mostly retained for unconditional image generation with specific DE methods using score aggregation. The scope of the contribution needs to be backed-up by further experiments, or narrowed down to a precise set of practices.\n- Ensemble variants largely keep architecture and weights fixed across members (independent U-Nets with identical design; MC-Dropout; class-subset specialization; weight-scale changes). Heterogeneous ensembles like mixed architectures (U-Net, CNN, etc.) are not explored and could affect conclusions. \n- Claims about lack of perceptual gains should be carefully scoped: Mixture-of-Experts outperforms the best individual model on FFHQ in Table~1, and on tabular data the Dominant aggregation improves Wasserstein distance. These are notable exceptions that the narrative could emphasize to avoid an overly broad takeaway.  \n- General claims relying on experiments need more grounding asserted by statistical significance (see specific Questions below). The limitations of these claims is often not considered, ultimately hindering the overall soundness.\n- The authors do not provide a repository to reproduce their experiments, impairing proper verification of their practical findings.\n\n**Questions and comments:**\n- **p.2, Figure 1 (\"Our observations show that an ensemble of five models does not achieve the highest image quality even though it beats the average one\"):** Figure 1 contrasts strongest, weakest, and the ensemble, but not an \"average\" model in the FID-10k sense. Providing an illustration for the median/mean-FID model model would help supporting the claim. Besides, adding the distribution of the FID-10k scores across the ensemble method in Appendix would help visualize the discrepancies between the models of the ensemble by highlighting possible outliers.\n- **p.3: \"[Deep ensemble] involves constructing K models, each sharing the same neural network architecture [...] A key ingredient for its success is the diversity between the individual models\":** The authors state Deep Ensemble members share the same architecture and diversity weights \"naturally arises\" from stochasticity of the learning process; later they cite Xu et al. (2024) noting convergence to similar solutions in diffusion models sharing the same architecture, contradicting their claim p.5: \"we could expect models trained on different seeds to converge to noticeably different solutions\". It seems mandatory to reconcile these points to substantiate the claim, either by testing or evaluating diversity (as in Appendix G.5) from learned weighting across members. Adding heterogeneous architectures to compare the diversity against same parameterization ensembles would be interesting as well. Alternatively the claim of diversity from the depicted ensembles should be tempered and the scope of the contribution re-assessed.\n- **Equation (6) and guidance equivalence (p. 6–8):** The manuscript rightly notes that Eq. (6) and related equalities do not always hold in the context of diffusion models, then mentions MCMC/Feynman–Kac correctors. Appendix C.4 also mentions these approaches p.27. Yet, it is unclear whether these correctors were used in the experiments to properly sample from the diffusion. Please clarify whether any correctors were used in experiments; if not, the scope of the contribution should be specified. \n- **Figure 3 presentation (p. 8):** The caption is not informative, axis labeling is hard to connect across subplots, and uncertainty depiction is inconsistent (only LDDSM shows \"$\\pm$ Std Dev\"). Standardizing axis labels/legends, defining the uncertainty bands, and harmonizing the $K$ range across (a)–(e) would improve readability. Note that (e) explores up to $K=10$ while others stop earlier, even though interpretations would benefit from more points, why is that?\n- **Table 1 caption (p. 9):** The caption is incomplete as some elements of the table are not described. Clarify which entries include standard deviations (parentheses appear for individual models but not all ensembles, probably because the experiment was not repeated to assess variability). Reporting variability and the number of training runs for all entries would improve substantiate the claim. \n- **About weight scaling with $\\lambda$: \"This suggests that increasing predictive diversity alone is insufficient to enhance ensemble performance in diffusion models.\" (p. 10):** Given Figure 11 provides the core analysis of this claim, the authors should consider bringing it into the body of the manuscript. Besides, the authors are encouraged to discuss possible trade-offs  and limitations of this claim (e.g., when does diversity help/harm). \n- **\"We observe that ensembles built from weaker models do yield bigger improvements in FID, while gains diminish and even turn negative as the base models become stronger\" (p. 11):** Due to the lack of repetitions to estimate $\\Delta$ at each iteration, Figure 4 might be insufficient in its current state to support this statement. It would be mandatory to assess the variability of $\\Delta$ to substantiate the claim and assert a pattern. Similar results with other values of $K$ would be expected to show a convincing trend in $\\Delta$. Also, the definition of $\\Delta$ in the Caption of Figure 4 should be provided out of the descriptor (a) as it is also used in (b), and plotted confidence regions need to be defined.\n- **Figure 5 (p. 11):** The authors mention a reference models from the study of Jolicoeur-Martineau et al. (2024) in the context of this experiment. Adding the reference performance onto Figure 5 would help situating the improvements made by the Dominant aggregation. Also, adding explicit $x$/$y$ labels would improve readability.\n- **Figure 8 interpretation (p. 35):** The figure shows standard deviations of predicted scores (Arithmetic vs. Dominant vs. XGBoost). Please clarify the relationship between under/overestimation of noise magnitude and Wasserstein performance, and whether increasing the amount of denoising steps or alternative schedulers would change the conclusion in the underestimation regime.\n- **\"It may be caused by an underfitting effect coming from decision trees\" (p. 11):** This claim requires further supporting evidence, as it does not align with typical behaviors of Random Forest. Appendix A.2 does not list tree depths, which can be relating to under/overfitting in some instances. \n- **Figure 6 and \"the disconnect between what diffusion models are trained to optimize (score matching) and how they are typically evaluated explains why ensembling struggles to diminish image quality metrics.\" (p. 12):** The authors are encouraged to add confidence intervals over multiple training seeds and a  $\\log y$-scale for readability where appropriate. The sensitivity plots are central to the argument that FID reacts chaotically to small score perturbations; thus additional repetitions would strengthen this point. The validity of the aforementioned claim appears shaky without mathematical consolidations or improved benchmarks.\n- **Sampling procedure missing (Appendix A):** It would help to consolidate sampling details: solver (SDE/ODE or DDIM), step counts, $\\eta$ or SNR schedules, guidance/temperature (if any), usage of correctors, and seeds for each experiment. Some of this is dispersed in the preliminaries but not fully specified per experiment.\n- **Appendix C.4 (proof clarity):** This counterexample is a key result of the manuscript. The proof would read more smoothly with proper declaration of fixed quantities and a lemma-first structure for intermediate steps. Some notational tightenings would also be appreciated to make the proof more rigorous.\n- **Appendix C.4 scope/title:** This appendix also includes remarks on concept-conditional modeling and linear inverse problems. The authors should consider retitling or splitting into subsections so readers can locate these topics easily. Also, it is unclear why linear inverse problems are mentioned. The section should either be more clearly involved in the manuscript or removed. \n- **Metric formatting in tables:** Reporting KID and LDDSM with a consistent scientific-notation factor (e.g., $\\times 10^k$) would improve readability across tables.\n- **Appendix E.2, Figure 7:** The uncertainty band dominates the plot, making trends hard to interpret. The authors should consider adjusting scale, adding more runs, or plotting relative changes to improve readability. As is, the figure bears limited utility.\n- **Cited work:** Several citations are preprints. Where possible, the authors should prefer the latest peer-reviewed versions and consider replacing some instances with recently published work on ensemble guidance/density composition.", "broader_impact_concerns": "Not applicable.", "additional_comments": "Not applicable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gjWtibw1OK", "reviewer": "Reviewer_wo7R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the effect of the ensembling techniques on the diffusion models. Experiments on different tasks and models show that it improved the score matching objective but does not help the perception metrics (e.g. FID score).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Experiments on different dataset and models demonstrate the effect of ensembling on the model performance. Also different aggregation methods are explored. However, I have questions/comments about some of the contributions stated in the introduction:\n\n1. ”We also provide theoretical insights on its [ensembling]  effect on the training objective error”. I would not call this a theoretical insight as it directly follows from the definitions. \n2.  “[We] contextualize ensembling within other contexts”. I do not really understand what is meant by this. I would be happy to hear the authors’ clarifications.\n3. “addressing the mismatch between the score matching loss and perceptual image quality metrics”. From this formulation it seems that the authors address the problem of the mismatch between the loss and the metric, which is not the case. The mismatch is indeed empirically demonstrated, but I would say that this is a widely known fact for diffusion models. Also, not obvious, why addressing the mismatch would help the ensembling.", "audience_yn": "No", "audience_explanation": "I think that the community, especially the practitioners, would be interested to see how useful (or not) the ensembling techniques are for diffusion models. \nHowever, I think for the paper to be insightful, a more diffusion-specific experiments are required. For example:\n1. Do we need to apply ensembling to all the timestamps? Or will it have a different effect? E.g. closer to the “data space” the model is generating high-frequency components that are important for FID score. Maybe averaging is harmful there but can help on the earlier stages\n2. Can we reduce the amount of sampling steps but keep the performance comparable using ensembling? That is, can the averaging smooth out the larger discretization error when we use fewer diffusion steps?", "requested_changes": "1. Please clarify / correct the contributions\n2. Please add a more insightful experiment to demonstrate why DE do not work that well or how it can be useful specifically for diffusion models (see specific suggestions above)", "broader_impact_concerns": "NA", "additional_comments": "Minor corrections:\n1. Figure 3 (CIFAR): Why loss have the result for up to 10 components, but FID and KID for up to 5\n2. Typo (Section 5.3.1): “the optimized objective differs from **te** one of the perceptual metric of interest”", "extra_scores": {}, "extra_text": {}}, {"review_id": "zlg1AeP6Dv", "reviewer": "Reviewer_22sG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive investigation into the effectiveness of ensembling techniques for unconditional score-based diffusion models. The authors explore whether combining multiple models provides similar benefits in the context of generative modeling.\nFrom my point of view, the primary contributions are:\n\nEmpirical Evaluation: The authors conduct extensive experiments across multiple datasets (CIFAR-10, FFHQ, tabular data), model architectures (U-Nets, Random Forests), and ensemble methods (Deep Ensembles, MC Dropout). They test a variety of aggregation rules for combining model outputs, including arithmetic mean, median, and more heuristic approaches.\n\nKey Finding of a Disconnect: The central finding is that while ensembling consistently reduces the score-matching loss (L \nDDSM) and improves model likelihood, it fails to reliably improve perceptual quality metrics like Fréchet Inception Distance (FID) or Kernel Inception Distance (KID). The ensemble often performs better than the average individual model but rarely surpasses the single best-performing model in the ensemble.\n\nTheoretical Insights: The paper provides theoretical justification for some of the empirical results. It formally shows why averaging scores reduces the expected L_{DDSM} loss. Crucially, it also demonstrates with a clear counterexample that the operations of diffusion (adding noise) and model composition (e.g., via a Product-of-Experts) do not commute (Proposition 4.2). This clarifies a common heuristic used in model guidance and composition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Claim**: Ensembling improves score loss but not perceptual quality. This is the paper's core claim and is strongly supported. Figure 3e clearly shows a monotonic decrease in the $L _{DDSM}$ loss as the ensemble size (K) increases for CIFAR-10. In contrast, Figures 3a-d and the visual example in Figure 1 show that FID and KID do not consistently improve and that the ensemble fails to beat the strongest individual model. Table 1 provides further quantitative evidence, where on FFHQ, the \"Arithmetic mean\" ensemble achieves an FID of 23.44, which is better than the average individual model (24.20) but worse than the best individual model (21.7).\n\n**Claim**: Alternative ensemble strategies offer limited improvement. The authors comprehensively test numerous aggregation rules in Table 1. Most methods that aggregate scores at each step (mean, median, geometric mean) show marginal or no improvement over the simple average. The \"Mixture of experts\" strategy does show a notable improvement on FFHQ , but the authors correctly identify that this is not a true ensemble in the traditional sense, as no aggregation occurs within the generation of a single sample. Their investigation into MC Dropout (Table 5) and diversity-promoting strategies (Tables 2 and 3)  further strengthens the conclusion that simple ensembling is not a silver bullet.\n\n**Claim**: The failure is due to a disconnect between the training objective and perceptual metrics. This explanation is well-argued and supported. Figure 6 provides compelling evidence: Figures 6a and 6b show that during a single model's training, the validation $L_DDSM$\n​stagnates while FID continues to improve, demonstrating their different behaviors. Furthermore, Figures 6c and 6d brilliantly illustrate that small perturbations added to the score function cause large, chaotic swings in FID and KID, highlighting the sensitivity of these perceptual metrics in a way that the smooth, averaged $L_2$ score loss is not.\n\nTheoretical Claims: The proofs provided in the appendix are clear and appear correct. Proposition 4.1 is a straightforward but important application of Jensen's inequality to the diffusion context. Proposition 4.2, which proves the non-commutativity of diffusion and Product-of-Experts with a simple Gaussian case, is a valuable and elegant theoretical contribution that formalizes a subtlety often overlooked in guidance-based methods", "audience_yn": "Yes", "audience_explanation": "It is a nice paper, and I think that there will be an audience of TMLR interested in it.\n\nRelevance to Diffusion Model Practitioners: Diffusion models are a central topic in modern machine learning. Ensembling is a fundamental and widely used technique. This paper addresses the natural and important question of whether these two concepts can be combined effectively. The largely negative result is itself a significant contribution, as it can save other researchers significant time and effort by showing that the most straightforward application of ensembling is not fruitful for improving sample quality.\n\nImplications for Generative Model Evaluation: The paper's investigation into why ensembling fails highlights the critical disconnect between common training objectives (like score matching) and popular perceptual metrics (like FID). This is a broader issue relevant to the entire field of generative modeling. It encourages the community to think more deeply about how we train and evaluate these models, a topic of ongoing research and debate.\n\nTheoretical Value: The theoretical analysis of score composition, particularly the proof that diffusion and PoE do not commute, has implications beyond ensembling. It provides a formal basis for understanding why techniques like classifier-free guidance are heuristics rather than exact methods for sampling from a target product distribution. This will be of interest to researchers working on the theory and foundations of diffusion models", "requested_changes": "**Emphasize the \"Mixture of Experts\" Finding**: In Table 1, the \"Mixture of experts\" method is the only one that clearly outperforms the best single model on the high-resolution FFHQ dataset (FID 20.36 vs. 21.7). While you correctly note this isn't a \"true\" ensemble because scores aren't averaged within a sample's generation, this is still a very interesting result. It implies that the diversity captured by independently trained models is valuable, but the act of averaging their scores is what is detrimental to perceptual quality. This point could be highlighted more strongly in the discussion and conclusion. It suggests that future work should focus on methods that can leverage inter-model diversity without the \"blurring\" effect that simple averaging seems to cause.", "broader_impact_concerns": "No.", "additional_comments": "No.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)