EnsIR: An Ensemble Algorithm for Image Restoration via Gaussian Mixture Models
Authors: Shangquan Sun
Wenqi Ren
Zikun Liu
Hyunhee Park
Rui Wang
Xiaochun Cao
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Ensemble
Image Restoration
Gaussian Mixture Models
Expectation Maximization
Primary area: machine_vision
TL;DR: In this work, a training-free ensemble algorithm is developed to boost the performance of image restoration by combining multiple pre-trained base models at the inference stage.

Abstract: Image restoration has experienced significant advancements due to the development of deep learning. Nevertheless, it encounters challenges related to ill-posed problems, resulting in deviations between single model predictions and ground-truths. Ensemble learning, as a powerful machine learning technique, aims to address these deviations by combining the predictions of multiple base models. Most existing works adopt ensemble learning during the design of restoration models, while only limited research focuses on the inference-stage ensemble of pre-trained restoration models. Regression-based methods fail to enable efficient inference, leading researchers in academia and industry to prefer averaging as their choice for post-training ensemble. To address this, we reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. We estimate the range-wise ensemble weights on a reference set and store them in a lookup table (LUT) for efficient ensemble inference on the test set. Our algorithm is model-agnostic and training-free, allowing seamless integration and enhancement of various pre-trained image restoration models. It consistently outperforms regression-based methods and averaging ensemble approaches on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. The codes and all estimated weights have been released in Github.

EnsIR: An Ensemble Algorithm for Image Restoration via Gaussian Mixture Models
Shangquan Sun Hyunhee Park
Wenqi 3.4 3.4
Zikun Liu
Rui Wang 1.2 Xiaochun Cao
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China "ccooo of Cyber Security, University of Chinese Academy of Sciences, Beijing, China Schooo of Cyber Science and Technology. Shenzhen Campus of Sun Yat-sen University
Guangdong Provincial Key Laboratory of Information Security Technology "Samsung Research China Beijing (SRC-B) Caamra Innovation Group, Samsung Electronics
(sunshangquan, wangrui)@@it ac. cn {zikun. 1iu, inextg....... (arkk)aammauuu {renwq3, caxxiacchuijammma.... sysu. edu. cn
Abstract
Image restoration has experienced significant advancements due the development deep learning. Nevertheless, it encounters challenges related to ill-posed prob- lems, resulting in deviations between single model predictions and ground-truths Ensemble learning, as a powerful machine learning technique, aims to address these deviations by combining the predictions of multiple base models. Most existing works adopt ensemble learning during the design of restoration models, while only limited research focuses the stfeee ensemble of pre-trained restoration models. Regression-based methods fail to enable efficient inference, leading researchers in academia and industry prefer averaging as their choice for post-training ensemble. To address this, we reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an ex- pectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. We estimate the range- wise ensemble weights on a reference set and store them in a lookup table (LUT) for efficient ensemble inference on the test set. Our algorithm is -agostic and training-free. allow- ing seamless integration and enhancement of various pre-trained image restoration models. consistently outperforms regression- based methods and averaging en- semble approaches on 14 benchmarks across image restoration tasks, including super-resolution, deblurring and deraining. The codes and all estimated weights have been released in Github.
1 Introduction
Image restoration has witnessed significant progress the decades, especially with the advent of deep learning. Numerous architectures have been developed to solve the problem of restoration, including convolutional neural networks (CNNs) [10, 82], vision Transformers (ViTs) [39, 81, 93] and recently, vision Mambas [27]. However, single models with different architectures or random initialization states exhibit prediction deviations from ground-truths, resulting in sub-optimal restoration results.
*Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To alleviate this problem, ensemble learning, a traditional but influential machine learning technique, has been applied to image restoration. It involves combining several base models to obtain better result in terms of generalization and robustness [18, 20, 26, 49, 57J. However, most ensemble methods in image restoration focus on training-stage ensemble requiring the ensemble strategy to be determined while training multiple models, thus sacrificing flexibility of changing models and convenience for plug-and-play usage [31, 40, 44, 46, 54, 59, 70, 76]. In contrast, there a demand for advanced post-training ensemble methods in the image restoration industry, where researchers still prefer averaging as their primary choice [1. 15, 41, 52, 66, 68, 92].
Despite the industrial demand, post-training ensemble in image restoration is challenging for tradition ensemble algorithms originally designed for classification or regression. Unlike classification and regression, image restoration predictions are matrices with each pixel is correlated with others and range from 0 to 255. As a result, traditional methods like bagging [6] and boosting [28] either require enormous computational resources for the restoration task or fail to generalize well due to the imbalance between candidate number and feature dimension. As an alternative, Jiang er al. propose post-training ensemble algorithm for super-resolution by optimizing a maximum posteriori problem with a reconstruction constraint [3 I]. However, this constraint requires an explicit expression of the degradation process, which is extremely difficult to define for other restoration tasks beyond super-resolution. It also necessitates prior knowledge of the base models' performance, further limiting its practical application. These issues with both traditional and recent ensemble methods lead researchers in image restoration to prefer weighted averaging as their primary ensemble approach [1, 15,52, 52, 68, 92].
To this end, we formulate the ensemble of restoration models using Gaussian mixture models (GMMs), where ensemble weights can be efficiently learned via the expectation maximization (EM) algorithm and stored in a lookup table (LUT) for subsequent inference. Specifically, we first base on the Gaussian prior that assumes the prediction error of each model follows a multivariate Gaussian distribution. Based on the prior, the predictions multiple samples can be appended into a single variable following Gaussian. By partitioning pixels into various histogram-liks bins based on their values, we can convert the problem of ensemble weight estimation into various solvable univariate GMMs. As result, the univariate GMMs can be solved to obtain range- wise ensemble weights by the EM algorithm, with the means and variances of Gaussian components estimated based on the observation priors. We estimate these weights inference on the test set. Our method does not require training or prior knowledge of the base models and degradation processes, making it applicable to various image restoration tasks.
reference and store them in LUT for efficient
Our contributions mainly lie in three areas:
Based on the Gaussian prior, we partition the pixels of model predictions into range-wise bin sets of mutually exclusive ranges and derive the ensemble of multi-model predictions into weight estimation of various solvable univariate GMMs.
To solve the univariate GMMs and estimate ensemble weights, we leverage the EM algorithm with means and variances initialized by observed prior knowledge. We construct LUT to store the range-wise ensemble weights for efficient ensemble inference.
Our ensemble algorithm does not require extra training or knowledge of the base models. It outperforms existing post-training ensemble methods on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining.
2 Related Works
Ensemble Methods. Ensemble methods refer to approaches that fuse the predictions of multiple base models to achieve better results than any individual model [20]. Traditional ensemble methods include bagging [6]. boosting [28], random forests [7], gradient boosting [24], histogram gradient boosting [14, 34], etc. These methods have been applied to various fields in classification and
regression, such as biomedical technology [77, 78], intelligent transportation [53, 84], and pattern recognition [61, 91].
Image restoration. Image restoration, as a thriving area of computer vision, has been making significant progress since the advent of deep learning [10, 65]. Various model architectures have been

---

proposed to address image restoration tasks, such as convolutional neural networks (CNNs) [19, 56, 73, 90], multilayer perceptron (MLPs) [67]. vision Transformers (ViTs) [39, 62, 81], etc. Additionally, various structural designs like multi-scale [56], multi-patch [64, 89], and progressive learning [82] have been adopted improve the representative capacity. It is known that CNNs excel encoding local features, while ViT's are adept at capturing long range dependencies. Despite this significant progress, single models still generate predictions that deviate from ground-truths, leading researchers in industry to adopt multi-model ensembles to achieve better performance [15, 52, 68, 92].
Ensemble Learning in Image Restoration. Some works incorporate ensemble learning into image restoration by training multiple networks simultaneously and involving ensemble strategy during the training process [2, 9, 11-13, 16, 17,21, 21, 29, 33, 35-38, 40, 42-46, 54, 55, 59, 63, 69, 70, 72, 75, 76, 79, 80, 85, 87]. However, the majority of them require additional training or even training from scratch alongside ensemble. Only a few focus on ensemble at the post-training stage [31, 66]. Among them, self-ensemble [66] geometrically augments an input image, obtains super-resolution predictions of augmented candidates and applies averaging ensemble to the candidates, which is orthogonal to the scope of our work. RefESR [31] requires reconstruction objective which must get access to the degradation function, prohibiting its application to tasks other than ruper-resolution. In general, limited works focus on the training-free ensemble of image restoration. There lacks general ensemble algorithm for restoration despite industry demand [1, 15, 52, 68, 92].
3 Proposed Ensemble Method for Image Restoration
In Sec. 3.1, we first present the formulation of the ensemble problem in image restoration. We then formulate our ensemble method in the format of Gaussian mixture models (GMMs) over partitioned range-wise bin sets in Sec. 3.2. We derive the expectation maximization (EM) algorithm with known mean and variance prior knowledge to solve the GMMs problems in Sec. 3.3
3.1 Ensemble Formulation of Image Restoration
Given a test set T = {X,,, Y with numerous pairs of input images and ground-truths, suppose we have M pre-trained base models for image restoration, f1,..,, Sas. For model fm where m {l,.., ..., .M its prediction is denoted by Xm.n = fm (X,,) for abbreviation. We consider the ensemble of the predictions as a weighted averaging, i.c.,
Y, =B {
X XM Vn
(1)
where Y, is the ensemble result, and Bn, E R2 is the vector weight parameters for the ensemble. The widely-used averaging ensemble strategy in image restoration assigns equal weights for all samples and pixels, i.e., Bn = 1 assigns weights inversely proportional to the mean squared error between the predictions and their average [92]. However, they adopt globally constant weights for all pixels and samples, neglecting that the performances of base models may fluctuate for different patterns and samples.
AT recent method in the NTIRE 2023 competition
Alternatively, we start from the prospective of GMMs and assign range-specific weights based on the EM algorithm.
3.2 Restoration Ensemble as Gaussian Mixture Models
Similar to RefESR [31], suppose we have a reference set DD = (,,, n. YN with N pairs of input images and ground-truths. We assume the reference set and test set are sampled from the same data distribution, i.e., D, T ~ DD
For each model fm. its prediction is denoted by Xm,, fm (X,)) 3 We use Xm, Yn R- to represent the flattened vector of the matrices X Y,.. n. where L = 3/W. Based Gaussian prior, it can be assumed that the estimation error of a model on an image follows zero-mean Gaussian distribution, namely Cm,, yn Xm,n ~ (0, Em,.), where Σm,, E Lx is the covariance matrix of the Gaussian. Then the observed ground-truth -tunt can be considered following a multivariate Gaussian with the mean equal to the prediction, i.e., yn Jm, x,, ~ NX Em,,).
We can consider the ensemble problem as the weighted averaging of Gaussian variables and estimate the weights by solving its maximum likelihood estimation. However, solving the sample-wise mixture
3

---

Algorithm 1: EnsIR: an ensemble algorithm for image restoration
Input: A small reference dataset (x for ensemble weight estimation, test set {X,,, M pre-traiee pre-trained models fi...., fas. bin width b, Empty lookup table LUT
Output: Ensemble result (Y,)
Estimation Stage:
Obtain restoration predictions by Xm.n flatten (fm (Xm)) Vm E {I,..,,M} M
2 Append restoration predictions and ground-truths into y YI.N and Xm,1:N based on Eq, 2
Define bin set space B { b). [b. 26), [(T 1)b, 255]]:
for each hin set (B..... BM)) E B do
Compute the partition map R, = I IRn (fm (xn)) Partition images and obtain range- wise patches (y,,1:N, X,,1,1:,, X,,MM::.) by Eq. 3; (1 Or,M) ) <- (P Xr,,,I:N, M,,N
Store BM.) + (Or,1, Or,M);
9 end
Inference Stage:
for each test data Xn, do for each bin set (BM)...BM) E B. do Retrieve (ar,11 ar,M) f LUT ......a)]]l
12
Partition input as X X,m,n x- R. fm (Xn), where R, II Isn (fm (Xn)):
14
Yr, + Em Or Or Nr.m.rmmnn
/* Inner sunmation of Eq. 15 */
15
end
Y + ITH Yin
16
/* Outer sunmation of Eq. 15 */
end
of Gaussian is not feasible because the covariance matrices are sample-wise different and thus hard to estimate. Besides, the number of prediction samples is much fewer than feature dimension, resulting in the singularity of the covariance matrices. Please refer to Sec. A. 1.1 in Appendix for details.
In contrast, we alternatively append the reference into single sample following Gaussian
YI.N|fm, XI.N ~ N (Xm, I,N, diag( (mm, fm.......
E Em,,))
where Y1:N
Xm.v] RNL are the con- catenation of observed ground-truthe truths and restored samples respectively. Since data samples can be considered following i. f.id data distribution D, the variance of the concatenated samples is diagonal.
[y
yN] NL and X:,1:N [Xm,1
However, the covariance matrix still singular due to the imbalance between prediction sample number and feature dimension. Thus, directly mixing the multivariate Gaussian is still infeasible to solve. We thus alternatively categorize pixels into various small bins of mutually exclusive ranges such that the pixels within each range can be considered following univariate Gaussian distribution according to the central limit theorem. Concretely, we separate the prediction range of models into T bins with each of width b, i.e., B = { b), [b, 26). ..... [T 1)6, 255] }. The upper bound would be 1 instead of 255 the value range is within [0, 1]. Given a bin Bn = [[t 1)b, tb) e B and a pixel of prediction location i, we define indicator function Ip_,. (x))) such that (
returns if m.1:N E Bm and otherwise. For multiple models, we have M bins to form bin set (B1, ..., BM) E B.M, and define the mask map as R, II In,,, (Xm,1.N) E {0, 1) where ..., TM, It holds DT" R, 1 and I R2 O. For each bin set (BI,.., Bas) E BM we can select pixels within the bin set from the original image vectors by
Vr,1:N R, YI:N;
Xy,,,1:N' R,, Xm,I:N,
where the operation {-} denotes the element-wise product. By the central limit theorem, we assume the nonzero pixels within each bin follow Gaussian distribution with the mean 1r,m,1:N and variance I:N, I:N, i.e.,
Lid Yr.sN If X ~ N C orm,.i:), Vi [L...., Nr],
(4)

---

where N2 the number of nonzero pixels R, such that Σx= Mr. and 1.M can be estimated by the mean and variance of N, prediction pixels within the current bin set.
TMh N, NL, and the values of
The reference set is therefore separated into TM number of bin sets, and the ground-truth pixels inside each of them form a solvable univariate GMM. We then introduce the latent variable such that = m if the pixel Yr, follows the m-th Gaussian by the model fm- It represents the probability of the pixel belonging to the m-th Gaussian component, which is equivalent to the role the ensemble weight for the m-th base model. By writing Or,m
P( m), have
M
M
(0 N Nr.iN Es [.
Σ
Ply
Σ m==
(yrLLN = m) (5)
ar
1:N
ar
m==
where Plytai 12 = m) = óly Pr,m.1:N, Trm.1:N) is the density function of Gaussian
N M( 0r.m1:NN)
The value of ensemble weights can be estimated by the maximum likelihood estimates of the observed ground-truths, i.e..
(ormm)r. E arg max P(y
(6)
Because arbitrary two bin sets mutually exclusive, can safely split the optimization of maximum likelihood over YI:N into TM optimization problems of maximum likelihood over yr,1N-- Each of them is formulated
Nr II
Or,m
E arg max P(y,,1.)) = arg max
P (INN)
drym
Gr.m
We have formulated the expression of GMMs for estimating the range-specific ensemble weights.
3.3
Restoration Ensemble via Expectation Maximization and Lookup Table
3.3.1 Weight Estimation via EM Algorithm
For each bin set (B1, ..., Bat), estimate ensemble weights by maximizing the log likelihood
Nr
log P(y..:N) log
Σ
Orra (y Mr.mNiMr.m.i. Tr.m.:.)
(=1
Nr
M
log
(yr.inii Mr.m Tr.m1::))
(8)
m=1
P
m (1)
log ar ply rl:iii 1:N
(t)
I:N, Tr.m.to)
(6)
m=1
2 m yr,1:N)
We have E-step to estimate the posterior distribution by
Oryym & (y
(2)
(r. Mr, I: 1ar.m.I.,.Tr,.m.. Tram,1:N
11a..1:N + P (=== (:=mmy:: -myyiii v
(9)
(y(²)
Dmml Ory,
yrl:Ni Mr.,1.N Tr,,1:.N
After that, we have an M-step to obtain the maximum likelihood estimates by
Orym
(10)
N
Nr 1=
y Hrm.11:N
Tr.m.1:N
Trm,1::N
No D
(11)
Prom.1:N
5

---

Thanks to the separation of bin sets, we have prior knowledge of the mean and variance of each model, which can be estimated and initialized by
N.
1
Mr
f
Nr
M.M.:N
(12)
1=1
1r.
Nr
1r.m,1:0122-
(13)
The complete and detailed derivation of the EM algorithm can be found in Sec. A.3 of Appendix.
3.3.2 Lookup Table and Inference
We store the range-specific weights estimated on the reference set into LUT with each key of (Bi.... BM). During the inference stage for test sample Xn, we have the prediction of the mi-th base model Xm,, fm (. For a bin set (Bi.... BM), partition input pixels of multiple models into each bin set as
M II I (Xm,n)) m=1
m R, Xm,, where R.
(14)
Then we retrieve the estimated range-wise weights ar,m from the LUT based on each key of the bin set and obtain the aggregated ensemble by
TM
TM M Σ
Σ
Ory O..m
(15)
r=1 m==1
The main algorithm can be found in Algo. I and the EM algorithm with known mean and variance priors is shown in Algo.
4 Experiment
4.1
Experimental Settings
Benchmarks. We evaluate ensemble method on 3 image restoration tasks includ- ing super-resolution, deblurring, and derain- ing. For super-resolution, we use Set5 [5], Set14 [83], BSDS [47]. Urban [30] and Manga109 [48] as benchmarks. For deblurring, we use GoPro [51], HIDE [60], RealBlur-J and -R [58]. For deraining, adopt R0H [74], Rain [74], Test100[88), Tes [86], and Test2800 [25].
Algorithm MPEM: EM algorithm
with known Mean Prior
Input: Yr,1.N, Xr,111:;...., Xr,M,1.N
Output: (Qr,1, 0y,M)
1 N, & number of nonzero pixels in Vr.iaN
2 Initialize 1r by Eq. 12;
3 Initialize 1:M. by Eq. 13;
4 while not converge do for i E N, do
for m E [1, M)] do Update 7 by Eq. 9;
Metrics. We use peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) [71] to quantitatively evaluate the image restoration quality. Additionally, we compare average runtime per image in seconds to eval- uate the ensemble efficiency. Following prior works [27, 39, 67, 81, 82, 93], PSNR and SSIM computed on Y channel of the YCbCr color space for image reper-resolution and deraining.
8
end
end
for
E [1, M] do Update dry, by Eq. 10;
11
Update J,,,11:N by Eq. 11;
13
end
14 end
Base Models. To evaluate the generalization of ensemble methods against model choices, we employ a wide variety of base models, including CNNs, VITs, MLPs and Mambas. For image super-resolution, we use SwinIR [39], SRFormer [93], and MambalR [27]. We choose MPRNet [82]. DGUNet [50], and Restormer [81] for deblurring, as well as MPRNet [82], MAXIM [67], and Restormer [81] for deraining.

---

D.D.D.
P
(h) HR & LR PSNR/SSIM
(c) Sw (d) HGBT [34]
(e) Average
(f) 77PM [92]
(g) Ours
Image
17.1 33.919/0.966 90.99683.83 33.8 33.873/0.966 9695 33.874/0.966 9694 33. 23.93550.9696
Figure 1: A visual comparison of ensemble on an image from Manga [48] for the task of super- resolution. "HR & LR" means high-resolution and bicubic-upsealed low-resolution images. The second line of (g)-(g) error maps. Please zoom in for better visual quality.
Baselines. We utilize regression algorithms including bagging [6], AdaBoost [22], random forests (RForest) [7], gradient boosting decision tree (GBDT) [23]. histogram gradient boosting decision tree (HGBT) [14, 34] as baselines. Averaging is also commonly used ensemble baseline. A recent method proposed by team ZZPM in the NTIRE 2023 competition [92] is also included for comparison. Additionally, we adopt RefESR [31] for image super-resolution ensemble.
Implementation Details. We choose the bin width as by default for the balance of efficiency and performance. The EM solver of GMM stops after 1000 iterations or when the change of log likelihood less than 1e 1e-5 For cases where N, is fewer than 100 or the EM solution is undetermined, we use averaging weights by default. The values are reported by taking the average of 4 trials. For the construction of the reference set, we randomly select one image from the training set of DIV2K [4] for super-resolution, while for deblurring and deraining, we sample 10 images from the training sets of GoPro [51] and Rain 13K [32], respectively. All the experiments are run in Python on device with 8-cores 2. 10GHz Intel Xeon Processor and 32G Nvidia Tesla V V100. The regression-bastt ensemble algorithms are implemented based on scikit-learn [8].
Table 1: Ablation study of bin width on Rain 100H [74] with maximum step number 1000. "Runtime" is the average runtime [s].
Table 2: Ablation study of maximum step num- ber in the EM algorithm on Rain [74] with b= b= 32. "Time" is the time of EM algorithm [s].
16
32
64
96
128
#step 10
100
500
1000
10000
Runtime
1.2460
0.1709
0.0265
0.0132
0.0059
Time
12.108
28.516
30.409
30.518
30.524
PSNR
31.745
31.739
31.720
31.713
31.725
PSNR
31.734
31.738
31.738
31.739
31.739
SSIM
0.9093
0.9095
0.9094
0.9093
0.9093
SSIM
0.9093
0.9094
0.9095
0.9095
0.9095
Table 3: The ensemble results on the task of image super-resolutton. The categories of "Base", "Regr." and "IR." in the first column mean base models, regression-based ensemble methods, and those ensemble methods designed for image restoration. The best and second best ensemble results emphasized in bold and underlined respectively.
Datasets
Set5 [5]
Set14 [83] BSDS100 [47] Urban100 [30] Manga [48]
Metrics
PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR
SSIM
SwinIR [39]
32.916 0.9044 29.087 0.7950 27.919 0.7487 27.453 0.8254 32.024 0.9260
Base SRFormer [93]
32.922 0.9043 29.090 0.7942 27.914 0.7489 27.535 0,8261 32,203 0.9271
MambalR [27] 33.045 0.9051 29.159 0.7958
27.967 0.7510
27.775 0.8321
32.308
0.9283
Bagging [6] 33.006 0.9050 29.119 0.7950 27.946 0.7498
27.546 0,8273 32.154
0.9270
AdaBoost [22]
33.072 0.9049 29.175 0.7959 27.975 0.7503 27.786 0,8302 32.457 0.9286
Regr. RForest [7]
33.032 0.9052 29.158 0.7954 27.964 0.7500 27.640 0.8287 32.287 0,9279
GBDT [23]
33.085 0.9050 29.196 0.7956 27.980 0.7500 27.792 0.8311 32.467 0.9285
HGBT [34]
33.078 0.9051 29.201
0.7959 27.984
0.7502 27.783
0,8310
32.444
0.9282
Average
33.097 0.9057 29.202
0.7964
27.983
0,7506
27.785 0.8313
32,466
0.9290
RefESR [31]
33.091 0.9052 29.172
0.7960
27.972 0.7504
27.785 0,8312
32.447
0.9288
IR.
77PM [92]
33.094 0.9057 29.203
0.7963
27.981 0.7506
27.786
0.8313
32.467
0.9290
EnsIR (Ours)
33.103
0.9058
29.205
0.7964
27.984
0.7507
27.795
0.8315
32.468
0.9291

---

Table 4: The ensemble results on the task of image deblurring. The categories of "Base", "Regr" and
"IR." in the first column mean base models, regression-basee ensemble methods, and those ensemble
methods designed for image restoration. The best and second best ensemble results are emphasized
in bold and underlined respectively.
Datasets
GoPro [51]
HIDE [60]
RealBlur-R [58]
RealBlur-J [58]
Metrics
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
MPRNet [82]
32.658
0.9362
30.962
0.9188
33,914
0.9425
26.515
0.8240
Base
Restormer [81]
32,918
0.9398
31.221
0.9226
33.984
0.9463
26.626
0.8274
DGUNet [50]
33.173
0.9423
31.404
0.9257
33.990
0.9430
26.583
0,8261
Bagging [6]
33.194
0.9418
31.437
0.9250
34.033
0.9456
26.641
0.8277
AdaBoost [22]
33.205
0.9412
31,449
0.9251
34.035
0.9455
26.652
0,8280
Regr.
RForest [7]
33.173
0.9416
31,439
0.9247
34.039
0.9457
26.647
0,8280
GBDT [23]
33.311
0.9418
31.568
0.9256
34.052
0.9465
26,684
0.8285
HGBT [34]
33.323
0,9427
31.583
0.9267
33,986
0,9436
26.684
0.8296
Average
33,330
0,9436
31.579
0,9277
34.090
0.9471
26.689
0.8309
IR.
ZZPM [92]
33.332
0,9436
31,580
0.9277
34.057
0.9468
26.688
0,8308
EnsIR (Ours)
33.345
0.9438
31.590
0.9278
34.089
0.9472
26.690
0.8309
Table The ensemble results on the task of image deraining. The categories of "Base", "Regr." and "IR." " in the first column mean base models, regression-based ensemble methods, and those ensemble
methods designed for image restoration. The best and second best ensemble results are emphasized
in bold and underlined respectively.
Datasets
Rain 100H [74] Rain [74] Test100 [88] Test [86] Test2800 [25]
Metrics
PSNR SSIM PSNR SSIM
PSNR SSIM PSNR
SSIM PSNR SSIM
MPRNet [82]
30.428 0.8905 36.463 0.9657 30.292 0.8983 32,944 0.9175 33.667 0.9389
Base
MAXIM [67]
30.838
0.9043 38. 152 0.9782 31.194 0.9239 32.401 0,9240 33.837 0,9438
Restormer [81]
31.477 0.9054
39.080 0.9785
32.025
0,9237
33.219 0.9270
34.211 0.9449
Bagging [6]
31.461
0.9001 39.060 0.9782 31.865 0.9107 33.115 0,9152 34.216 0.9446
AdaBoost [22]
31.472 0,9006 39.067 0.9782 31.88 36.866 0,9112 33.117 0.9153 34.221 0.9443
Regr. RForest [7]
31.492 0.9012 39.089 0.9784 31.900 0,9127 33.147 0.9169 34.224 0.9447
GBDT [23]
31.581
0.9058 39.044 0.9778 32.001 0,9236 33,276 0,9274 34,211 0,9446
HGBT [34]
31.698 0.9075
39.115 0.9784 31.988 0.9241 33.305 0.9282
34.229 0.9450
Average
31,681
0.9091
38.675
0.9770
31.626
0.9225
33.427
0,9286
34,214
0,9449
IR.
ZZPM [92]
31.689 0.9091
38.725
0.9771
31.642
0,9227 33.434
0.9286
34.231
0.9450
EnsIR (Ours)
31.739
0.9095
39.216
0.9792
32.064
0.9258
33.445
0.9289
34.245 0.9451
4.2
Experimental Results
4.2.1
Ablation Study
We conduct ablation studies on the bin width b and the maximum step number of the EM algorithm on the benchmark of Rain 100H [74]. The results are shown in Tab. and Tab. 2 respectively. We can observe that the bin width involves a trade-off between ensemble accuracy and the efficiency small b generates better ensemble results but makes ensemble inference slower. In the following experiments, we choose the bin width b = 32 by default for the balance of efficiency and performance. On the other hand, a large maximum step number does not necessarily yield better ensemble result. Therefore, we choose 1000 as the maximum step number.
4.2.2 Quantitative Results
We present quantitative comparisons with existing ensemble methods in Tab. 3 for super-resolution, Tab. 4 for deblurring, and Tab. 5 for deraining. Our method generally outperforms all existing methods across the image restoration tasks and 14 benchmarks. Note that Average and 77PM [92] generally perform better than regression- based ensemble methods. However, in cases where one of the base
8

---

(b) GT & LQ PSNR/SSIM
(c) Restormer (d) HGBT [34] Average (f) 77PM [92]
(g) Ours
(a) Image
29. 98090.9377 32.1 10/0.9481 32.177/0.9479
32.1 02.177/0.947 32.584/0.9458 9486
Figure 2: A visual comparison of ensemble deblurring. "GT & LQ" means ground-truth and low quality blurry images. The second line of (c)-(g) error maps. Please zoom in for better visual quality.
image from GoPro [51] for the task of image
(b) GT & LQ PSNR/SSIM
(c) MPRNet (d) HGBT [34] (e) Average (1) ZZPM [92]
(g) Ours
(a) Image
34.008/0.9359 37.052/0.9615 36.775/0.9603
36.777/0.966 37.372/0.9639
Figure 3: A visual comparison of ensemble on an image from Test100 [88] for the task of image deraining. "GT & LQ" means ground-truth and low quality rainy images. The second line of (c)-() are error maps. Please zoom in for better visual quality.
models significantly underperforms compared to the others, such as MPRNet [82] on Rain100L [74] and Test100 [88], these regression methods outperform Average and ZZPM [92]. In contrast, our method, which learns per-value weights, can recognize performance biases and alleviate such issue. ZZPM [92] performs comparably to Average in our experiments rather than outperforming it, because the base models are not always equally good and one model may be consistently better than the others. Thus, weights negatively proportional to the mean squared error may exaggerate deviation from optimal prediction. In contrast, our method consistently performs well for all cases.
4.2.3 Qualitative Results
We also provide qualitative visual comparisons in Fig. 1, and 3. In Fig. 1, the base model SwinIR [39] mistakenly upscales the character's ' eye. While existing ensemble algorithms partially alleviate this mistake, they cannot fully discard the hallucinated line inside the eye. In contrast, method with the bin width b = 32 that learns fine-grained range-wise weights successfully recovers the pattern. In Fig. 2, only our method can effectively obtain a better ensemble with sharp edges and accurate colors. In Fig. 3, we can observe that MPRNet [82] removes all stripe patterns on the ground together with rain streaks. The conventional weighted summation yields dimmer ensemble result, and the HGBT method fails to learn accurate weight distributions, resulting in an unsmooth result. In contrast, ours alleviates the issue. More visual comparisons are provided in Fig. 7-18 in Appendix.
4.2.4 Extensions
Efficiency.
The efficiency of ensemble methods is compared by measuring the average runtime on Rain [74], as shown in Tab. 6. Although our method is slower than Average and 7ZPM [92],
Table 6: The average runtime per image in seconds of the ensemble methods on Rain 100H [74].
Method Bagging [6] AdaBoost [22] RForest [7] GBDT [23] HGBT [34] Average ZZPM [92] Ours
Runtime 1.0070
1.1827
9.8598
1.2781
0.1773
0.0003
0.0021
0.1709

---

it is much faster than all the regression-basee based methods. By slightly sacrificing performance with b 64, it can achieve real-time inference, as indicated in Tab. 1.
Visualization. We also present the visualization examples of ensemble weights, image features and pixel distributions in Fig. 4-6 in Appendix. Due to page limit, please refer to Sec. B of Appendix.
Limitation and Future Work. The trade-off between the runtime and performance has not been solved yet. Achieving real-time ensemble would lead to performance degradation. The issue could be resolved by GPU vectorization acceleration and distributed computing. Additionally, if all base models fail, ensemble methods cannot generate better result. We leave them in the future work.
Conclusion
In this paper, propose ensemble algorithm for image restoration based GMMs. We partition the pixels of predictions and ground-truths into separate bins of exclusive ranges and formulate the ensemble problem using GMMs over each bin. The GMMs are solved estimated ensemble weights are stored in a lookup table for the ensemble inference on the test set. Our algorithm outperforms regression- strategies on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. It is training-free, model-agnostic, and thus suitable for plug-and-play usage.
reference set, and the
ensemble methods well as commonly used averaging
Acknowledgement
This work has been supported in part by National Natural Science Foundation of China (No. 62322216, 62025604, 62172409), in part by Shenzhen Science and Technology Program (Grant No. JCY120022081810201220 KOTD20221101093355 RCXXX000000000002944060
References
Abdelrahman Abdelhamed, Mahmoud Afifi, Radu Timofte, and Michael S Brown. Núie 2020 challenge on real image denoising: Dataset, methods and results. CVPRW, 2020. 2,
[2] Hazique Aetesam and Suman Kumar Maji. Noise dependent training for deep parallel ensemble denoising in magnetic resonance images. Riomedical Signal Processing and Control, 2021. 3
[3] Subhash Chand Agrawal and Anand Singh Jalal. Distortion-free image dehazing by superpixels and ensemble neural network. The Visual Computer, 2022. 3
[4] Eirikur Agustsson and Radu Timofte. Núre 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017. 7
[5] Marco Bevilacqua, Aline Roumy, Christine M. Guillemot, and Marie-Line Alberi- AbbriiMMrel. Low- Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 6,
[6] Leo Breiman. Bagging predictors. Machine learning. 1996. 2, 7, 8,
[7] Leo Breiman. Random forests. Machine learning, 2001. 2, 7, 8,9
[8] Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculac, Peter Prettenhofer, Alexandre Gramfort, Jaques Grohler, Robert Layton, Jake VanderPlas, Arnaud Joly. Brian Holt. and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. ECML PKDD Workshop: Languages for Duta Mining and Machine Learning.
pages 108-122, 2013. 7
[9] Chung Chan, Jian Zhou, Li Yang, Wenyuan Qi. and Evren Asma. Noise to noise ensemble learning for pet image denoising. In IEEE NSS/MIC, 2019.
[10] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In ECCV, 2022. 2
[11] Mingqin Chen. Yuhui Quan, Yong Xu, and Hui Ji. Self superviuee blind image deconvolution via deep generative ensemble learning, IEEE TCSVT, 2022.
10

---

[12] Rui Chen, Huizhu Jia, Xiaodong Xie, and Gao Wen. A setucture-preserving image restoration method with high-level ensemble constraints. In VCIP, 2016.
[13] Ruoyu Chen, Hua Zhang. Siyuan Liang, Jingzhi Li, and Xianchun Cao. Less is more: Fewer interpretable region via submodular subset selection. ICT.R, 2024.
[14] Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In KDD, 2016. 2,7
[15] Zheng Chen, Zongwei Wu, Eduard Zamfir, Kai Zhang. Yulun Zhang, Radu Timofte, Xiaokang Yang- Hongyuan Yu, Cheng Wan, Yuxin Hong, et al. Ntire 2024 challenge image super-resolution (x4): Methods and results. arXiv preprint (eXx::2444.09790 2024. 2,
[16] Zhibo Chen, Jianxin Lin, Tiankuang Zhou, and Feng Wu. Sequential gating ensemble network for noise robust multiscale face restoration. IEEE TCYB, 2019. 3
[17] Subhrajit Dey, Rajdeep Bhattacharya, Friedhelm Schwenker, and Ram Sarkar. Median filter aided enn based image denoising: an ensemble approach. Algorithms, 2021. 3
[18] Thomas G Dietterich et al. Ensemble learning. The handbook of brain theory and neural networks, 2002. 2
[19] Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, 2014.
[20] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. survey on ensemble learning. Frontiers of Computer Science, 2020. 2
[21] Masud An Nur Islam Fahim, Nazmus Saqib, Shafkat Khan Siam, and Ho Yub Jung. Denoising single images by feature ensemble revisited. Sensors, 2022. 3
[22] Yoav Freund and Robert E Schapire. A decision-theoreeic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 1997. 7, 8,
[23] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, 2001. 7, 8,9
[24] Jerome H Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 2002. 2
[25] Xucyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via deep detail network. In CVPR, 2017. 6, 8
[26] Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai Suganthan. Ensemble deep learning: review. Engineering Applications of Artificial Intelligence, 2022. 2
[27] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Ta Xia. Mambair: A simple baseline for image restoration with stale-space model. arXiv preprint arXiv:2402. 15648, 2024. 1, 6, 22
[28] Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. Multi-class adaboost. 2009. 2
Statistics and its Interface.
[29] Trung Hoang, Haichuan Zhang, Amirsaeed Yazdani, and Vishal Monga. Transer: Hybrid model and
ensemble-based sequential learning for non-homogenous dehazing. In CVPR, 2023. 3
[30] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015. 6, 7,
[31] Junjun Jiang, Yi Yu, Zheng Wang, Suhua Tang, Ruimin Hu, and Jiayi Ma. Ensemble super-resolution with a reference dataset. IEEE TCYB, 2019. 3,
[32] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In CVPR, 2020. 7
[33] Zhiying Jiang. Shuzhou Yang. Jinyuan Liu, Xin Fan, and Risheng Liu, scale synergism ensemble progressive and contrastive investigation for image restoration. IEEE TIM, 2023. 3
[34] Guolin Ke, Qi Meng, Thomas Finlcy, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Yc, and Tic- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. NewrIPS, 2017. 2, 7, 8, 9, 18, 22, 23, 24
[35] Chao Li, Dongliang He, Xiao Liu, Yukang Ding, and Shilei Wen. Adapting image reportresolution state-of oftte-of-hh-arre the--rtes and learning multi-model ensemble for video super-resolution. In CVPRW, 2019. 2.3
11

---

[36] Jinyang Li and Zhijing Liu. Ensemble dictionary learning for single image deblurring via low-rank regularization. Sensors, 2019.
[37] Yuenan Li, Yuhang Liu, Qixin Yan, and Kuangshi Zhang. Deep dehazing network with latent ensembling architecture and adversarial learning. IEEE TIP, 2020.
[38] Yufeng Li, Jiyang Lu, Zhentao Fan, and Xiang Chen. Learning an ensemble dehazing network for visible remote sensing images. Journal of Applied Remote Sensing, 2023.
[39] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, 2021. 1, 3, 6, 7, 9, 19, 22
[40] Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, and Jiaya Jia. Vidco super-resolution via deep araftlcnsemmle learning. In ICCV, 2015. 2,
[41] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 2
[42] Jianxin Lin, Tiankuang Zhou, and Zhibo Chen. Multi-scale face restoration with sequential gating ensemble network. In AAAI. 2018. 3
[43] Pengju Liu, Hongzhi Zhang, Jinghui Wang, Yuzhi Wang, Dongwei Ren, and Wangmeng Zuo. Robust deep ensemble method for worldworld image denoising. Neurocomputing, 2022.
[44] Yingnan Liu and Randy Clinton Paffenroth. Ensemble enn in transform domains for image super-resolution from small data sets. In ICMLA, 2020. 2
[45] Yu Luo, Menghua Wu, Qingdong Huang, Jian Zhu, Jie Ling, and Bin Sheng. Joint feedback and recurrent deraining network with ensemble learning. The Visual Computer, 2022.
[46] Qing Lyu, Hongming Shan, and Ge Wang. Mri super-resolution with ensemble learning and complementary priors. IEEE TCI, 2020. 2,
[47] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 6, 7
[48] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga 099 datasct. Multimedia tools and applications, 2017.
6,7 7
[49] Joao Mendes-Moreira, Carlos Soares, Alípio Mário Jorge, and Jorge Freire De Sousa. Ensemble approaches for regression: survey. ACM Computing Surveys, 2012.
[50] Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restoration. In CVPR, 2022. 6, 8, 18, 19, 20,
[51] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017. 6, 7, 8,8,
[52] Seungjun Nah, Sanghyun Son, Suyoung Lee, Radu Timofte, Kyoung Mu Lee, Liangyu Chen, Jie Zhang, Xin Lu, Xiaojic Chu, Chengpeng Chen, et al. Ntire 2021 challenge on image deblurring. In CVPR, 2021.
2,
[53] Luciano Oliveira, Urbano Nunes, and Paulo Peixoto. On exploration of classifier ensemble synergism in pedestrian detection. JEEE TITS, 2009. 2
[54] Zhihong Pan, Baopu Li, Teng Xi. Yanwen Fan, Gang Zhang, Jingtuo Liu, Junyu Han, and Errui Ding. Real image super resolution via heterogeneous model ensemble using gp-nas. In ECCVW, 2020. 2,
[55] Long Peng, Aiwen Jiang, Haoran Wei, Bo Liu, and Mingwen Wang. Ensemble single image deraining network via progressive structural boosting constraints. Signal Process Image Commun., 2021." 3
[56] Wenqi Ren, Si Liu, Hua Zhang, Jinshan Pan, Xiaochun Cao, and Ming- Hsuun Yang. Single image dehazing via multi-scale convolutional neural networks. In ECCV, 2016. 3
[57] Ye Ren, Le Zhang, and Ponnuthurai N Suganthan. Ensemble classification and regression-reeent develop- ments, applications and future directions. IEEE CIM, 2016. 2
12

---

[58] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV, 2020. 6, 23
[59] Ali Shahsavari, Sima Ranjbari, and Toktam Khatibi. Proposing novel cascade ensemble super resolution generative adversarial network (cesr-gan) method for the reconstruction of super-resolution skin lesion images. Informatics in Medicine Unlocked, 2021. 3,3
[60] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human- aware motion deblurring. In ICCV, 2019. 6, 8, 18, 19, 20, 21, 23
[61] Yu Su, Shiguang Shan, Xilin Chen, and Wen Gao. Hicrarchical ensemble of global and local classifiers for face recognition. IEEE TIP, 2009. 2
[62] Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, and Xiaochun Cao. Restoring images in adverse weather conditions via histogram transformer. In ECCV, 2024. 3
[63] Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In CVPR, 2024. 3
[64] Shangquan Sun, Wenqi Ren, Jingzhi Li, Kaíhao Zhang, Mciyu Liang, and Xiaochun Cao. Event-aware video deraining via multi-patch progressive learning. IEEE TIP, 2023. 3
[65] Shangquan Sun. Wenqi Ren, Tao Wang, and Xiaochun Cao. Rethinking image restoration for object detection. NewrIPS, 2022. 2
[66] Radu Timofte, Rasmus Rothe, and Luc Van Gool. Seven ways to improve example-base single image super resolution. CVPR, 2016. 2,
[67] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang. Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim; Multi-axis mlp for image processing. In CVPR, 2022. 3, 6, 8, 19, 23, 24
[68] Florin-Alexandru Vasluianu. Tim Seizinger, Radu Timofte, Shuhao Cui, Junshi Huang, Shuman Tian, Mingyuan Fan, Jiaqi Zhang, Li Zhu, Xiaoming Wei, et al. Ntire 2023 image shadow removal challenge report. In CVPR, 2023. 3,3
[69] Jixiao Wang, Chaofeng Li, and Shoukun Xu. An ensemble multi-scale residual attention network (emra-net) for image dehazing. Multimedia Tools and Applications, 2021.
[70] Lingfeng Wang, Zehao Huang, Yongchao Gong, and Chunhong Pan. Ensemble based deep networks for image super-resolution. Pattern recognition, 2017. 2, 3
[71] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE TIP, 2004,
[72] WenBo Wu, Yun Pan, Na Su, JingTao Wang, ShaoChuan Wu. ZeKun Xu, YouJian Yu, and YaPeng Liu. Multi-scale network for single image deblurring based on ensemble learning module. Multimedia Tools and Applications. 2024.
[73] Li Xu, Jimmy S Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvolution. NeurIPS, 2014.3
[74] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In CVPR, 2017. 8, 9, 23, 24
[75] Xuhui Yang. Yong Xu, Yuhui Quan, and Hui Ji. Image denoising via sequential ensemble learning. IEEE TIP, 2020. 3
[76] Chenyu You, Guang Li, Yi Zhang, Xiaoliu Zhang, Hongming Shan, Mengzhou Li, Shenghong Ju, Zhen Zhao, Zhuiyang Zhang, Wenxiang Cong, et al. Ct super-resolution gan constrained by the identical, residual, and eycle learning ensemble (gan-circle). IEEE TMI, 2019. 2.3
[77] Dong-Jun Yu, Jun Hu, Jing Yang, Hong-Bin Bin Shen, Jinhui Tang, and Jing- Yu Yang. Designing template- free predictor for targeting protein-ligand binding sites with classifier ensemble and spatial clustering. IEEE/ACM TCBB, 2013.
[78] Guoxian Yu, Huzefa Rangwala, Carlotta Domeniconi, Guoji Zhang, and Zhiwen Yu. Protein function prediction using multilabel ensemble classification. IEEE/ACM TCBB, 2013. 2
[79] Mingzhao Yu, Venkateswararao Cherukuri, Tiantong Guo, and Vishal Monga. Ensemble dehazing networks for non non- non-homggeneous haze. In CVPRW, 2020. 3
13

---

[80] Yankun Yu, Huan Liu, Minghan Fu, Jun Chen. Xiyao Wang, and Keyan Wang. A two-branch neural network for non-homogeneous dehazing via ensemble learning. In CVPR, 2021.
[81] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahhaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022. 1,3, 6, 8,
19. 20, 21, 23,
[82] Syed Wagas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021. 1, 3, 8, 18, 19, 20,
21, 23, 24
[83] Roman Zeyde, Michael Elad, and Matan Protter. single image scale-up using pppreetrrprssnntatioonn. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, 2012. 6, 7, 22
[84] Bailing Zhang. Reliable classification of vehicle types based on cascade classifier ensembles. JEEE TITS.
2012. 2
[85] Dan Zhang, Yingbing Xu, Liyan Ma, Xiaowei Li, Xiangyu Zhang, Yan Peng, and Yaoran Chen. Srenet: Structure recovery ensemble network for single image deraining. Applied Intelligence, 2024, 3
[86] He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi-stream dense network.
In CVPR, 2018. 6, 8, 24
[87] He Zhang, Vishwanath Sindagi, and Vishal M Patel. Multi-scale single image dehazing using perceptual pyramid deep network. In CVPRW, 2018. 3
[88] He Zhang, Vishwanath Sindagi. and Vishal M Patel. Image de-raining using a conditional generative adversarial network. IEEE TCSVT, 2019. 6, 8,
[89] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 2019. 3
[90] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In CVPR, 2017. 3
[91] Ping Zhang, Tien D Bui, and Ching Y Suen. A novel cascade ensemble classifier system with high recognition performance handwritten digits. Puttern Recognition, 2007.
[92] Yulun Zhang, Kai Zhang, Zheng Chen, Yawei Li, Radu Timofte, Jumpei Zhang, Kexin Zhang, Rui Peng, Yanbiao Ma, Licheng Jia, et al. Ntire 2023 challenge image super-resolution (x4): Methods and results. In CVPR, 2023. 2, 3, 7, 8, 18, 19, 22, 23, 24
[93] Yupeng Zhou, Zhen Li, Chun-Le Guo, Song Bai, Ming-Ming Cheng, and Qibin Hou. Srformer: Permuted self-attention for single image super-resolution. In ICCV, 2023. 1, 22
14

---

Appendix
A Derivation
We first list the theorems necessary for the subsequent derivations
Theorem A.1. Suppose X Nux σx) and Y N(uy, are two independent random f following univariate Gaussian distribution. If we have another random variable defined as Z = X + Y, then the variable variable,follos Z ~ Nux + jyy ox + +vy).
variables
Theorem A.2. Suppose X ~ NNux Σx) and Y ~ N(y Ey) are two independent random variables following multivariate Gaussian distribution. If we have another rundom variable defined as Z X + Y, then the variable follows Z ~ (ux + pv, Ex Ev).
Theorem A.3. Suppose X ~ ~N(,, Σ) is random variable following multivariate Gaussian distribution, where μ RRL. Given a constant vector Re,, we have the variable Y = N(a a a. E).
In Sec. A.I and A.2, is shown the mixture of multivariate Gaussian is difficult to solve with limited samples while large feature dimension. In Sec. A.3, derive the updating of GMMs for the pixels within a bin set. Its convergence is proved Sec. A.4.
A.1 Sample-wise Maximum Likelihood Estimation of multivariate Gaussian Ensemble
we directly consider the ensemble problem by the sample-wise weighted summation of multivariate Gaussian with the ensemble weights fn.m such that 2m Bn,m 1, i.c.
yalfi.-. - Mil,,,--,,,,,.
fas, Xn ~ N
Σ
Bn.m X
32 Fn.m Σ
E 2m,
(16)
m==
The log likelihood is therefore
log P (y1, (y YN..., SMS X1, a, XN)
log II P (y Sas, Xn),
N
M
M
log
Yni
Bn X
32 n,m Em,n
(17)
nm
m
m=m
N
M
Yn EM Pn.mX Xmm
log
2m
32 nmm
Em,n
2 2MM
32 Em.n 11,
The weights can be found via computing the maximum likelihood estimates by aB.... However, the derivative complicated, sample-wisely different and related to both unknown and sample-wise covariance matrices. Directly making sample-wise ensemble is thereby difficult to solve.
0L
Besides, if we use multivariate GMMs with the EM algorithm, the estimation of covariance requires the computation of its inverse. However, we only have M observed samples while L features with > M for each prediction sample. The covariance matrix singular and thus multivariate GMMs cannot be solved.
A.2 Set-level Maximum Likelihood Estimation of multivariate Gaussian Ensemble
If we directly consider the ensemble problem by the weighted summation of multivariate Gaussian over the reference set with the ensemble weights 8r.m such that Σ 3m 1, i.c.,
M Σ
M 32 diag Em,,, ..., Em,,r)
YIsNJfi JM, XINN N
Bm X
I,N
(18)
m=1
15

---

The log likelihood is therefore
log P(yl:N fi. fi....,,,, sas X1:N)
M
VI:N -EM
Pm Xm,
(19)
log
2T
Σ m=1
L diag(2 mm... Em.xr)
2
> M
S diag Em,1, Em.n)
OL O. Because we only We can compute the optimal maximum likelihood estimate by making 83m have M observed samples while NL features with NL M for each prediction sample. The unknown covariance matrix still cannot be estimated by multivariate GMMs with the EM algorithm,
The estimation of covariance requires the computation its inverse. The covariance matrix singular and thus the multivariate GMMs cannot be solved.
A.3 Derivation of GMMs in bin set
For each bin set, we have converted the problem into the format of GMMs, i.e,,
M
Ply
Σ
P(= = m)P
(y1flIN/= m)
=
m=1
(20)
M
Σ
arym
Vr.iiiirrmm.i Trm.1:))
where: z (1,-... M) } is the latent variable that represents the probability of the m-th mixture, and (3) P (r.IsN == m) is the m--th mixture probability component. We use dry, P(= m) to denote
the mixture proportion or the probability that Yr,I.N belongs to the m-th mixture component. We assume the pixels within the bin follow the Gaussian distribution based on the central limit theorem,
(+)
i.e., yr.1:N Lid (M Orm,1::V and 11,1:// 2= = m) & (y M:N Tram.:N)
(4)
The mean and variance of the bin are denoted as Pram,1:N and 0r,,,1:N.
Suppose we have N, data samples in the bin, then for the N, observations, we have its joint probability
N. II
P(y..NN) = P (YrIN
(N.) MIN
PlyliN))
(21)
N- M
II
amm
(y Trmm.1:N
Im 1 m=1
Different from conventional GMMs, we have the observation prior that
Pr,,11:
M.M.IN
(22)
and thus we want to find the maximum likelihood estimates of armm The initial variance, Gr.m, I:N can also be estimated by
N. N,
Trm,1::
Σ
Mr.,,1:.N
(23)
Theorem A.4. (Jensen's 's Inequality) Given convex convex funnction f. we have f(E[X]) 2 B[f(X)
16

---

The log likelihood of the GMMs is
log P(Yr,1:N) log
P(==m)P
/ =m=m) m
m=1
Nr
P = m)P (y 2 m)
log
M
P(= = m)P
y r,11:N m
(6)
log
P 2 m 1r,1:N
p 2
(4)
(24)
m=1
r,1:N
N
p(z m)P
(yrlsN 2
(6)
log
P 2 m
m
(1)
P(z m)P P (y1,1.N 2 m
2 m mpyniii/ log
P mly mly r,1:N)
Plug the expression of P(2 m) and P (y (ynLiN m
in and get
N.
M
Crym ar,mp pl (1) Pll: Mr.m,1:N 0r.,1:N)
log P(y,,IN))
,,,1,,)) log
(1)
(25)
P(z = m mly.dan)
m==1
where the inequality is based on the Jensen's Inequality.
The problem be effectively solved by the EM algorithms. We have the an E-step to estimate posterior distribution of by
1:N + 2 = m=mmy mly r,1::N
P (y = -m) ((z=m)
P P(X!1LN))
P yy m) P(z = m)
(26)
Σ M P 1r.1:01== m) P(= = m)
(r Mr..N.MM,,m...... Tr.m,l:N)
ΣM=1 am o (y Mr,m,1:N Trm,1::)
(8)
After that, we have an M-step to obtain maximum likelihood estimates by making the derivative of the log likelihood equal zero,
N. Σ
Or
Tram.1:N
(27)
N,
Miin
1:N (y
P.M.INN Mr,n,
Or.,1::N
(28)
ΣN Tr.m.1:N
We do not update the mean because we have its prior knowledge of value as the mean of base model prediction pixels within the bin set.
A.4 Convergence of GMMS with mean prior
Let 0 = [[ar1,, Mr,M Or.11:NN ..., Or,M.I:N] } be the estimate variable. To validate the convergence of GMMs, we want to prove
y (yf.l.//00 (0+t1) 2 (y!.).N (y 10²)
(29)
17

---

where 6 means the estimate variables at the t-th step.
We start from
log (Yr,i.v 0) - log (y,,..n: llog P (2
(30)
Based on the objective of the M-step, i.e.,
0 E arg max log P (y 21)
(31)
we can naturally guarantee
log P r,I::N (yr...N,2000) 2 2 log P (1)
(32)
We also have
P (2)y (t)
yr,1:N+ 0
DKL (P (Elx (i) (2l (s(x
I #ly
log
50
(8)
P
(33)
where DKL denotes the Kullback-Leible (KL) divergence function.
can thus obtain
log P yrlN (0+t1) log P (v log P (2)x))un)
(34)
log P yri (r) log P (x1 2 log P (2
The convergence is therefore guaranteed. The prior of mean does not affect the convergence.
B
More Visualizations
B.1 Weight Visualization
We plot the heatmap of ensemble weights on an example from HIDE [60]. The weight heatmaps for averaging, ZZPM [92] and ours shown in Fig. 4. We that ours assigns more detailed weight on its prediction to preserve textures.
B.2 Feature Visualization
We plot the image feature visualization via the dimension reduction of principal components analysis (PCA). We randomly sample 8 images from HIDE [60]. For each image, we obtain its restoration predictions of base models, the ensemble results of averaging. ZZPM [92], HGBT [34] and ours. A pre-trained ResNetl 10 is used to encode images into features. The visualizations are shown in Fig. be found that ours can consistently yield the results closer to the gorund-truths (GT).
B.3 Distribution Visualization
Note that we hypotheses of pixels within each bin set following Gaussian and their weights of GMMs learnt on the reference set can be used to fuse pixels on the test set. We validate them by plotting the histograms within bin sets in Fig. 6. As seen, the distribution of the pixels within bin follows either typical Gaussian, plateau that can be considered as a flat Gaussian, or discrete signals that can be considered as steep Gaussian. The relative location of the distributions among base model predictions and ground-truthe is also well preserved from the reference set to the test set.
We can notice that the simple averaging strategy can suffice for the case of the last row, while a biased first two rows.
averaging towards DGUNet [50] and MPRNet [82] will better for the case For the case of the third row, directly applying the prediction of DGUNet [50] will be the best. Our method estimating the range-wise weights as the latent probability of GMMs can handle all the cases, while the averaing strategies will fail.
18

---

(a) MPRNet [82] 25.132 / 0.8827
(c) Weight map of MPRNet [82]
(i) Weight map of MPRNet [82]
(m) Weight map of MPRNet [82]
(b) Restormer [81] 24.202 0,8587
(f) Weight map Restormer [81]
Weight map of Restormer [81]
Weight map of Restormer [81]
(c) DGUNel [50] 24.670 0.8656
(g) Weight map of DGUNet [50]
(k) Weight map of DGUNet [50]
(o) Weight map of DGUNet [50]
(d) Ground-trun PSNR / SSIM
(h) Averaging 25.232 0.8817
()) ZZPM [92] 25.233 0.8837
(p) Ours w/ b 32 25.247 0.8840
Figure 4: A sample of weight visualizations HIDE [60]. Base models are DGUNet [50], MPR- Net [82], and Restormer [81]. The first column shows the base model predictions and ground-truth. The second column shows the ensemble weights and result of the averaging strategy. The third column shows the ensemble weights and result of ZZPM [92]. The last column shows the ensemble weights and result of our method.
C More Visual Comparisons
We show visual comparisons in Fig. 7-10 for image super-resolution, in Fig. 11-13 for image deblurring, and in Fig. 14-18 for image deraining.
In Fig. 7, only is able preserve gray reflection of the boat from SwinIR [39]'s prediction. In Fig. 8-10, our method obtains the most accurate ensemble results in the case that one base models generates mistaken textures.
In Fig. 11, our method yield the sharpest and straight line like the ground -trtth. In Fig. 12 and 13, our ensemble method obtains the closest textures to the ground-truths.
In Fig. 14, method gets the best ensemble despite the mistake from MAXIM [67]. In Fig. 15, ours preserves the cleanest background to the ground-truth. In Fig. 16, only ours recovers the grid near the eyeball, In Fig. 17 and 18, ours preserves the closest background textures to the ground-truths.
19

---

0.4
Base
HGBT
ZZPM
0.2
0.0
-0.2
-0.4
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
1.00
Base
GT
Average
0.3
0.75
HIGHT
ZZPM
0.1
3.5
-0.1
Average
HGET
-0.50
ZZPM
-0.4
Ours
-0.5
8.5
3.5
2.5
3.0
-0.2
0.0
0.2
0.4
0.6
0.8
Base
GT
Average
0.3
Average
HGst
3.5
ZZPM
0.2
ZZPN
Ours
3.00
-0.1
-0.50
-0.2
-0.75
-0.5
0.0
05
2.0
2.5
0.5
0.0
0.3
1.b
1.5
0.4
0.5
Average
Averago
0.3
ZZPM
0.2
ZZPM
Durs
Ours
0.2
0.3
-0.1
-0.1
-0.2
0.3
-0.4
-22
0.0
0.2
0.4
0.5
0.8
3.0
12
0.0
2.5
1.5
1.5
Figure 5: sampled group of feature visualizations on HIDE [60]. "Base" denotes the features of three base models, i.e., DGUNet [50], MPRNet [82], and Restormer [81].
20

---

GT
DGulee
DGuhte
MFRNet
MPRket
Restorme
3900
2500
1500
2000
1000
500
10
3.0
DGUNet
DGUNet
2.5-
Rastor mar
0.8
1.5-
0.2
0.5
0.0
120
46
60
80
100
120
2.00
4.0
GT
DQUNet
MPRhet
3.5-
Restormer
1.50
3.0
1.25
1.00
0.50
1.0
0.25
0.5
0.00
0.0
60
120
140
25030
14000
G1 DGUNe 14R00
12000
Restormer
Restormer
10000
15000
5020
10000
4000
5000
10
75
95
75
Reference set
(b) Test set
Figure group of distribution visualizations on HIDE [60]. The bin sets of the first row is
(B, [0. 32), B2 [0, 32). B3 [32, 64)). The bin sets of the second row is (B) |64, 96), Bg
(32, 64), B3 = [96, 128).. The bin sets of the third row (B, = [64, 96), B2 = [128, 160), B3
[228, 160)). The bin sets of the last row is (B₁ [64, 96). B2 [64, 96), B3 [64,96)). Base
models are DGUNet [50], MPRNet [82], and Restormer [8]
21

---

(b) HR
(c) Bicubic 25.133 / 00.117
(d) SwinIR [39] 26.401 /05578
(c) SRFormer [93]
PSNR SSIM
26.313 / 0.5766
(I) HGBT [34] 26.409 0.5790
(g) Average
(h) ZZPM [92] 26.413 0.5799
(i) Ours w/ b 32
(a) Image
26.414 / 0.5793
26.430 0.5794
Figure 7: visual comparison ensemble on an image from Set14 [83] for the task super- resolution. Please zoom in for better visual quality.
(b)
(c) Bicubic 20.085 / /0.4550
(d) SwinTR [39] 21.396 /.6656
(e) SRFrrmer [93] 21.529 / 0.6496
PSNR SSIM
(f) HGBT [34]
(g) Average 21.878 0.6616
(h) 7ZPM [92] 21.879 006615
(i) Ours w/ b 32
(a) Image
21.883/ 0.6604
21.893 0.6619
Figure 8: A visual comparison of ensemble on an image from Urban 100 [30] for the task of super- resolution. Please zoom in for better visual quality.
(b) PSNR SSIM
(c) Bicubic 20.860 / 0.6756
(d) SwinTR [39] 24.682 / 0.8855
(c) MambalR [27]
25.693 0,9001
HGBT [34] 25.728 0.8987
(g) Average 26. / 0.9021
(h) 77PM [92] 26. 0.9020
(i) Ours w/ b 32
(a) Image
26. 197 0.9023
Figure 9: visual comparison of ensemble on an image from Urban100 [30] for the task of super- resolution. Please zoom in for better visual quality.
(b) HR
(c) Bicubic 21.735 / 0.5589
(d) SwinIR [39] 25.750 / 0.8275
(c) SRFormer [93] 25.706 0.8310
PSNR SSIM
HGBT [34] 26.180 0.8406
(g) Average 26.150 0.8402
(h) ZZPM [92] 26.149/ 0.8401
(i) Ours w/ b 32
(a) Image
26. 26.189/ 0.8409
Figure 10: A visual comparison of ensemble on an image from Urban 100 [30] for the task of super- resolution. Please zoom in for better visual quality.
22

---

(b) Reference
(c) Blurry 24.874 / 0.8636
(d) MPRNet [82] 28.811 0.8800
(c) HGBT [34]
PSNR / SSIM
29.802 0.8936
Average 29.783 0.8941
(g) ZZPM [92] 29.781 00.9935
(h) Ours w/ b 64
(i) Ours w/ b 32
(a) Image
29.840 0.8943 29.864 0.8949
Figure 11: visual comparison of ensemble on an image from HIDE [60] for the task of image deblurring. Please zoom in for better visual quality.
(b) Reference PSNR / SSIM
(c) Blurry 19.158 0.523
(d) MPRNet [82] 21.175 / 0.6451
(e) Restormer [81]
20.814 0.5858
HGBT [34] 21.498 0.6340
(g) Average
(h) 77PM [92] 21.470 0.6366
(i) Ours w/ b 32 21.504 0.6374
(a) Image
21.471 0..377
Figure 12: A visual comparison of ensemble on an image from RealBlur-J [58] for the task image deblurring. Please zoom in for better visual quality.
API
(b) Reference PSNR / SSIM
(c) Blurry 21.782 / 00.911
(d) MPRNet [82] 23.973 0 0.7847
(e) Restormer [81]
24.037 0.7853
HGBT [34] 24.025 0.7848
(g) Average
(h) 77PM [92] 24.040 0.7865
(i) Ours w/ b 32
Image
24.044 / 0.7873
24.079 0.7885
Figure 13: A visual comparison of ensemble on an image from RealBlur-J [58] for the task image deblurring. Please zoom in for better visual quality.
(b) Clean
(c) Rainy 8.018 0.5572
(d) MPRNet [82] 24.528 0.8146
(c) MAXIM [67]
PSNR SSIM
24.968 0.8363
HGBT [34] 26.041 0.8541
(g) Average 26.033 008531
(h) ZZPM [92] 26.034 0.8530
(i) Ours w/ b 32
(a) Image
26.146 0.855
Figure 14: A visual comparison of ensemble on an image from Rain 100H [74] for the task image deraining. Please zoom in for better visual quality.
23

---

(b) Clean
(c) Rainy 27.683 / 0.9886
(d) MPRNet [82] 33.134 0.9488
(c) Restormer [81]
PSNR SSIM
32.765 / 0.9538
HGBT [34] 33.235 0.9562
(g) Average 34.659 / 00.6632
(h) ZZPM [92] 34.656 00,6611
(i) Ours w/ b 32
(a) Image
34.823 0.9658
Figure 15: A visual comparison ensemble on an image from Rain 1001. [74] for the task of image deraining. Please zoom in for better visual quality.
(b) Clean PSNR / SSIM
(c) Rainy 11.270/0.900
(d) MPRNet [82]
(c) Restormer [81]
26.984 / 0.9331
38.342 0.9682
HGBT [34] 38.011 0.9725
(g) Average 34.532 0..720
(h) ZZPM [92] 34.533 0099704
(i) Ours w/ 32 39.525 0.9791
(a) Image
Figure 16: A visual comparison of ensemble on an image from Rain 100L [74] for the task of image deraining. Please zoom in for better visual quality.
(b) Clean
(c) Rainy 24.095 / 0.9310
(d) MAXIM [67]
(e) Restormer [81]
PSNR SSIM
28.236 / 0.9180
32.598 0.9281
(f) HGBT [34]
(g) Average 32.371 0.9309
(h) 77PM [92] 32.380 0.9306
(i) Ours w/ b 32 32.893 0.9310
(a) Image
32.516 0.9286
Figure 17: A visual comparison of ensemble on an image from 1e0 [86] for the task of image deraining. Please zoom in for better visual quality.
(b) Clean
(c) Rainy 24.095 / 0.9310
(d) MPRNet [82] 35.583 0.9669
(c) MAXIM [67]
PSNR SSIM
33.262 0.9700
HGBT [34] 35,950 0.9724
(g) Average 35.926 / 0.9740
(h) ZZPM [92] 35.931 0.97339
(i) Ours w/ b 32
(a) Image
36.318/ 0.9746
Figure 18: A visual comparison of ensemble on an image from Test2800 [86] for the task of image deraining. Please zoom in for better visual quality.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction has accurately reflected the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
We have discussed the limitation of the last page of the main manuscript.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We have provided the complete proof in Appendix and the assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included all implementation and experimental details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have released our codes and all ensemble weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSodmmmiiicc SooodeeubisssioPPooe
/ //nips... /tippscc/pputt wttp:ccccppppppcccccceocococccca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the implementation details in Sec. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The scales the tables are huge and the inclusion of error bars expensive and violate the page limit.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have shown the computer resources in Sec. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttt://een.......ee. cwterrssscccpppeeeeee cc/ppblic/ heooolicc/hiiieeeee htthhhhchhieeeeees...
Answer: [Yes]
Justification: The research in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is social impact of the work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all the original paper referenced to.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "dv2PJylY2o", "reviewer": "Reviewer_EWje", "summary": "This paper reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. Importantly the authors' method achieves state-of-the-art performance without training. I think this work is interesting and can inspire researchers.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed method is more interpretable than traditional deep learning (ensemble learning) methods.\n2.The proposed method obtains state-of-the-art performance compared to other ensemble methods.\n3.The authors performed comparisons on a large number of datasets and image restoration tasks, which fully validated the generalization of the ensemble method.", "weaknesses": "1. I have some confusion. How are the averages in Table III achieved? Why is averaging able to achieve higher results than every single method (non-ensemble method)? Isn't it the average of every single method?\n\n\n2. I would like to know the computational burden of the method?\n\n3.ensemble learning doesn't provide a \"huge\" gain, but it makes sense. I'm just wondering how much extra computation is introduced by this gain? Table 6 reports the runtime, and I would like to know the difference in time between the ensemble and the non-ensemble.\n\n4. It's not easy to see the difference in Figure 1, so the author could add an error map or a different figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q-1.** I have some confusion. How are the averages in Table III achieved? Why is averaging able to achieve higher results than every single method (non-ensemble method)? Isn't it the average of every single method?\n\n**A-1.** Yes, the \"Average\" refers to the average of the results of all single restoration models. Similar to the ensemble approaches in high-level tasks like classification and regression, the ensemble method of averaging also works well in image restoration tasks [Reference 1-4]. The reasons why it could be better than every single method can be interpreted in two aspects. \n\n1. Because image restoration tasks lie in ill-posed problem, the majority of restoration models produce results that deviate from the ground-truth to some extent. Then it could alleviate the deviation issue by averaging the results of several models.\n2. The restoration task could essentially be regarded as multiple prediction/classification problems, where the output values range from 0 to 255 for each pixel. If all base models perform comparably well, their ensemble will be more likely to produce results that match the clean images more closely.\n\n**Q-2.** I would like to know the computational burden of the method?\n\n**A-2.** The computational complexity of our method is $\\mathcal{O}(3HWMT^M)$, where $3HW$ represents the number of pixels, $M$ is the number of base models, and $T$ is the number of bins. The average runtimes of our method with different configurations are presented in Table 1 of the manuscript. \nTable 2 of the PDF rebuttal file illustrates the average runtimes for all steps, including preprocessing an image, inference by each base model, generating the ensemble using our method, and saving the result as an image file. \nTable 6 of the manuscript compares the average runtimes of different ensemble methods.\n\n**Q-3.** Ensemble learning doesn't provide a \"huge\" gain, but it makes sense. I'm just wondering how much extra computation is introduced by this gain? Table 6 reports the runtime, and I would like to know the difference in time between the ensemble and the non-ensemble.\n\n**A-3.** We have measured the runtime of each step, as shown in Table 2 of the rebuttal file. As we can see, our method does not introduce significant additional time (0.1709 seconds compared to a total time of 0.9039 seconds). The time of inferencing by three models is 0.7330 seconds, while the time of our method is 0.1709 seconds.\n\n\n**Q-4.** It's not easy to see the difference in Figure 1, so the author could add an error map or a different figure.\n\n**A-4.** Thank you for your helpful advice. We will revise our figures with additional error maps. For the images in the rebuttal file, we have included error maps for your convenience.\n\n[Reference 1] Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017.\n\n[Reference 2] Ntire 2021 challenge on image deblurring. In CVPR, 2021.\n\n[Reference 3] Ntire 2023 challenge on image super-resolution (x4): Methods and results. In CVPR, 2023.\n\n[Reference 4] Ntire 2024 challenge on image super-resolution (×4): Methods and results. In CVPR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cS4vNwUil7", "reviewer": "Reviewer_j7aF", "summary": "This paper proposes an ensemble algorithm called EnsIR for image restoration tasks using Gaussian mixture models (GMMs). The method partitions pixels into range-wise bins, formulates the ensemble as GMMs over these bins, and solves for ensemble weights using expectation-maximization. The weights are stored in a lookup table for efficient inference. Experiments are conducted on super-resolution, deblurring and deraining tasks.", "questions": "1. Why was bin size 32 chosen as default when 16 performed better in ablations?\n2. How does this method fundamentally differ from and improve upon existing ensemble approaches?\n3. What explains the performance degradation on deraining tasks?", "limitations": "The limitations section is brief and does not adequately address key shortcomings like marginal gains and lack of novelty. A more thorough discussion of limitations would strengthen the paper. Given the marginal improvements, lack of novelty, and limited analysis, I recommend rejecting this paper. Significant revisions would be needed to make this a compelling contribution.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Provides a formulation of image restoration ensemble as GMMs.\n2. Experiments conducted on multiple image restoration tasks and datasets.", "weaknesses": "1.\tMarginal improvement over existing methods: The performance gains are minimal compared to simpler approaches. For example, on the Rain100H dataset (Table 5), the proposed method achieves a PSNR of 31.725, only marginally better than the Average (31.681) and ZZPM (31.679) baselines. In some cases, like on Rain100L, the improvement is less than 0.6 dB over averaging.\n2.\tLack of distinction from existing approaches: The paper does not clearly articulate how this method fundamentally improves upon or addresses limitations of existing ensemble techniques. The core idea of using GMMs and LTU for weighting seems combination of existing approaches, and the paper doesn't adequately explain its novelty.\n3.\tInsufficient analysis of results: There is a notable lack of in-depth analysis of the experiment results. For instance, the paper doesn't discuss why the method performs worse on some deraining tasks (e.g., Test100 in Table 5) compared to HGBT. This lack of analysis makes it difficult to understand the method's strengths and weaknesses.\n4.\tInconsistency in parameter selection: The ablation study in Table 1 shows that a bin width of 16 achieves the best PSNR (31.742), yet the authors choose 32 as the default \"for the balance of efficiency and performance\" (line 220) without adequate justification for this trade-off.\n5.\tLimited efficiency advantages: The paper claims to address efficiency, but Table 6 shows that the proposed method (0.1709s) is significantly slower than Average (0.0003s) and ZZPM (0.0021s) approaches. The efficiency gain over regression-based methods is not a strong selling point given the performance trade-offs.\n6.\tShallow theoretical analysis: While the paper provides derivations in the Appendix, the core idea essentially reduces to a lookup table for ensemble weights. The theoretical contribution and novelty are limited, especially given the marginal performance improvements.\n7.\tOverstated claims: The paper claims to \"consistently outperform\" existing methods (lines 17-19), but this is not supported by the results, particularly in deraining tasks where it underperforms HGBT on some datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q-1.** Marginal improvement over existing methods: The performance gains are minimal compared to simpler approaches. For example, on the Rain100H dataset (Table 5), the proposed method achieves a PSNR of 31.725, only marginally better than the Average (31.681) and ZZPM (31.679) baselines. In some cases, like on Rain100L, the improvement is less than 0.6 dB over averaging.\n\n**A-1.** As noted by Reviewer EWje, our method is an ensemble approach that does not require extra training or fine-tuning, rather than a new architecture or network. Our method is designed for the inference stage and can be applied off-the-shelf to all existing restoration models. It would be unreasonable to expect an ensemble method to achieve improvements over 0.6 dB. However, our method shows more stable and significant improvements over Average and ZZPM, as demonstrated in Tables 3-5. An ensemble method capable of improving performance by 0.2 dB would be beneficial for competition participants.\n\n**Q-2.** Lack of distinction from existing approaches: The paper does not clearly articulate how this method fundamentally improves upon or addresses limitations of existing ensemble techniques. The core idea of using GMMs and LTU for weighting seems combination of existing approaches, and the paper doesn't adequately explain its novelty.\n\n\n**A-2.** The existing ensemble methods do not involve Gaussian Mixture Models (GMM) or Lookup Tables (LUT),  while our method addresses the ensemble problem in image restoration using GMM, the Expectation-Maximization (EM) algorithm, and LUT. Unlike previous works, our derivation shows that ensemble in image restoration can be transformed into multiple GMM problems, where the weights of the GMMs serve as the ensemble weights.\n\nWe then leverage a modified EM algorithm to solve the GMMs, with the means of each Gaussian distribution known as prior knowledge. LUT is used to save the estimated weights for inference. Our contributions lie in deriving the restoration ensemble problem into GMMs, modifying the EM algorithm to solve these GMMs, and ultimately proposing a novel ensemble method for image restoration.\n\n**Q-3.** Insufficient analysis of results: There is a notable lack of in-depth analysis of the experiment results. For instance, the paper doesn't discuss why the method performs worse on some deraining tasks (e.g., Test100 in Table 5) compared to HGBT. This lack of analysis makes it difficult to understand the method's strengths and weaknesses.\n\n**A-3.** In Table 5, our method (32.002 dB, 0.9268) clearly outperforms HGBT (31.988 dB, 0.9241) on the Test100 dataset.\n\nRegarding the analysis of results, we have discussed its limitations, such as \"if all base models fail, ensemble methods cannot generate a better result,\" in Section 4.2.4. We illustrated the ensemble weights, image features, and pixel distributions in Figures 4-6 in the Appendix. Additionally, we analyzed the scenario where one model may consistently outperform others in Section 4.2.2.\n\n**Q-4.** Inconsistency in parameter selection: The ablation study in Table 1 shows that a bin width of 16 achieves the best PSNR (31.742), yet the authors choose 32 as the default \"for the balance of efficiency and performance\" (line 220) without adequate justification for this trade-off.\n\n**A-4.** As noted in Table 1 of the manuscript, it is slow (1.2460 seconds per image) when using a bin width of 16. When dealing with thousands of images, it would takes hours to obtain ensemble results for a test set. Therefore, we chose a bin width of 32 which is over seven times faster than the case of 16. The method is designed for real-world industrial scenarios, so balancing efficiency and performance is a crucial reason for choosing a bin width of 32.\n\n\n**Q-5.** Limited efficiency advantages: The paper claims to address efficiency, but Table 6 shows that the proposed method (0.1709s) is significantly slower than Average (0.0003s) and ZZPM (0.0021s) approaches. The efficiency gain over regression-based methods is not a strong selling point given the performance trade-offs.\n\n**A-5.** From Table 1, 5, and 6 of the manuscript, our method with a bin width of 128 still outperforms ZZPM (31.702 vs. 31.679 on Rain100H), and is comparably fast compared to ZZPM (0.0059 seconds vs. 0.0021 seconds). This demonstrates both the efficiency and performance gains of our method.\n\n\n**Q-6.** Shallow theoretical analysis: While the paper provides derivations in the Appendix, the core idea essentially reduces to a lookup table for ensemble weights. The theoretical contribution and novelty are limited, especially given the marginal performance improvements.\n\n**A-6.** The core idea of our method is **not related to the lookup table**. The core idea is to transform the ensemble problem of image restoration into multiple Gaussian mixture models and then solve it using a modified EM algorithm. The lookup table is simply used to store the solved weights for inference.\n\n**Q-7.** Overstated claims: The paper claims to \"consistently outperform\" existing methods (lines 17-19), but this is not supported by the results, particularly in deraining tasks where it underperforms HGBT on some datasets.\n\n**A-7.** Although our method achieves top performance in 27 out of 28 metrics across 14 datasets, which can be considered as \"consistently outperforming\", we will revise \"consistently\" to \"overall\".\n\n\n**Question 1.** Why was bin size 32 chosen as default when 16 performed better in ablations?\n\n**Answer 1.** Please refer to **A-4.**\n\n**Question 2.** How does this method fundamentally differ from and improve upon existing ensemble approaches?\n\n**Answer 2.** Please refer to **A-2** and **A-6**.\n\n**Question 3.** What explains the performance degradation on deraining tasks?\n\n**Answer 3.** Please refer to **A-3.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tmgsJwwKV2", "reviewer": "Reviewer_E4or", "summary": "A novel post-training ensemble learning method for image restoration is developed in this work by employing Gaussian mixture models and the EM algorithm to generate better restoration results. The authors reformulate the ensemble problem of image restoration into various gaussian mixture models, use the EM algorithm to estimate range-wise ensemble weights on a reference set, and store the weights in a lookup table during inference. The method effectively improves ensemble results on 14 benchmarks and 3 restoration tasks, including super-resolution, deblurring and deraining.", "questions": "- What is the result of the case with 2 or 4 base models, instead of 3?\n\n- What are the results for dehazing or enhancement?", "limitations": "Limitations are mentioned on Page 9 and include the \"trade-off between runtime and performance.\" A discussion and possible solution are provided.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is grounded on a reasonable derivation and assumption of Gaussian prior to restoration. The visualization in Fig. 6 also validates the assumption.\n\n- The method performs consistently well on the mentioned 14 datasets.\n\n- Extensive experiments and ablation studies show the features of the developed method.", "weaknesses": "- The experiments all show the cases of three base methods, and one of them may be worse than the other two methods. But if all the base models are comparably good, can the method surpass other methods? Experiments with 2 or 4 base models, instead of 3, would be informative.\n\n- The experiments are for the restoration task of local patterns such as deraining and deblurring. However, dehazing or enhancement requires a glocal restoration. What are the results of the ensemble comparisons for dehazing or enhancement? This can provide better insights into the generalizability of the method.\n\n- Some notations are duplicate or misused, like \"L\" is used for log-likelihood in Eq. 19 and also for the dimension length of the image. In Algorithm 2, both \"Equation\" and \"Eq. \" are used. The notations should be revised.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q-1.** The experiments all show the cases of three base methods, and one of them may be worse than the other two methods. But if all the base models are comparably good, can the method surpass other methods? Experiments with 2 or 4 base models, instead of 3, would be informative.\n\n**A-1.** Thank you for your advice. We have conducted additional experiments using two and four base models for the task of deblurring. The experimental results are shown in the last two columns of Table 1 in the rebuttal file. \n\nIn the case of two base models, the ZZPM method reduces to Average since their weights are equal. For the case of four base models, we selected NAFNet [Reference 1] as the fourth base model. From the experimental results, it is evident that our method consistently achieves the best ensemble results, whether using two or four base models.\n\n\n**Q-2.** The experiments are for the restoration task of local patterns such as deraining and deblurring. However, dehazing or enhancement requires a glocal restoration. What are the results of the ensemble comparisons for dehazing or enhancement? This can provide better insights into the generalizability of the method.\n\n**A-2.** Thank you for your advice. We have conducted additional experiments on low-light image enhancement (LLIE) and dehazing. The experimental results are shown in the first two columns of Table 1 in the rebuttal file. We also provide two visual comparisons in Figure 1 and 2 of the rebuttal file. \n\nThe datasets used for the experiments are LOLv1 [Reference 2] for LLIE and OTS [Reference 3] for dehazing. The pre-trained models are provided by their authors. For your convenience, we also include error maps between the restoration results and ground-truths, along with the visual results. Darker error maps indicate better performance. \nFrom the results, it is evident that our method outperforms other ensemble methods in both quantitative and qualitative measures.\n\n\n**Q-3.** Some notations are duplicate or misused, like \"L\" is used for log-likelihood in Eq. 19 and also for the dimension length of the image. In Algorithm 2, both \"Equation\" and \"Eq. \" are used. The notations should be revised.\n\n**A-3.** Thank you for your careful suggestions. We will revise them accordingly.\n\n**Question 1.** What is the result of the case with 2 or 4 base models, instead of 3?\n\n**Answer 1.** Please refer to **A-1** and the uploaded rebuttal file.\n\n**Question 2.** What are the results for dehazing or enhancement?\n\n**Answer 2.** Please refer to **A-2** and the uploaded rebuttal file.\n\n[Reference 1]  Simple baselines for image restoration. In ECCV, 2022.\n\n[Reference 2] Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560, 2018.\n\n[Reference 3] Benchmarking singleimage dehazing and beyond. IEEE TIP, 28(1):492–505, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgSuAdjBwV", "reviewer": "Reviewer_Aksr", "summary": "This paper proposes a post-training model ensenble method for image restorationn by leveraging Gaussian mixture models on split pixels and lookup table for fast innference. The pixels are split into various bin sets according to their value ranges and the ensemble problem of the pixels is reformulated into Gaussian mixture models. The EM algotithm is used to solve the ensemble weight of the models. This area lacks sufficient research attention and previous works are mainly using averaging or its variant. So this work has potential influence for industrial application. The proposed method is fast and effective to obtain better ensemble results for three image restoration tasks.", "questions": "I have listed some questions regarding the implementation and experiments in Weakness section.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The method is novel, properly derived and has a theoretical support. It is well ellaborated in pseudo-codes and discussed.\n2. The authors have conducted extensive experiments on three restoration tasks, including 14 benchmarks.\n3. Its results are promising and consistently better than other baselines.", "weaknesses": "1. ZZPM is a new and recently proposed method after 2022, and averaging is the most straightforward and traditional method. Why is its performance sometimes worse than the normal averaging? \n2. For the task of super-resolution, the input and output are not in the same size, so the notations should be clarified.\n3. Why can we suppose that image noise follows a Gaussian distribution with zero mean? Is there any clue or derivation to support the assumption?\n4. The authors say that the method is not accelerated by GPU vectorization yet. Is it because CPU is faster or any other reasons? \n5. The pixel numbers of each bin set will be highly diverse as shown in Figure 6. In some bin set, there will be many pixels and then EM algorithm can run properly without problem. However, if there are too few pixels in a bin set, EM algorithm may fail to find a solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q-1.** ZZPM is a new and recently proposed method after 2022, and averaging is the most straightforward and traditional method. Why is its performance sometimes worse than the normal averaging?\n\n**A-1.** Thank you for your insight. ZZPM was developed for the latest image restoration competition [Reference 1], where all base models are similar in terms of their structure and performance. However, as mentioned in L240-243, our experiments are conducted using models with different structures and sometimes diverse performances on several test sets. In such scenarios, one advanced model may consistently outperform the others. ZZPM assigns ensemble weights that are negatively proportional to the mean squared error between the result of a base model and the average of the base model results. This approach can exaggerate deviations from the optimal prediction.\n\n\n**Q-2.** For the task of super-resolution, the input and output are not in the same size, so the notations should be clarified.\n\n**A-2.** Thank you for your careful suggestion. We will revise the notation.\n\n**Q-3.** Why can we suppose that image noise follows a Gaussian distribution with zero mean? Is there any clue or derivation to support the assumption?\n\n**A-3.** Thank you for your insightful question. The Gaussian prior is commonly adopted in image reconstruction or restoration tasks [Reference 2-3]. The assumption that noise follows a Gaussian distribution can derive the mean squared error (MSE) loss or L2 norm we commonly used as loss function. If we assume it follows a Laplacian distribution, it would yield a L1 norm. The derivation is provided below.\n\n\n>Suppose the error between the ground-truth $y_n$ and the restoration result $f(x_n)$ by model $f$, follows zero-mean Gaussian, i.e., $\\epsilon_n = y_n - f(x_n) \\sim \\mathcal{N}(0, \\sigma^2)$. Then we have the log likelihood\n>$\\sum_{n=1}^N \\log P(\\epsilon_n)=-\\frac{N}{2}\\log(2\\pi) - N\\log\\sigma -\\sum_{n=1}^N \\frac{(y_n - f(x_n))^2}{2\\sigma^2}.$\n>Because we are optimizing the restoration model $f$, the objective can be simplified into the squared L1 norm loss $\\sum_{n=1}^N (y_n - f(x_n))^2$. The derivation is similar for Laplace and L1 norm.\n\n**Q-4.** The authors say that the method is not accelerated by GPU vectorization yet. Is it because CPU is faster or any other reasons?\n\n**A-4.** The reason is that the implementation of method is related to the number of base models. It involves the nesting of several ```for``` loops whose the number is dynamic and dependent on the number of base models, making it difficult to vectorize. \n\n\n**Q-5.** The pixel numbers of each bin set will be highly diverse as shown in Figure 6. In some bin set, there will be many pixels and then EM algorithm can run properly without problem. However, if there are too few pixels in a bin set, EM algorithm may fail to find a solution.\n\n**A-5.** Indeed, the EM algorithm requires a sufficient number of sample points. If the number of pixels in a bin set is insufficient, the EM algorithm will fail to solve the GMM. In this case, we adopt the average as the default method to assign equal ensemble weights. The setting is described in L216-217.\n\n\n[Reference 1] Ntire 2023 challenge on image super-resolution (x4): Methods and results. In CVPR, 2023.\n\n[Reference 2] Deep Gaussian Scale Mixture Prior for Image Reconstruction. IEEE TPAMI, 2023.\n\n[Reference 3] Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules. In CVPR, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers, ACs, SACs, and PCs for their effort and attention. We have uploaded a rebuttal file in PDF format to illustrate the tables and figures. 

Table 1 presents the experimental results of the ensemble for four tasks, i.e., low-light image enhancement (LLIE), dehazing, deblurring with two base models, and deblurring with four base models. Table 2 measures the time cost of each step. Figure 1 and 2 show the visual comparisons of ensemble for LLIE and dehazing, respectively. 

Due to space limitations, we list the newly cited works, including LOLv1 [1], OTS [2], RetinexFormer [3], RQ-LLIE [4], CIDNet [5], MixDehazeNet [6], DEA-Net [7], C2PNet [8], and NAFNet [9].


[1] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560, 2018.

[2] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. Benchmarking singleimage dehazing and beyond. IEEE TIP, 28(1):492–505, 2018.

[3] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One-stage retinex-based transformer for low-light image enhancement. In ICCV, 2023. 

[4] Yunlong Liu, Tao Huang, Weisheng Dong, Fangfang Wu, Xin Li, and Guangming Shi. Low-light image enhancement with multi-stage residue quantization and brightness-aware attention. In ICCV, 2023. 

[5] Yixu Feng, Cheng Zhang, Pei Wang, Peng Wu, Qingsen Yan, and Yanning Zhang. You only need one color space: An efficient network for low-light image enhancement. arXiv preprint arXiv:2402.05809, 2024. 

[6] LiPing Lu, Qian Xiong, DuanFeng Chu, and BingRong Xu. Mixdehazenet: Mix structure block for image dehazing network. arXiv preprint arXiv:2305.17654, 2023. 

[7] Zixuan Chen, Zewei He, and Zhe-Ming Lu. Dea-net: Single image dehazing based on detail-enhanced convolution and content-guided attention. IEEE TIP, 2024. 

[8] Yu Zheng, Jiahui Zhan, Shengfeng He, Junyu Dong, and Yong Du. Curricular contrastive regularization for physics-aware single image dehazing. In CVPR, 2023.

[9] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In ECCV, 2022.

Meta-review: The reviewers agreed that the paper is well written and presents a novel method for ensemble-based prediction. The method has theoretical support, and the evaluation is extensive and demonstrates a consistent improvement. 
There were some concerns about the resulting improvement being only marginal, and the limited practicality of the approach given the complexity of running many different base models, however the reviewers in general were in favour of accepting the paper.
Decision: Accept (poster)