EnsIR: An Ensemble Algorithm for Image Restoration via Gaussian Mixture Models

Authors: shangquan  sun
wenqi  ren
zikun  liu
hyunhee  park
rui  wang
xiaochun  cao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 20:54:27

Abstract: Image restoration has experienced significant advancements due to the development of deep learning. Nevertheless, it encounters challenges related to ill-posed problems, resulting in deviations between single model predictions and ground-truths. Ensemble learning, as a powerful machine learning technique, aims to address these deviations by combining the predictions of multiple base models. Most existing works adopt ensemble learning during the design of restoration models, while only limited research focuses on the inference-stage ensemble of pre-trained restoration models. Regression-based methods fail to enable efficient inference, leading researchers in academia and industry to prefer averaging as their choice for post-training ensemble. To address this, we reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. We estimate the range-wise ensemble weights on a reference set and store them in a lookup table (LUT) for efficient ensemble inference on the test set. Our algorithm is model-agnostic and training-free, allowing seamless integration and enhancement of various pre-trained image restoration models. It consistently outperforms regression-based methods and averaging ensemble approaches on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. The codes and all estimated weights have been released in Github.

Main paper: 
 introduction: 
 Image restoration has witnessed significant progress over the decades, especially with the advent of deep learning. Numerous architectures have been developed to solve the problem of restoration, including convolutional neural networks (CNNs) [10, 82] , vision Transformers (ViTs) [39, 81, 93] and recently, vision Mambas [27] . However, single models with different architectures or random initialization states exhibit prediction deviations from ground-truths, resulting in sub-optimal restoration results.To alleviate this problem, ensemble learning, a traditional but influential machine learning technique, has been applied to image restoration. It involves combining several base models to obtain a better result in terms of generalization and robustness [18, 20, 26, 49, 57] . However, most ensemble methods in image restoration focus on training-stage ensemble requiring the ensemble strategy to be determined while training multiple models, thus sacrificing flexibility of changing models and convenience for plug-and-play usage [31, 35, 40, 44, 46, 54, 59, 70, 76] . In contrast, there is a demand for advanced post-training ensemble methods in the image restoration industry, where researchers still prefer averaging as their primary choice [1, 15, 41, 52, 66, 68, 92] .Despite the industrial demand, post-training ensemble in image restoration is challenging for tradition ensemble algorithms originally designed for classification or regression. Unlike classification and regression, image restoration predictions are matrices with each pixel is correlated with others and range from 0 to 255. As a result, traditional methods like bagging [6] and boosting [28] either require enormous computational resources for the restoration task or fail to generalize well due to the imbalance between candidate number and feature dimension. As an alternative, Jiang et al. propose a post-training ensemble algorithm for super-resolution by optimizing a maximum a posteriori problem with a reconstruction constraint [31] . However, this constraint requires an explicit expression of the degradation process, which is extremely difficult to define for other restoration tasks beyond super-resolution. It also necessitates prior knowledge of the base models' performance, further limiting its practical application. These issues with both traditional and recent ensemble methods lead researchers in image restoration to prefer weighted averaging as their primary ensemble approach [1, 15, 52, 68, 92] .To this end, we formulate the ensemble of restoration models using Gaussian mixture models (GMMs), where ensemble weights can be efficiently learned via the expectation maximization (EM) algorithm and stored in a lookup table (LUT) for subsequent inference. Specifically, we first base on the Gaussian prior that assumes the prediction error of each model follows a multivariate Gaussian distribution. Based on the prior, the predictions of multiple samples can be appended into a single variable following Gaussian. By partitioning pixels into various histogram-like bins based on their values, we can convert the problem of ensemble weight estimation into various solvable univariate GMMs. As a result, the univariate GMMs can be solved to obtain range-wise ensemble weights by the EM algorithm, with the means and variances of Gaussian components estimated based on the observation priors. We estimate these weights on a reference set and store them in a LUT for efficient inference on the test set. Our method does not require training or prior knowledge of the base models and degradation processes, making it applicable to various image restoration tasks.Our contributions mainly lie in three areas:• Based on the Gaussian prior, we partition the pixels of model predictions into range-wise bin sets of mutually exclusive ranges and derive the ensemble of multi-model predictions into weight estimation of various solvable univariate GMMs.• To solve the univariate GMMs and estimate ensemble weights, we leverage the EM algorithm with means and variances initialized by observed prior knowledge. We construct a LUT to store the range-wise ensemble weights for efficient ensemble inference.• Our ensemble algorithm does not require extra training or knowledge of the base models. It outperforms existing post-training ensemble methods on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. 
related works: 
 Ensemble Methods. Ensemble methods refer to approaches that fuse the predictions of multiple base models to achieve better results than any of individual model [20] . Traditional ensemble methods include bagging [6] , boosting [28] , random forests [7] , gradient boosting [24] , histogram gradient boosting [14, 34] , etc. These methods have been applied to various fields in classification and regression, such as biomedical technology [77, 78] , intelligent transportation [53, 84] , and pattern recognition [61, 91] .Image restoration. Image restoration, as a thriving area of computer vision, has been making significant progress since the advent of deep learning [10, 65] . Various model architectures have been proposed to address image restoration tasks, such as convolutional neural networks (CNNs) [19, 56, 73, 90] , multilayer perceptron (MLPs) [67] , vision Transformers (ViTs) [39, 62, 81] , etc. Additionally, various structural designs like multi-scale [56] , multi-patch [64, 89] , and progressive learning [82] have been adopted to improve the representative capacity. It is known that CNNs excel at encoding local features, while ViTs are adept at capturing long range dependencies. Despite this significant progress, single models still generate predictions that deviate from ground-truths, leading researchers in industry to adopt multi-model ensembles to achieve better performance [15, 52, 68, 92] .Ensemble Learning in Image Restoration. Some works incorporate ensemble learning into image restoration by training multiple networks simultaneously and involving ensemble strategy during the training process [2, 3, 9, 11-13, 16, 17, 21, 29, 31, 33, 35-38, 40, 42-46, 54, 55, 59, 63, 69, 70, 72, 75, 76, 79, 80, 85, 87] . However, the majority of them require additional training or even training from scratch alongside ensemble. Only a few focus on ensemble at the post-training stage [31, 66] . Among them, self-ensemble [66] geometrically augments an input image, obtains super-resolution predictions of augmented candidates and applies averaging ensemble to the candidates, which is orthogonal to the scope of our work. RefESR [31] requires a reconstruction objective which must get access to the degradation function, prohibiting its application to tasks other than super-resolution.In general, limited works focus on the training-free ensemble of image restoration. There lacks a general ensemble algorithm for restoration despite industry demand [1, 15, 52, 68, 92] . 
proposed ensemble method for image restoration: 
 In Sec. 3.1, we first present the formulation of the ensemble problem in image restoration. We then formulate our ensemble method in the format of Gaussian mixture models (GMMs) over partitioned range-wise bin sets in Sec. 3.2. We derive the expectation maximization (EM) algorithm with known mean and variance as prior knowledge to solve the GMMs problems in Sec. 3.3Given a test set T = { Xn , Ŷn } with numerous pairs of input images and ground-truths, suppose we have M pre-trained base models for image restoration, f 1 , ..., f M . For a model f m where m ∈ {1, ..., M }, its prediction is denoted by Xm,n = f m ( Xn ) for abbreviation. We consider the ensemble of the predictions as a weighted averaging, i.e.,EQUATIONwhere Ỹn is the ensemble result, and β n ∈ R M is the vector of weight parameters for the ensemble. The widely-used averaging ensemble strategy in image restoration assigns equal weights for all samples and pixels, i.e.,β n = 1 M • • • 1 M .A recent method in the NTIRE 2023 competition assigns weights inversely proportional to the mean squared error between the predictions and their average [92] . However, they adopt globally constant weights for all pixels and samples, neglecting that the performances of base models may fluctuate for different patterns and samples.Alternatively, we start from the prospective of GMMs and assign range-specific weights based on the EM algorithm.Similar to RefESR [31] , suppose we have a reference set D = {X n , Y n } N n=1 with N pairs of input images and ground-truths. We assume the reference set and test set are sampled from the same data distribution, i.e., D, T ∼ D.For each model f m , its prediction is denoted by X m,n = f m (X n ) ∈ R 3×H×W . We use x m,n , y n ∈ R L to represent the flattened vector of the matrices X m,n , Y n , where L = 3HW . Based on Gaussian prior, it can be assumed that the estimation error of a model on an image follows a zero-mean Gaussian distribution, namely ϵ m,n = y n -x m,n ∼ N (0, Σ m,n ), where Σ m,n ∈ R L×L is the covariance matrix of the Gaussian. Then the observed ground-truth can be considered following a multivariate Gaussian with the mean equal to the prediction, i.e., y n |f m , x n ∼ N (x m,n , Σ m,n ).We can consider the ensemble problem as the weighted averaging of Gaussian variables and estimate the weights by solving its maximum likelihood estimation. However, solving the sample-wise mixture of Gaussian is not feasible because the covariance matrices are sample-wise different and thus hard to estimate. Besides, the number of prediction samples is much fewer than feature dimension, resulting in the singularity of the covariance matrices. Please refer to Sec. A.1 in Appendix for details.In contrast, we alternatively append the reference set into a single sample following Gaussian asEQUATIONwherey 1:N = [y 1 • • • y N ] ∈ R N L and x m,1:N = [x m,1 • • • x m,N ] ∈ R N Lare the concatenation of observed ground-truths and restored samples respectively. Since data samples can be considered following i.i.d data distribution D, the variance of the concatenated samples is diagonal.However, the covariance matrix is still singular due to the imbalance between prediction sample number and feature dimension. Thus, directly mixing the multivariate Gaussian is still infeasible to solve. We thus alternatively categorize pixels into various small bins of mutually exclusive ranges such that the pixels within each range can be considered following a univariate Gaussian distribution according to the central limit theorem. Concretely, we separate the prediction range of models into T bins with each of width b, i. EQUATIONwhere the operation {•} denotes the element-wise product. By the central limit theorem, we assume the nonzero pixels within each bin follow a Gaussian distribution with the mean µ r,m,1:N and variance σ r,m,1:N , i.e.,EQUATIONwhere N r is the number of nonzero pixels in R r such thatT Mr=1 N r = N L, and the values of µ r,m,1:N and σ r,m,1:N can be estimated by the mean and variance of N r prediction pixels within the current bin set.The reference set is therefore separated into T M number of bin sets, and the ground-truth pixels inside each of them form a solvable univariate GMM. We then introduce the latent variable z such that z = m if the pixel y EQUATIONwhere P (y(i) r,1:N |z = m ) = ϕ(y (i) r,1:N ; µ r,m,1:N , σ r,m,1:N ) is the density function of Gaussian N (µ r,m,1:N , σ r,m,1:N ).The value of ensemble weights can be estimated by the maximum likelihood estimates of the observed ground-truths, i.e., {α r,m } r,m ∈ arg max P (y 1:N ).Because arbitrary two bin sets are mutually exclusive, we can safely split the optimization of maximum likelihood over y 1:N into T M optimization problems of maximum likelihood over y r,1:N . Each of them is formulated asEQUATIONWe have formulated the expression of GMMs for estimating the range-specific ensemble weights.For each bin set (B 1 , ..., B M ), we estimate ensemble weights by maximizing the log likelihood asEQUATIONWe have an E-step to estimate the posterior distribution byEQUATION)After that, we have an M-step to obtain the maximum likelihood estimates byEQUATION)Thanks to the separation of bin sets, we have prior knowledge of the mean and variance of each model, which can be estimated and initialized byEQUATION)The complete and detailed derivation of the EM algorithm can be found in Sec. A.3 of Appendix.We store the range-specific weights estimated on the reference set into a LUT with each key of (B 1 , ..., B M ). During the inference stage for a test sample Xn , we have the prediction of the m-th base model as Xm,n = f m ( Xn ). For a bin set (B 1 , ..., B M ), we partition input pixels of multiple models into each bin set asEQUATION)Then we retrieve the estimated range-wise weights α r,m from the LUT based on each key of the bin set and obtain the aggregated ensemble byEQUATIONThe main algorithm can be found in Algo. 1 and the EM algorithm with known mean and variance priors is shown in Algo. 2. 
experimental settings: 
 Algorithm 2: MPEM: EM algorithm with known Mean Prior Input: y r,1:N , x r,1,1:N , ..., x r,M,1:N Output: (α r,1 , ..., α r,M ) 1 N r ← number of nonzero pixels in y r,1:N ; 2 Initialize µ r,m,1:N by Eq. 12;3 Initialize σ r,m,1:N by Eq. 13; 4 while not converge do5 for i ∈ [1, N r ] do 6 for m ∈ [1, M ] do 7Update γ r,m,1:N by Eq. 9; Benchmarks. We evaluate our ensemble method on 3 image restoration tasks including super-resolution, deblurring, and deraining. For super-resolution, we use Set5 [5] , Set14 [83] , BSDS100 [47] , Urban100 [30] and Manga109 [48] as benchmarks. For deblurring, we use GoPro [51] , HIDE [60] , RealBlur-J and -R [58] . For deraining, we adopt Rain100H [74] , Rain100L [74] , Test100 [88] , Test1200 [86] , and Test2800 [25] .Metrics. We use peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) [71] to quantitatively evaluate the image restoration quality. Additionally, we compare the average runtime per image in seconds to evaluate the ensemble efficiency. Following prior works [27, 39, 67, 81, 82, 93] , PSNR and SSIM are computed on Y channel of the YCbCr color space for image super-resolution and deraining.Base Models. To evaluate the generalization of ensemble methods against model choices, we employ a wide variety of base models, including CNNs, ViTs, MLPs and Mambas. For image super-resolution, we use SwinIR [39] , SRFormer [93] , and MambaIR [27] . We choose MPRNet [82] , DGUNet [50] , and Restormer [81] for deblurring, as well as MPRNet [82] , MAXIM [67] , and Restormer [81] for deraining. Baselines. We utilize regression algorithms including bagging [6] , AdaBoost [22] , random forests (RForest) [7] , gradient boosting decision tree (GBDT) [23] , histogram gradient boosting decision tree (HGBT) [14, 34] as baselines. Averaging is also a commonly used ensemble baseline. A recent method proposed by team ZZPM in the NTIRE 2023 competition [92] is also included for comparison. Additionally, we adopt RefESR [31] for image super-resolution ensemble.Implementation Details. We choose the bin width as 32 by default for the balance of efficiency and performance. The EM solver of GMM stops after 1000 iterations or when the change of log likelihood is less than 1e -5 . For cases where N r is fewer than 100 or the EM solution is undetermined, we use averaging weights by default. The values are reported by taking the average of 4 trials. For the construction of the reference set, we randomly select one image from the training set of DIV2K [4] for super-resolution, while for deblurring and deraining, we sample 10 images from the training sets of GoPro [51] and Rain13K [32] , respectively. All the experiments are run in Python on a device with an 8-cores 2.10GHz Intel Xeon Processor and 32G Nvidia Tesla V100. The regression-based ensemble algorithms are implemented based on scikit-learn [8] . We conduct ablation studies on the bin width b and the maximum step number of the EM algorithm on the benchmark of Rain100H [74] . The results are shown in Tab. 1 and Tab. 2 respectively. We can observe that the bin width involves a trade-off between ensemble accuracy and the efficiency .A small b generates better ensemble results but makes ensemble inference slower. In the following experiments, we choose the bin width b = 32 by default for the balance of efficiency and performance.On the other hand, a large maximum step number does not necessarily yield better ensemble result. Therefore, we choose 1000 as the maximum step number.We present quantitative comparisons with existing ensemble methods in Tab. 3 for super-resolution, Tab. 4 for deblurring, and Tab. 5 for deraining. Our method generally outperforms all existing methods across the 3 image restoration tasks and 14 benchmarks. Note that Average and ZZPM [92] generally perform better than regression-based ensemble methods. However, in cases where one of the base models significantly underperforms compared to the others, such as MPRNet [82] on Rain100L [74] and Test100 [88] , these regression methods outperform Average and ZZPM [92] . In contrast, our method, which learns per-value weights, can recognize performance biases and alleviate such issue. ZZPM [92] performs comparably to Average in our experiments rather than outperforming it, because the base models are not always equally good and one model may be consistently better than the others. Thus, weights negatively proportional to the mean squared error may exaggerate deviation from optimal prediction. In contrast, our method consistently performs well for all cases.We also provide qualitative visual comparisons in Fig. 1 , 2 and 3. In Fig. 1 , the base model SwinIR [39] mistakenly upscales the character's eye. While existing ensemble algorithms partially alleviate this mistake, they cannot fully discard the hallucinated line inside the eye. In contrast, our method with the bin width b = 32 that learns fine-grained range-wise weights successfully recovers the pattern. In Fig. 2 , only our method can effectively obtain a better ensemble with sharp edges and accurate colors. In Fig. 3 , we can observe that MPRNet [82] removes all stripe patterns on the ground together with rain streaks. The conventional weighted summation yields a dimmer ensemble result, and the HGBT method fails to learn accurate weight distributions, resulting in an unsmooth result. In contrast, ours alleviates the issue. More visual comparisons are provided in Fig. 7 -18 in Appendix.Efficiency. The efficiency of ensemble methods is compared by measuring the average runtime on Rain100H [74] , as shown in Tab. 6. Although our method is slower than Average and ZZPM [92] , Table 6 : The average runtime per image in seconds of the ensemble methods on Rain100H [74] .Method Bagging [6] AdaBoost [22] RForest [7] GBDT [23] HGBT [34] Average ZZPM [92] Visualization. We also present the visualization examples of ensemble weights, image features and pixel distributions in Fig. 4 -6 in Appendix. Due to page limit, please refer to Sec. B of Appendix.Limitation and Future Work. The trade-off between the runtime and performance has not been solved yet. Achieving real-time ensemble would lead to performance degradation. The issue could be resolved by GPU vectorization acceleration and distributed computing. Additionally, if all base models fail, ensemble methods cannot generate better result. We leave them in the future work. 
conclusion: 
 In this paper, we propose an ensemble algorithm for image restoration based on GMMs. We partition the pixels of predictions and ground-truths into separate bins of exclusive ranges and formulate the ensemble problem using GMMs over each bin. The GMMs are solved on a reference set, and the estimated ensemble weights are stored in a lookup table for the ensemble inference on the test set.Our algorithm outperforms regression-based ensemble methods as well as commonly used averaging strategies on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. It is training-free, model-agnostic, and thus suitable for plug-and-play usage.The log likelihood is thereforeEQUATIONWe can compute the optimal maximum likelihood estimate by making ∂L ∂βm = 0. Because we only have M observed samples while N L features with N L ≫ M for each prediction sample. The unknown covariance matrix still cannot be estimated by multivariate GMMs with the EM algorithm,. The estimation of covariance requires the computation of its inverse. The covariance matrix is singular and thus the multivariate GMMs cannot be solved.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have released our codes and all ensemble weights.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We have specified all the implementation details in Sec. 4. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For each bin set, we have converted the problem into the format of GMMs, i.e., P (yEQUATIONwhere z ∈ {1, ..., M } is the latent variable that represents the probability of the m-th mixture, and P y (i) r,1:N z = m is the m-th mixture probability component. We use α r,m = P (z = m) to denote the mixture proportion or the probability that y (i) r,1:N belongs to the m-th mixture component. We assume the pixels within the bin follow the Gaussian distribution based on the central limit theorem, i.e., y EQUATIONPlug the expression of P (z = m) and P y (i)r,1:N ; µ r,m,1:N , σ r,m,1:N ) P (z = m|yEQUATION)where the inequality is based on the Jensen's Inequality.The problem can be effectively solved by the EM algorithms. We have the an E-step to estimate posterior distribution of z by γ r,m,1:N ← P z = m|y We do not update the mean because we have its prior knowledge of value as the mean of base model prediction pixels within the bin set.Let θ = {α r,1 , ..., α r,M , σ r,1,1:N , ..., σ r,M,1:N } be the estimate variable. To validate the convergence of GMMs, we want to prove P y The second column shows the ensemble weights and result of the averaging strategy. The third column shows the ensemble weights and result of ZZPM [92] . The last column shows the ensemble weights and result of our method.EQUATIONWe show more visual comparisons in Fig. 7 -10 for image super-resolution, in Fig. 11 -13 for image deblurring, and in Fig. 14 -18 for image deraining.In Fig. 7 , only ours is able to preserve gray reflection of the boat from SwinIR [39] 's prediction. In Fig. 8 -10, our method obtains the most accurate ensemble results in the case that one of base models generates mistaken textures.In Fig. 11 , our method yield the sharpest and straight line like the ground-truth. In Fig. 12 and 13 , our ensemble method obtains the closest textures to the ground-truths.In Fig. 14 , our method gets the best ensemble despite the mistake from MAXIM [67] . In Fig. 15 , ours preserves the cleanest background to the ground-truth. In Fig. 16 , only ours recovers the grid near the eyeball. In Fig. 17 Figure 5 : A sampled group of feature visualizations on HIDE [60] . "Base" denotes the features of three base models, i.e., DGUNet [50] , MPRNet [82] , and Restormer [81] . • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The reviewers agreed that the paper is well written and presents a novel method for ensemble-based prediction. The method has theoretical support, and the evaluation is extensive and demonstrates a consistent improvement. 
There were some concerns about the resulting improvement being only marginal, and the limited practicality of the approach given the complexity of running many different base models, however the reviewers in general were in favour of accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. Importantly the authors' method achieves state-of-the-art performance without training. I think this work is interesting and can inspire researchers.
strength_weakness: strength_weakness: 1. I have some confusion. How are the averages in Table III achieved? Why is averaging able to achieve higher results than every single method (non-ensemble method)? Isn't it the average of every single method?


2. I would like to know the computational burden of the method?

3.ensemble learning doesn't provide a "huge" gain, but it makes sense. I'm just wondering how much extra computation is introduced by this gain? Table 6 reports the runtime, and I would like to know the difference in time between the ensemble and the non-ensemble.

4. It's not easy to see the difference in Figure 1, so the author could add an error map or a different figure.
questions: questions: see weaknesses
limitations: limitations: see weaknesses
paper_summary: paper_summary: This paper proposes an ensemble algorithm called EnsIR for image restoration tasks using Gaussian mixture models (GMMs). The method partitions pixels into range-wise bins, formulates the ensemble as GMMs over these bins, and solves for ensemble weights using expectation-maximization. The weights are stored in a lookup table for efficient inference. Experiments are conducted on super-resolution, deblurring and deraining tasks.
strength_weakness: strength_weakness: 1.	Marginal improvement over existing methods: The performance gains are minimal compared to simpler approaches. For example, on the Rain100H dataset (Table 5), the proposed method achieves a PSNR of 31.725, only marginally better than the Average (31.681) and ZZPM (31.679) baselines. In some cases, like on Rain100L, the improvement is less than 0.6 dB over averaging.
2.	Lack of distinction from existing approaches: The paper does not clearly articulate how this method fundamentally improves upon or addresses limitations of existing ensemble techniques. The core idea of using GMMs and LTU for weighting seems combination of existing approaches, and the paper doesn't adequately explain its novelty.
3.	Insufficient analysis of results: There is a notable lack of in-depth analysis of the experiment results. For instance, the paper doesn't discuss why the method performs worse on some deraining tasks (e.g., Test100 in Table 5) compared to HGBT. This lack of analysis makes it difficult to understand the method's strengths and weaknesses.
4.	Inconsistency in parameter selection: The ablation study in Table 1 shows that a bin width of 16 achieves the best PSNR (31.742), yet the authors choose 32 as the default "for the balance of efficiency and performance" (line 220) without adequate justification for this trade-off.
5.	Limited efficiency advantages: The paper claims to address efficiency, but Table 6 shows that the proposed method (0.1709s) is significantly slower than Average (0.0003s) and ZZPM (0.0021s) approaches. The efficiency gain over regression-based methods is not a strong selling point given the performance trade-offs.
6.	Shallow theoretical analysis: While the paper provides derivations in the Appendix, the core idea essentially reduces to a lookup table for ensemble weights. The theoretical contribution and novelty are limited, especially given the marginal performance improvements.
7.	Overstated claims: The paper claims to "consistently outperform" existing methods (lines 17-19), but this is not supported by the results, particularly in deraining tasks where it underperforms HGBT on some datasets.
questions: questions: 1. Why was bin size 32 chosen as default when 16 performed better in ablations?
2. How does this method fundamentally differ from and improve upon existing ensemble approaches?
3. What explains the performance degradation on deraining tasks?
limitations: limitations: The limitations section is brief and does not adequately address key shortcomings like marginal gains and lack of novelty. A more thorough discussion of limitations would strengthen the paper. Given the marginal improvements, lack of novelty, and limited analysis, I recommend rejecting this paper. Significant revisions would be needed to make this a compelling contribution.
paper_summary: paper_summary: A novel post-training ensemble learning method for image restoration is developed in this work by employing Gaussian mixture models and the EM algorithm to generate better restoration results. The authors reformulate the ensemble problem of image restoration into various gaussian mixture models, use the EM algorithm to estimate range-wise ensemble weights on a reference set, and store the weights in a lookup table during inference. The method effectively improves ensemble results on 14 benchmarks and 3 restoration tasks, including super-resolution, deblurring and deraining.
strength_weakness: strength_weakness: - The experiments all show the cases of three base methods, and one of them may be worse than the other two methods. But if all the base models are comparably good, can the method surpass other methods? Experiments with 2 or 4 base models, instead of 3, would be informative.

- The experiments are for the restoration task of local patterns such as deraining and deblurring. However, dehazing or enhancement requires a glocal restoration. What are the results of the ensemble comparisons for dehazing or enhancement? This can provide better insights into the generalizability of the method.

- Some notations are duplicate or misused, like "L" is used for log-likelihood in Eq. 19 and also for the dimension length of the image. In Algorithm 2, both "Equation" and "Eq. " are used. The notations should be revised.
questions: questions: - What is the result of the case with 2 or 4 base models, instead of 3?

- What are the results for dehazing or enhancement?
limitations: limitations: Limitations are mentioned on Page 9 and include the "trade-off between runtime and performance." A discussion and possible solution are provided.
paper_summary: paper_summary: This paper proposes a post-training model ensenble method for image restorationn by leveraging Gaussian mixture models on split pixels and lookup table for fast innference. The pixels are split into various bin sets according to their value ranges and the ensemble problem of the pixels is reformulated into Gaussian mixture models. The EM algotithm is used to solve the ensemble weight of the models. This area lacks sufficient research attention and previous works are mainly using averaging or its variant. So this work has potential influence for industrial application. The proposed method is fast and effective to obtain better ensemble results for three image restoration tasks.
strength_weakness: strength_weakness: 1. ZZPM is a new and recently proposed method after 2022, and averaging is the most straightforward and traditional method. Why is its performance sometimes worse than the normal averaging? 
2. For the task of super-resolution, the input and output are not in the same size, so the notations should be clarified.
3. Why can we suppose that image noise follows a Gaussian distribution with zero mean? Is there any clue or derivation to support the assumption?
4. The authors say that the method is not accelerated by GPU vectorization yet. Is it because CPU is faster or any other reasons? 
5. The pixel numbers of each bin set will be highly diverse as shown in Figure 6. In some bin set, there will be many pixels and then EM algorithm can run properly without problem. However, if there are too few pixels in a bin set, EM algorithm may fail to find a solution.
questions: questions: I have listed some questions regarding the implementation and experiments in Weakness section.
limitations: limitations: The authors have discussed the limitations.

Comments: title: Response to Reviewer EWje
Response to Reviewer E4or
Response to Reviewer Aksr
Official Comment by Reviewer E4or
After rebuttal
Response to Reviewer j7aF
comment: We sincerely appreciate your valuable feedback and constructive comments. We are truly thankful for your recognition of our work.
Thank you so much for your constructive comments you've provided. We are truly thankful for your recognition of our work.
Thank you for your thoughtful review and the constructive comments you've provided. We are truly grateful for your recognition of our work.
Thanks for your prompt response. My concerns have been addressed, and I keep my previous positive rating.
Thanks to the author's detailed response, I choose to maintain my score (Accept).
Thank you to the authors for their detailed response. After reviewing all the rebuttal, I can confirm that all of my concerns have been fully addressed.
Thank you for your response and consideration. We would like to address your remaining concerns individually:

Regarding the first concern about the improvement and analysis, we would like to respond from three perspectives:

1. We wish to emphasize that the enhancement of our method is a complementary benefit that requires no additional training or fine-tuning. It can serve as a vital auxiliary tool for real-world image restoration applications. For instance, in the recent NTIRE 2024 competition, the difference between the best and second-best results was often less than 0.01 dB in PSNR, as shown in the table below. In such cases, our method, which consistently outperforms existing ensemble techniques, can prove decisive.

2. Our approach falls under the category of weakly supervised methods that only require the means and variances of bin sets from degraded images, whereas traditional methods like GBDT and HGBT are fully supervised, requiring access to all pixel values. Under these circumstances, our method is still more stable across various image restoration tasks and less sensitive to performance variations among base models. It is, therefore, commendable and challenging for our method to consistently demonstrate superiority over supervised methods across five tasks and 16 benchmarks.
3. Despite the improvement of 0.001 dB on Test100, our method can achieve a significantly larger improvement of up to 0.20 dB on many other benchmarks, such as Test1200 (33.276 for GBDT versus 33.475 for ours).

The second concern is mainly about the phenomenon where ensemble results are inferior to those of the original model. It is because one base model may be consistently better than the other base models on a benchmark.  We would like to respond with two points:

1. In real-world scenarios, where the ground-truths of the test set are often unavailable, it is difficult to determine which base models are underperforming and contributing to the ensemble's lower results compared to a single original model. In such cases, the best course of action is to mitigate this limitation of ensemble methods, and our approach has demonstrated superior performance in doing so compared to other ensemble techniques.

2. In common industrial scenarios, all base models tend to be similar in structure and performance. As mentioned on Lines 240-243, we aimed to assess our method’s robustness across architectures by conducting experiments using models with different structures and, at times, diverse performances on several test sets (for example, 30.292 for MPRNet and 31.194 for MAXIM versus 32.025 for Restormer). However, this phenomenon could be easily avoided, in the first place, by selecting similar structures or even the same network with different initial states, which is a common practice in industrial settings. Therefore, we cannot acknowledge this phenomenon as a limitation of ensemble methods.

**Table A.** PSNR comparison of the best and second-best results from four challenges at the 9th New Trends in Image Restoration and Enhancement (NTIRE) Workshop and Challenges in 2024.
|Challenge | 2nd best | Best |
| ------------- | ------------- | ------------- |
|Stereo Image Super-Resolution Challenge - Track 1 | 23.6492 | 23.6496 |
|RAW Image Super Resolution Challenge | 43.39 | 43.40 |
|Low Light Enhancement Challenge | 24.52 | 24.52 |
|Image Shadow Removal Challenge - Track 1 | 24.81 | 24.81 |
Thanks for your detailed rebuttal. After careful consideration, I maintain my original rating of 4 (Borderline reject). My decision is primarily based on two key concerns:

1.	Marginal improvements: While I acknowledge the overall improvements across various tasks (SR, Image Deblur, Deraining), the gains are often minimal (0.01/0.001 dB level). As the proposed approach belongs to the classical Ensemble Learning field, it should be primarily compared to well-established methods like GBDT or HGBT. The marginal improvements over these approaches do not sufficiently justify the novelty of your method. More importantly, the paper and rebuttal lack an in-depth analysis in the Experiments section explaining why these marginal improvements occur and under what conditions your method excels or falls short.

2.	Limited efficiency advantages: Your method does not seem to adequately address the limitations of existing approaches. For instance, in Table 5 (Image Deraining results), for the Test100 dataset, both GBDT and HGBT perform worse than the original model. The proposed approach, while showing some improvement, does not significantly overcome this limitation. This raises questions about its practical applicability compared to well-established algorithms like GBDT or HGBT.
I think while the paper presents an interesting approach to ensemble learning for image restoration, the marginal improvements and limited contributions still outweigh its strengths. The proposed method, when compared to well-known algorithms like GBDT or HGBT, does not appear sufficiently practical or innovative to warrant acceptance in its current form.