Title: EnsIR: An Ensemble Algorithm for Image Restoration via Gaussian Mixture Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Model Ensemble

Image Restoration

Gaussian Mixture Models

Expectation Maximization
Abstract: Image restoration has experienced significant advancements due to the development of deep learning. Nevertheless, it encounters challenges related to ill-posed problems, resulting in deviations between single model predictions and ground-truths. Ensemble learning, as a powerful machine learning technique, aims to address these deviations by combining the predictions of multiple base models. Most existing works adopt ensemble learning during the design of restoration models, while only limited research focuses on the inference-stage ensemble of pre-trained restoration models. Regression-based methods fail to enable efficient inference, leading researchers in academia and industry to prefer averaging as their choice for post-training ensemble. To address this, we reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. We estimate the range-wise ensemble weights on a reference set and store them in a lookup table (LUT) for efficient ensemble inference on the test set. Our algorithm is model-agnostic and training-free, allowing seamless integration and enhancement of various pre-trained image restoration models. It consistently outperforms regression-based methods and averaging ensemble approaches on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. The codes and all estimated weights have been released in Github.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post-training model ensenble method for image restorationn by leveraging Gaussian mixture models on split pixels and lookup table for fast innference. The pixels are split into various bin sets according to their value ranges and the ensemble problem of the pixels is reformulated into Gaussian mixture models. The EM algotithm is used to solve the ensemble weight of the models. This area lacks sufficient research attention and previous works are mainly using averaging or its variant. So this work has potential influence for industrial application. The proposed method is fast and effective to obtain better ensemble results for three image restoration tasks.\n\nSTRENGTHS:\n1. The method is novel, properly derived and has a theoretical support. It is well ellaborated in pseudo-codes and discussed.\n2. The authors have conducted extensive experiments on three restoration tasks, including 14 benchmarks.\n3. Its results are promising and consistently better than other baselines.\n\nWEAKNESSES:\n1. ZZPM is a new and recently proposed method after 2022, and averaging is the most straightforward and traditional method. Why is its performance sometimes worse than the normal averaging? \n2. For the task of super-resolution, the input and output are not in the same size, so the notations should be clarified.\n3. Why can we suppose that image noise follows a Gaussian distribution with zero mean? Is there any clue or derivation to support the assumption?\n4. The authors say that the method is not accelerated by GPU vectorization yet. Is it because CPU is faster or any other reasons? \n5. The pixel numbers of each bin set will be highly diverse as shown in Figure 6. In some bin set, there will be many pixels and then EM algorithm can run properly without problem. However, if there are too few pixels in a bin set, EM algorithm may fail to find a solution.\n\nQUESTIONS:\nI have listed some questions regarding the implementation and experiments in Weakness section.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA novel post-training ensemble learning method for image restoration is developed in this work by employing Gaussian mixture models and the EM algorithm to generate better restoration results. The authors reformulate the ensemble problem of image restoration into various gaussian mixture models, use the EM algorithm to estimate range-wise ensemble weights on a reference set, and store the weights in a lookup table during inference. The method effectively improves ensemble results on 14 benchmarks and 3 restoration tasks, including super-resolution, deblurring and deraining.\n\nSTRENGTHS:\n- The method is grounded on a reasonable derivation and assumption of Gaussian prior to restoration. The visualization in Fig. 6 also validates the assumption.\n\n- The method performs consistently well on the mentioned 14 datasets.\n\n- Extensive experiments and ablation studies show the features of the developed method.\n\nWEAKNESSES:\n- The experiments all show the cases of three base methods, and one of them may be worse than the other two methods. But if all the base models are comparably good, can the method surpass other methods? Experiments with 2 or 4 base models, instead of 3, would be informative.\n\n- The experiments are for the restoration task of local patterns such as deraining and deblurring. However, dehazing or enhancement requires a glocal restoration. What are the results of the ensemble comparisons for dehazing or enhancement? This can provide better insights into the generalizability of the method.\n\n- Some notations are duplicate or misused, like \"L\" is used for log-likelihood in Eq. 19 and also for the dimension length of the image. In Algorithm 2, both \"Equation\" and \"Eq. \" are used. The notations should be revised.\n\nQUESTIONS:\n- What is the result of the case with 2 or 4 base models, instead of 3?\n\n- What are the results for dehazing or enhancement?\n\nLIMITATIONS:\nLimitations are mentioned on Page 9 and include the \"trade-off between runtime and performance.\" A discussion and possible solution are provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an ensemble algorithm called EnsIR for image restoration tasks using Gaussian mixture models (GMMs). The method partitions pixels into range-wise bins, formulates the ensemble as GMMs over these bins, and solves for ensemble weights using expectation-maximization. The weights are stored in a lookup table for efficient inference. Experiments are conducted on super-resolution, deblurring and deraining tasks.\n\nSTRENGTHS:\n1. Provides a formulation of image restoration ensemble as GMMs.\n2. Experiments conducted on multiple image restoration tasks and datasets.\n\nWEAKNESSES:\n1.\tMarginal improvement over existing methods: The performance gains are minimal compared to simpler approaches. For example, on the Rain100H dataset (Table 5), the proposed method achieves a PSNR of 31.725, only marginally better than the Average (31.681) and ZZPM (31.679) baselines. In some cases, like on Rain100L, the improvement is less than 0.6 dB over averaging.\n2.\tLack of distinction from existing approaches: The paper does not clearly articulate how this method fundamentally improves upon or addresses limitations of existing ensemble techniques. The core idea of using GMMs and LTU for weighting seems combination of existing approaches, and the paper doesn't adequately explain its novelty.\n3.\tInsufficient analysis of results: There is a notable lack of in-depth analysis of the experiment results. For instance, the paper doesn't discuss why the method performs worse on some deraining tasks (e.g., Test100 in Table 5) compared to HGBT. This lack of analysis makes it difficult to understand the method's strengths and weaknesses.\n4.\tInconsistency in parameter selection: The ablation study in Table 1 shows that a bin width of 16 achieves the best PSNR (31.742), yet the authors choose 32 as the default \"for the balance of efficiency and performance\" (line 220) without adequate justification for this trade-off.\n5.\tLimited efficiency advantages: The paper claims to address efficiency, but Table 6 shows that the proposed method (0.1709s) is significantly slower than Average (0.0003s) and ZZPM (0.0021s) approaches. The efficiency gain over regression-based methods is not a strong selling point given the performance trade-offs.\n6.\tShallow theoretical analysis: While the paper provides derivations in the Appendix, the core idea essentially reduces to a lookup table for ensemble weights. The theoretical contribution and novelty are limited, especially given the marginal performance improvements.\n7.\tOverstated claims: The paper claims to \"consistently outperform\" existing methods (lines 17-19), but this is not supported by the results, particularly in deraining tasks where it underperforms HGBT on some datasets.\n\nQUESTIONS:\n1. Why was bin size 32 chosen as default when 16 performed better in ablations?\n2. How does this method fundamentally differ from and improve upon existing ensemble approaches?\n3. What explains the performance degradation on deraining tasks?\n\nLIMITATIONS:\nThe limitations section is brief and does not adequately address key shortcomings like marginal gains and lack of novelty. A more thorough discussion of limitations would strengthen the paper. Given the marginal improvements, lack of novelty, and limited analysis, I recommend rejecting this paper. Significant revisions would be needed to make this a compelling contribution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. Importantly the authors' method achieves state-of-the-art performance without training. I think this work is interesting and can inspire researchers.

STRENGTHS:
1.The proposed method is more interpretable than traditional deep learning (ensemble learning) methods.
2.The proposed method obtains state-of-the-art performance compared to other ensemble methods.
3.The authors performed comparisons on a large number of datasets and image restoration tasks, which fully validated the generalization of the ensemble method.

WEAKNESSES:
1. I have some confusion. How are the averages in Table III achieved? Why is averaging able to achieve higher results than every single method (non-ensemble method)? Isn't it the average of every single method?


2. I would like to know the computational burden of the method?

3.ensemble learning doesn't provide a "huge" gain, but it makes sense. I'm just wondering how much extra computation is introduced by this gain? Table 6 reports the runtime, and I would like to know the difference in time between the ensemble and the non-ensemble.

4. It's not easy to see the difference in Figure 1, so the author could add an error map or a different figure.

QUESTIONS:
see weaknesses

LIMITATIONS:
see weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12