LIME-Eval: Rethinking Low-light Image Enhancement Evaluation via Object Detection

Authors: Mingjia Li, Hao Zhao, Xiaojie Guo
Primary area: datasets and benchmarks
Keywords: Low-light enhancement, Image quality assessment

Abstract:
Due to the nature of enhancement--the absence of paired ground-truth information, high-level vision tasks have been recently employed to evaluate the performance of low-light image enhancement.  A widely-used manner is to see how accurately an object detector trained on enhanced low-light images by different candidates can perform with respect to annotated semantic labels. In this paper, we first demonstrate that the mentioned approach is generally prone to overfitting, and thus diminishes its measurement reliability. In search of a proper evaluation metric, we propose LIME-Bench, the first online benchmark platform designed to collect human preferences for low-light enhancement, providing a valuable dataset for validating the correlation between human perception and automated evaluation metrics. We then customize LIME-Eval, a novel evaluation framework that utilizes detectors pre-trained on standard-lighting datasets without object annotations, to judge the quality of enhanced images. By adopting an energy-based strategy to assess the accuracy of output confidence maps, our LIME-Eval can simultaneously bypass biases associated with retraining detectors and circumvent the reliance on annotations for dim images. Comprehensive experiments are provided to reveal the effectiveness of our LIME-Eval. Our code will be made publicly available.

Decision: reject
Reviewer scores: 8, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1729944814986

thread_type:
official_review

<thread_object>
creator: reviewer_wgk4

<content>
summary:
This paper reconsiders the evaluation of enhancement quality in low-light tasks, identifies the shortcomings of previous enhancement quality assessment methods, proposes a new dataset based on human preferences, and introduces a novel unlabeled image quality assessment method for enhanced images. The practical significance of the issues addressed in this study is commendable. However, the analysis of the problems is somewhat lacking, and the experimental section could be further improved, as detailed in the Weaknesses section.

strengths:
This paper provides a systematic analysis of methods for measuring the quality of enhanced images in low-light enhancement tasks using object detection. It identifies the shortcomings of previous methods and introduces a new low-light dataset based on human preferences, analyzing the correlation between the evaluations of the object detector and human preferences. Finally, the authors propose a novel unlabeled image quality assessment method that addresses a significant challenge in the current field of low-light enhancement. Overall, this study holds substantial research significance in the domain.

weaknesses:
1. In Chapter 3, the introduction of the need to verify the issue of information loss due to quantization seems abrupt. The earlier sections do not mention the implementation of quantization strategies during the detector's testing process. If quantization was indeed utilized, please provide a brief overview of the quantization process.
2. In Chapter 5, the author fails to analyze how to achieve unlabeled evaluation metrics. The methods section merely outlines the computational processes related to detection-oriented energy-based modeling. The transition from problem introduction to method description is rather abrupt and lacks an analysis of this issue, such as: why is this approach capable of evaluating the quality of image enhancement in an unlabeled manner? Why is the image with the lowest energy value considered optimal? The author should provide relevant analytical descriptions.
3. It is necessary to standardize the meaning of variable x_bg. Why is it referred to as "object output" in line 402, while in line 415 it denotes "background prediction"? Additionally, the method of obtaining x_bg should be specified in this paper. If the network produces multiple object outputs, how should we derive x_bg?
4. In Chapter 5.2, Table 2 presents a comparison of the Spearman correlation coefficients for only IQA algorithm, which is insufficient. I recommend including Spearman correlation coefficients for several additional algorithms to provide a more convincing conclusion.
5. In Chapter 5.3, Table 3 presents the improvements of the method on the object detection metrics mAP and AP50. The author should also indicate whether there are improvements in other metrics (such as recall mAR and AR50). Furthermore, the author could include experimental analyses of additional object detection algorithms (e.g., YOLOX) to enhance the persuasiveness of the conclusions.
6. The article lacks citations for certain terms and methods, such as YOLOX, LOL dataset, and LPIPS. The author should include the relevant references in the text and check for any other missing citations.
7. This paper only discusses the application of detection-oriented energy-based modeling in low-light enhancement tasks. Is this evaluation strategy also applicable to other enhancement tasks (such as rain removal, fog removal, deblurring, etc.)? If possible, I suggest that the authors include a section reflecting on this aspect, such as adding it to the future work section.

questions:
The authors need to solve the questions mentioned, particularly question 2, in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440864458

thread_type:
official_review

<thread_object>
creator: reviewer_nsif

<content>
summary:
The paper argues that using object detectors trained on enhanced low-light images for evaluation is prone to overfitting, reducing reliability. To address this, the authors introduce LIME-Bench, an online platform collecting human preferences to validate the link between human perception and automated metrics. They also present LIME-Eval, a framework using pre-trained detectors without annotations to evaluate image quality, avoiding retraining biases and dependency on dim image annotations.

strengths:
In the background of low-light enhancement, the quality of the enhanced image may indeed be independent of the target detection performance. To verify this phenomenon, they established an online benchmarking platform, LIME - Bench, and developed a label-free assessment metric, LIME-Eval, for quality evaluation of image enhancement. This is a metric that considers both human vision and machine vision.

weaknesses:
I think the experiment lacks conviction.
1. Qualitative comparisons on samples from the ExDark dataset are inadequate and even wrong in my opinion. The reason for this is that the augmentation strategy is random and not a fair input to the target detector.
2. How is LIME-Eval integrated into Retinexformer? I don't see it in the appendix either, rather than you just stating the training details.
3. If LIME-Eva is integrated into the detection, will it participate in the calculation of the parameter gradient of the neural network? If it is involved, then I think that LIME-Eva just has the efficacy of a loss function, since the low-light enhanced image is then fed to the target detector for training, which is not referential in nature, as you conclude in Table I.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527695540

thread_type:
official_review

<thread_object>
creator: reviewer_egde

<content>
summary:
This paper addresses the challenge of evaluating low-light image enhancement without paired ground-truth data, as current high-level vision task evaluations can often lead to overfitting and unreliable metrics. To improve this, the authors introduce LIME-Bench, an online platform that collects human preference data for low-light enhancement, creating a valuable dataset to correlate human perception with automated metrics. They further propose LIME-Eval, a novel evaluation framework using pre-trained detectors without retraining or annotation dependencies, to assess enhanced image quality. Through energy-based confidence map assessment, LIME-Eval effectively gauges enhancement accuracy. Extensive experiments validate LIME-Eval's effectiveness, and the code will be publicly available.

strengths:
1. this paper addresses the challenges of evaluating low-light enhancement techniques via the proposed LIME-Bench online platform, collecting human preference low-light images for low-light enhancement techniques, and the LIME-Eval framework without fine-tuned detection models and annotation labels.

2. the paper demonstrates the uncorrelation of high detection performance to the enhancement quality of the enhanced images through comprehensive studies and analysis. This provides a scientific foundation to support their proposed LIME-Bench platform and LIME-Eval framework.

3. the paper addresses the challenge of evaluating low-light enhancement techniques without fine-tuning detection models and acquiring annotation labels through the energy-based modeling technique.

weaknesses:
1. the purpose of consdering background prediction in the LIME-Eval framework is vauge. It would be great for authors to describe the benefit of integrating background prediction into the energy-based evaluation function.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639634385

thread_type:
official_review

<thread_object>
creator: reviewer_3lrc

<content>
summary:
The work: LIME-Eval: Rethinking Low-Light Image Enhancement Evaluation via Object Detection, critiques the standard practice of evaluating low-light image enhancement by retraining object detectors on enhanced images—a method prone to overfitting and reduced reliability. Instead, the authors propose LIME-Eval, an energy-based evaluation framework that eliminates the need for retraining detectors or relying on annotated data. They also introduce LIME-Bench, an online platform that collects human preferences on low-light enhancement to align automated metrics with human perception. Experiments on datasets like ExDark demonstrate LIME-Eval’s effectiveness in evaluating enhanced images in a way that closely mirrors human judgment.

strengths:
1. Higher detection accuracy does not necessarily correlate with superior enhancement quality, which aligns with my understanding of low-light enhancement models. There exists a gap between low-level, human-visual-oriented perception and high-level, machine-vision-oriented objectives. Additionally, as noted in line 68, 'Is fine-tuning detectors a valid approach to evaluating enhancers?' this raises a valuable point. Relying on fine-tuning detectors to fully assess low-light detection performance can lead to overfitting and is not necessarily a scientifically rigorous approach.

2. The paper is well written, easy to understand, and authors conduct extensive experiments to validate their approach, offering thorough insights into the correlations between detector performance, human ratings, and LIME-Eval metrics.

3. LIME-Bench provides a structured way to incorporate human preferences into evaluation, bridging the gap between machine-based metrics and human perceptual quality. Which could also be helpful to future low-light vision research.

weaknesses:
1. In the 'enhanced before data augmentation' setting (Table.1), is there a process like 'color jitter'? I think 'color jitter' could introduce significant interference, as it might substantially degrade the results of low-light enhancement if applied. Also the normalization of image maybe also affect the low-light enhancement models. Could the authors provide more detail on this part? 

2. The paper mainly uses ExDark and synthetic datasets, can this evaluation tool also been used in the low-light face detection dataset UG2+ DARK FACE ?

3. Some previous low-light object detection works, although have not used pre-defined enhancement modules, they remain closely related to this study. They did not use image enhancement models but instead employed other approaches, such as self-supervised learning and domain adaptation, to address the issue of low-light object detection. It would be beneficial for the authors to reference and discuss these works in the related works section (e.g., [1, 2, 3]).

[1]. Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection, ICCV 2021

[2]. Similarity min-max: Zero-shot day-night domain adaptation, ICCV 2023

[3]. Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation, CVPR 2024

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732431930902

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Notes for our rebuttal revision

comment:
Dear ACs and Reviewers,

We would like to thank you for your time and effort spent on our manuscript. The concerns and suggestions are valuable for us to further improve the quality of this work. We have taken these comments on board to make the rebuttal revision. 

The following summarizes the major points of our rebuttal:
 
+ We enriched the Related Work Section, by adding the discussion on low-light object detection schemes, thanks to Reviewer 3LRc's valuable advice.

+ We refined Section 4 and added more explanation to our energy-based modeling framework, as suggested by Reviewer Wgk4 and egde.

+ We added additional competitors in Table 2, and more evaluation metrics in Table 3 to make comparisons more comprehensive, thanks to Reviewer Wgk4's kind reminder.

+ We detailed the training recipe for the detectors in the Appendix A.3 as suggested by Reviewer 3LRc.

+ We added a discussion on the applicability of our work on face detection, and generalization potential to other low-level vision tasks in Appendix B, inspired by Reviewers 3LRc and Wgk4.

+ We highlighted the contribution of our work to avoid possible confusion, as mentioned by Reviewer nSif, on the pipeline and technical contribution of this manuscript.

+ We performed thorough proofreading to eliminate typos and mistakes in the previous version. We also added appropriate references, as suggested by Reviewer Wgk4. 

We have provided detailed responses to the reviewers. We sincerely hope that our response can address the concerns.

Thanks again for your review. 

The authors of LIME-Eval
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the challenges in evaluating low-light image enhancement due to the lack of ground-truth references. It critiques a widely-used evaluation approach, which measures object detection accuracy on enhanced low-light images, arguing that it is prone to overfitting and unreliable. To address this, the authors propose a LIME-Bench, an online benchmarking platform designed to gather human preferences for low-light image enhancement. Besides, they propose a novel evaluation framework, LIME-Eval, that uses pre-trained object detectors without requiring annotations.

This paper receives significant dispersed scores among the reviews. The two reviewers, 3LRc and egde, largely appreciate the paper's contributions, particularly its emphasis on highlighting the distinction between low-light enhancement and detection. Besides, they all think this paper is well-written and well-organized.

The major negative comments come from Reviewer nSif, concerning center on the adequacy of the experiments, particularly the fairness of the augmentation strategy, the role of LIME-Eval in detection, and its integration with Retinexformer, which lacks clarity and rigor. These issues indicate that the methodology and experimental validation require further refinement, making the paper unsuitable for acceptance at this stage.