LIME-Eval: Rethinking Low-light Image Enhancement Evaluation via Object Detection

## Abstract

Abstract Due to the nature of enhancement–the absence of paired ground-truth information, high-level vision tasks have been recently employed to evaluate the performance of low-light image enhancement. A widely-used manner is to see how accurately an object detector trained on enhanced low-light images by different candidates can perform with respect to annotated semantic labels. In this paper, we first demonstrate that the mentioned approach is generally prone to overfitting, and thus diminishes its measurement reliability. In search of a proper evaluation metric, we propose LIME-Bench, the first online benchmark platform designed to collect human preferences for low-light enhancement, providing a valuable dataset for validating the correlation between human perception and automated evaluation metrics. We then customize LIME-Eval, a novel evaluation framework that utilizes detectors pre-trained on standard-lighting datasets without object annotations, to judge the quality of enhanced images. By adopting an energy-based strategy to assess the accuracy of output confidence maps, our LIME-Eval can simultaneously bypass biases associated with retraining detectors and circumvent the reliance on annotations for dim images. Comprehensive experiments are provided to reveal the effectiveness of our LIME-Eval. Our benchmark platform and code are available online.

## 1 Introduction

Low-light conditions significantly challenge imaging by reducing the visibility of important details and/or introducing distortions in captured images, such as noise, blur, and color shifts. The poor quality of images captured in such conditions not only hampers everyday photography but also poses serious issues in fields where image clarity is critical, such as surveillance, navigation, and astrophotography. Consequently, low-light image enhancement has emerged as an essential technique to improve the quality of images taken in unsatisfactory lighting environments.

The objective of low-light image enhancement is to concurrently brighten dark regions, enhance suppressed details, preserve color fidelity, and eliminate potential artifacts. In other words, it is desired to generate high-quality images that closely resemble those taken under “good”(^1^11As an enhancement task, there is no well-defined optimal lighting condition.) lighting conditions. While substantial progress has been made in this domain, spanning from histogram equalization (Trahanias & Venetsanopoulos, ) to advanced deep-learning approaches (Cai et al., ; Zhang et al., ; Guo & Hu, ), a key challenge persists in objectively evaluating the performance of enhancement algorithms. Historically, the image quality assessment (IQA) of enhancement has depended on reference-based metrics (*e.g.*, PSNR and SSIM), which compare the enhanced results to a reference image deemed to be of high quality.

We argue that reference-based metrics are unsuitable for low-light enhancement, because the very nature of the problem precludes the existence of reliable reference images. On the one hand, capturing such references with identical settings, except for proper illumination, is inherently challenging. On the other hand, even if reference images are obtained under well-controlled settings, many/infinite variations of “well-lit” conditions exist, making it hard to determine which specific scenario aligns with the “best”. This absence of a definitive standard (actually for all enhancement tasks) complicates the evaluation process, and necessitates the development of no-reference assessments. One might suggest using no-reference IQA metrics, such as NIQE (Mittal et al., ) and BRISQUE (Mittal et al., ), which are currently mainstream. However, assessing the quality of enhanced images involves a complex interplay of quality and aesthetic issues (color restoration and lightness). Existing no-reference IQA metrics are often inconsistent or even contradictory with human perception, rendering them unreliable in low-light scenarios (Saha et al., ; Guo & Hu, ), as we will demonstrate in Sec. [4](#S4).

Alternatively, the community has begun to reassess the effectiveness of low-light image enhancement techniques by examining their impact on downstream vision tasks, particularly object detection. The core idea is that the performance of downstream models can serve as a proxy for human perception. If objects within an image are adequately illuminated, they should be easily identifiable by both humans and machines. A widely-adopted evaluation protocol in recent studies requires fine-tuning an object detector on images enhanced by different methods (Cui et al., ; Cai et al., ; Zhou et al., ).
However, this retraining process raises a significant concern: *i.e.*, overfitting. The detector may become overly tailored to the specific characteristics of the enhanced images, disregarding its resemblance to a natural, well-illuminated image. This leads to a fundamental question: *Is fine-tuning detectors a valid approach to evaluating enhancers?* In Sec. [4](#S4), we will demonstrate that the performance of fine-tuned detectors does not necessarily correlate with the quality of enhanced images. With the application of an appropriate augmentation strategy, fine-tuning on inadequately enhanced images can still yield better detection performance than even the most advanced enhancement methods. This finding indicates that the fine-tuning protocol conflates the effectiveness of the enhancement algorithms with the adaptability of detection models, ultimately compromising the fairness and reliability of the evaluation.

To remedy the aforementioned flaw, a straightforward strategy shall deploy detectors pre-trained on data captured under normal-lighting conditions to evaluate enhanced images, using annotations from the original low-light images (Wang et al., ; Ma et al., ). The underlying premise is that the closer the enhanced results resemble the normal-lit image domain, the better the detection performance will be.
To validate this approach, we introduce the first online benchmark platform, *LIME-Bench*, designed to collect human preference on assessing low-light enhancement methods. Through the data collected on this platform, we verify that directly applying pre-trained detectors serves as a more effective critique for evaluating low-light image quality than a series of previously applied quality assessment methods, offering a reliable proxy for enhancement performance.

While this manner takes advantage of the inherent generalizability of models trained under standard illumination to assess the fidelity of low-light enhancements, it introduces its own set of challenges related to semantic labels. For one thing, obtaining accurate annotations for low-light images is more difficult and time-consuming than for those captured under normal lighting conditions, as the reduced visibility and contrast in low-light images increase ambiguities in object boundaries and classifications. For another thing, the reliance on annotated labels restricts the flexibility of evaluation. Moreover, annotating a large dataset of low-light images to establish a reliable benchmark for evaluation is labor-intensive, limiting the scalability of this approach.

This work proposes a novel framework, called *LIME-Eval*, for evaluating low-light image enhancement. Grounded in a pioneering energy-based criterion, our method sidesteps the biases and time-consuming processes associated with retraining detectors while liberating the demand for both reference images and detection labels. These features broaden the applicability of LIME-Eval to unlabeled and reference-free low-light scenarios.
Our primary contributions are summarized as follows:

- •
By retraining detectors on enhanced images produced by various low-light enhancement methods, we find that, under appropriate data augmentation conditions, higher detection accuracy does NOT necessarily correlate with superior enhancement quality.
- •
We collect 6,362 feedback pairs from 750 users, encompassing factors such as blurriness, exposure, noisiness, color, and overall quality across 14 low-light enhancement methods to construct the first low-light user preference dataset, LIME-Bench. Utilizing this data, we benchmark non-reference image quality assessment methods in prior arts and validate the correlation between detector-based evaluations and human preferences.
- •
We introduce a novel energy-based evaluation framework, say LIME-Eval, which effectively links the quality of enhanced images with the performance of object detection without object labels or reference images. Comprehensive experimental results and analyses confirm LIME-Eval’s effectiveness and evidence its potential to guide low-light enhancers.

## 2 Related Work

Low-light Image Enhancement
aims to tackle multiple degradations present in dark images such as noise, low contrast, and color shift. Early methods, like histogram equalization and its variants, sought to improve image visibility by adjusting global and/or local contrast. The advent of deep learning has led to a range of innovative approaches. Within this context, Retinex theory–which conceptualizes an image as the product of reflectance and illumination components–has gained significant attention. Several schemes based on this paradigm endeavor to produce normal-light images by modulating the illumination component and estimating reflectance (Fu et al., ; Ng & Wang, ). In advancing the exploration of attention mechanisms, the transformer architecture integrates self-attention with convolutional processes to simultaneously extract long-range and short-range dependencies. As a representative work, Retinexformer (Cai et al., ) introduces a new self-attention IG-MSA module, based on retinex theory and transformer architecture. Focusing on the illumination of an image, an illumination adaptive transformer (IAT) was proposed (Cui et al., ), notable for its minimalist design of just 90k parameters and its efficiency in addressing illumination adjustments. Guo and Hu (Guo & Hu, ) decoupled the entanglement of noise and color distortion, further alleviating the challenges of low-light enhancement in the presence of complex degradations. In the absence of ground truth, Guo *et al.* (Guo et al., ) proposed an unsupervised method adjusting the illumination with LE-curve, achieving reasonable results at an impressively fast pace. These developments represent a significant leap in low-light image enhancement. However, as previously discussed, the lack of exact reference images for enhancement tasks necessitates further research to explore methods for assessing enhanced images in reference-free fashions.

Image Quality/Aesthetic Assessment
has always been a fundamental task in image processing, especially in enhancement, compression, and restoration. Traditional methods heavily rely on full-reference metrics, with Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as two prominent examples, comparing processed results against high-quality references. Recently, several deep-learning-based variants for full-reference image quality assessment have been proposed. For instance, Kim *et al.* (Kim et al., ) employed an end-to-end CNN model. While being effective under certain conditions, this approach frequently fails to encompass the perceptual quality as perceived by human observers. This discrepancy has spurred the development of no-reference image quality assessment (NR-IQA) methods, which forgo the need for any reference images. Early NR-IQA research primarily focused on specific distortions, notably JPEG compression (Wang et al., ; Marziliano et al., ). The introduction of the LIVE dataset (Sheikh et al., ) marked a shift toward general-purpose NR-IQA, which leverages natural scene statistics (NSS) from spatial (Mittal et al., ; ) or transform domains (Moorthy & Bovik, ; Saad et al., ) to assess image quality, predicated on the premise that deviations from the statistical regularities found in natural images correlate with perceived visual quality. (Simoncelli & Olshausen, ). With the expansion of IQA datasets and the growing influx of images, deep learning has emerged as the predominant force in NR-IQA. To compensate for the shortage of manually-labeled data, strategies like patchwise training (Bosse et al., ; Kang et al., ), transfer learning (Zeng et al., ), and quality-aware pre-training (Ma et al., ; Liu et al., ) have been developed. Up-to-date NR-IQA research cooperated with innovations like active learning, meta-learning, patch-to-picture mapping, loss normalization, and adaptive convolution. These advancements aim to enhance generalizability (Wang & Ma, ), enable rapid adaptation (Zhu et al., ), improve local quality prediction (Ying et al., ), expedite convergence (Li et al., ), and facilitate content-aware quality assessment (Su et al., ). These IQA methods have demonstrated significant success in evaluating the quality of enhanced images. Nevertheless, there remains a noticeable gap in understanding the relationship between these quality evaluations and the performance of subsequent downstream tasks. Bridging this gap is crucial for a more comprehensive assessment of image enhancement techniques in real-world applications.

Benchmarking Low-light Enhancement with Detection is tough due to the subjective nature of image quality and the lack of suitable standards for comparison. This has led researchers to explore alternative evaluation strategies, *e.g.*, subjective assessment by human observers, or the use of synthetic datasets where ground truth is artificially generated. The ExDark dataset (Loh & Chan, ) serves as a repository of low-light object images, defined by criteria including low illumination levels or pronounced variations in lighting. The Matching in the Dark (MID) dataset (Song et al., ) offers a collection of stereo image pairs spanning 54 indoor and 54 outdoor settings, culminating in a total of 108 scene pairs. Despite the value of these datasets, they suffer from limitations in scalability and often fail to capture the complexity of real-world scenarios. Thus, it is imperative to develop innovative methodologies for evaluating enhanced images in the absence of reference images.

Energy-based Models (EBMs) are versatile, non-normalized probabilistic models introduced by (LeCun et al., ). They define relationships among variables by assigning a scalar energy value to each multivariate instance. Unconstrained by the need to maintain normalized probabilities, EBMs have found application across a wide array of tasks (Li et al., ; Du et al., ; ). Thanks to their ability to represent complex, high-dimensional data distributions, EBMs have also been applied in generative modeling tasks (Arbel et al., ). The work (Grathwohl et al., ) demonstrates how classifiers can inherently function as EBMs, further broadening their applicability. This perspective on energy has been harnessed for tasks such as out-of-distribution detection (Lafon et al., ) and automated evaluation of classification models (Peng et al., ). Inspired by these advancements, our approach adopts energy-based statistics as a proxy for average accuracy, showcasing the model’s adaptability and effectiveness in evaluation contexts.

## 3 Rethinking Evaluation Protocol

The information bottleneck theory (Tishby et al., ) suggests that neural network operations can result in information loss, potentially obscuring critical clues for high-level tasks. This understanding has influenced the evaluation of low-light enhancement methods, where performance is often assessed by retraining downstream recognition models on images enhanced by these enhancers. But, the validity of such an evaluation scheme is questionable. To manifest this, we carefully select four low-light enhancement techniques that exemplify the diversity in current low-light enhancement approaches, including Zero-DCE (Guo et al., ), Bread (Guo & Hu, ), IAT (Cui et al., ) and RetinexFormer (Cai et al., ).

**Table 1: Quantitative comparison in detection accuracy mAP on the test split of ExDark. The best and second-best results by each scheme are in bold and underlined, respecitvely. The results in the ‘Dim image’ column are obtained by directly training the detector on images without enhancement.**
|  | Dim image | Zero-DCE | Bread | Bread-round | IAT | RetinexFormer |
| --- | --- | --- | --- | --- | --- | --- |
| Enhance First | N/A | 45.9 | 45.9 | N/A | 47.6 | 46.1 |
| Augmentation First | 49.2 | 49.2 | 48.9 | 48.8 | 49.0 | 48.5 |
| Direct Eval | 35.0 | 34.0 | 35.4 | N/A | 35.3 | 34.2 |

Figure: (a) Input
Refer to caption: /html/2410.08810/assets/fig/input_2015_00003.png

Without loss of generality, we initiate our investigation with object detection on the ExDark dataset, a widely recognized benchmark for low-light conditions. As for our base detector, we choose the medium version of YOLOX, aiming to provide a balanced and comprehensive analysis of the performance of low-light enhancers in challenging real-world conditions. All the involved enhancers, pre-trained on the LOL dataset, remain fixed during detector training. We experiment with two distinct training settings:
1) *Enhancement First Augmentation After*: Low-light images are enhanced first and saved in a lossless format, strong augmentation techniques (*e.g.*, Mosaic, shear, mixup); and 2) *Augmentation First Enhancement After*: Augmentation techniques are applied directly to the original low-light images, enhancement is performed on-the-fly, applied to the augmented output.

Referencing Tab. [1](#S3.T1), it is noteworthy that while Bread exhibits superior visual results (please see Fig. [1](#S3.F1) for visual comparisons), it does not achieve the highest detection performance under either of the tested settings. Interestingly, the detection performance of Bread (45.9), IAT (47.6), and RetinexFormer (46.1), when enhanced before data augmentation, does not surpass dim (low-light) images (49.2). In contrast, when adopting the augmentation-first strategy, all the methods show a marked improvement in performance. To ensure that performance gains is not attributable to information loss during quantization, we conducted an experiment with Bread by clipping and quantizing its intermediate output before detector training. The modified version, referred to as Bread-round, demonstrated performance comparable to the original Bread. This indicates that the performance drop in the enhancement-first setting is inherent to the enhancement-first scheme itself.

This evidence highlights a key limitation of the retraining approach: it encourages detectors to optimize the utilization of available input clues and adapt specifically to the enhanced input domain. Under this paradigm, the training process compels the detector to rely solely on these input clues, rather than leveraging the common sense that underpins human perception.
To tackle the overfitting issue, a straightforward solution is to forego training the detector, and instead use models trained on large-scale normal-light datasets (for instance, the MS-COCO dataset) for direct inference on low-light images. The results is detailed in Tab. [1](#S3.T1), under “Direct Eval”. The findings reflect that Bread and IAT, which visually resemble normal-light images more closely, outperform those without enhancement and models Zero-DCE (which suffers from poor noise suppression) and RetinexFormer (which introduces artifacts due to overfitting). As the misalignment between the focus of fine-tuned detectors and actual perceptual quality results in skewed evaluation fairness, direct evaluation using pre-trained detectors without additional fine-tuning seems to offer a more unbiased protocol for assessing low-light enhancement methods.

## 4 Benchmarking Machine-Human Consensus via LIME-Bench

However, the direct evaluation approach raises new questions that warrant further investigation: *Is there always a consensus between human preference and direct detection performance?* and *If discrepancies arise, which factors have the most significant impact?*

To address these questions, we conducted user studies using the images from ExDark dataset Loh & Chan (), comparing dim images with outputs from 14 low-light enhancement methods. These methods include one optimization-based approach (LIME (Guo et al., )), 7 supervised methods (Bread (Guo & Hu, ), Kind (Zhang et al., ), Retinexformer (Cai et al., ), IAT (Cui et al., ), SNR (Xu et al., ), LLFlow (Wang et al., ) and PyDiff (Zhou et al., )), and 6 unsupervised methods (QuadPrior (), LightenDiffusion (Jiang et al., ), SCI (Ma et al., ), ZeroDCE (Guo et al., ) PairLIE (Fu et al., ) and NeRCo (Yang et al., ) ).
Inspired by Chatbot Arena (Chiang et al., ), we presented users with pairs of images generated by different enhancement methods, and randomly selected one of five aspects—overall quality, illumination, noise/artifacts, blurriness, or color. Participants were then asked to choose the better option based on the selected criterion. To quantify user preferences across all methods, we employed the Elo rating system to convert these pairwise comparisons into a comprehensive rating. Further details of the study can be found in the appendix.

As depicted in Fig [2](#S4.F2) (a), the direct evaluation of detection performance shows a strong correlation with user preferences from the study. The Spearman correlation coefficient $r$ is 0.703, indicating a robust positive relationship that suggests a general consistency between detection scores and user-assigned image ratings across different methods. This correlation is statistically significant, with a p-value of 0.0035. However, several outliers are evident in the figure: 1) While IAT demonstrates strong detection performance, it ranks 13th in overall user preference. As shown in Fig. [2](#S4.F2)(b), IAT’s color rendering is particularly unappealing, a crucial factor to human perception that may be overlooked by detectors trained with extensive color jitter augmentation. 2) The input image ranks 8th in the Elo rating, but achieves third-best detection performance. As illustrated in Fig. [2](#S4.F2)(b), although the dim image is poorly-illuminated, its noise and JPEG artifacts are (of course) less noticeable in the darker areas. In contrast, some enhancement methods (*e.g.*, NeRCo) may inadvertently introduce additional artifacts during the enhancement process, resulting in lower detection performance despite potential improvement in illumination.

Consequently, direct evaluation with detectors is less sensitive to color shift and poor illumination, but is more sensitive to noise and artifact. Despite these discrepancies, the overall correlation indicates that direct detection performance can serve as a reasonable proxy for assessing enhancement quality. To illustrate the advantage of using direct evaluation scheme, we selected 6 popular IQA/IAA methods in low-light enhancement, including NIQE (Mittal et al., ), BRISQUE (Mittal et al., ), MUSIQ (Saha et al., ), ClipIQA (Wang et al., ), NIMA (Esfandarani & Milanfar, ) and LIQE (Zhang et al., ). We fed the same input images used in the user study for benchmarking. The results can be found in Fig. [3](#S4.F3). BRISQUE, ClipIQA, LIQE, and MUSIQ tend to favor the outputs from PyDiff and NeRCo while overlooking the performance of LightenDiffusion. Among these methods, NIMA reaches the best correlation with human preferences, with a Spearman $r$ of 0.457. However, the alignment between these quality assessment approaches and human perception remains inferior to that of the detection-based evaluation, highlighting the reliability of direct detection performance as an
assessment metric.

Figure: (a)
Refer to caption: /html/2410.08810/assets/fig/rank.png

Figure: (a) NIQE
Refer to caption: /html/2410.08810/assets/fig/niqe.png

## 5 Towards Label-free Evaluation through LIME-eval

Previous experimental validation still relies on annotated datasets. The dependency on annotated labels presents a series of significant hurdles. Primarily, securing precise annotations for low-light images is notably more difficult and time-consuming than for well-lit images due to the inherent challenges associated with low visibility. The reduced contrast and clarity in low-light conditions often lead to unclear object boundaries and categories, elevating the potential for inaccuracies in annotations. Furthermore, the endeavor to annotate an large-scale dataset of low-light images for establishing a dependable benchmark demands considerable labor and expense, thereby constraining the scalability and practicality of such evaluative methods.

Given the reliance on annotated datasets in these experiments, there emerges a pressing need for an evaluation methodology that operates independently of labels. A label-independent evaluation approach would streamline the assessment of low-light image enhancement techniques and broaden the applicability across diverse and unlabeled datasets. Consequently, exploring and developing an evaluation strategy that transcends the need for annotated datasets becomes a critical next step in advancing the field of low-light image enhancement. In what follows, we shall introduce our LIME-Eval, a label-free evaluation metric, as a pioneering exploration of this problem.

### 5.1 Detection-oriented Energy-based Modeling

The energy-based model (EBM)  (LeCun et al., ) can map data point $x$ with any dimension into a scalar through an energy function $Z(x):\mathbb{R}^{D}\rightarrow\mathbb{R}$. To transfer the energy function into a probability density function $p(x)$, one could adapt the Gibbs distribution as follows:

$$ $p(y\mid x)=\frac{e^{-Z(x,y)/T}}{\int_{y^{\prime}}e^{-Z(x,y^{\prime})/T}}=\frac{e^{-Z(x,y)/T}}{e^{-Z(x)/T}},$ (1) $$

where $\int_{y^{\prime}}e^{-Z(x,y^{\prime})/T}$ is the partition function by marginalizing over label $y$, and $T$ is a positive temperature constant. Now the Gibbs free energy $Z(x)$ at the data point $x$ with the negative of the log partition function can be written as:

$$ $Z(x)=-T\cdot\log\int_{y^{\prime}}e^{-Z(x,y^{\prime})/T}.$ (2) $$

The inherent connection between energy-based models and discriminative models has been explored in (Grathwohl et al., ; Peng et al., ). Consider a $K$-category classifier $f$, which maps input vector $x$ into $K$ logits, with the softmax function, we can parameterize a categorical distribution via:

$$ $p(y\mid x)=\frac{e^{f_{y}(x)/T}}{\sum_{k=1}^{K}e^{f_{k}(x)/T}},$ (3) $$

where $f_{y}(\cdot)$ denotes logit corresponding to $y$-th term of $f(x)$. Thus, the energy function can be expressed as:

$$ $Z(x)=T\cdot\log\sum_{j=k}^{K}e^{f_{k}(x)/T}.$ (4) $$

The above modeling can only be applied to the task of classification, which has been adapted to the AutoEval (Peng et al., ). The classification task only involved one overall prediction. To adopt it into our target task, i.e. object detection(^2^22For simplicity, here we omit the multi-scale outputs and consider the output as heatmaps in $H\times W$.), which contains both classification output $x_{cls}\in\mathbb{R}^{K\times H\times W}$ and object output $x_{bg}\in\mathbb{R}^{H\times W}$, we propose to fuse the logits in $x_{cls}$ with the objectness information $x_{bg}$ into re-weighted energy $x_{r}$ as follows:

$$ $x_{r}^{y}=\sqrt{x_{cls}^{y}\cdot(1-x_{bg})},$ (5) $$

where $x_{r}^{y}$ denotes the $y$-th logit of $x_{r}$. After that, we integrate Eq. ([4](#S5.E4)) into final evaluation function $E(x_{cls},x_{bg})$ as follows:

$$ $E(x_{cls},x_{bg})=-\sum_{i,j}^{H,W}{T\log(\sum_{y}e^{x_{r}^{y,i,j}/T})},$ (6) $$

where $x_{r}^{y,i,j}$ denotes logit corresponding to $y$-th term of $x_{r}$ at $(i,j)$. This indicator transforms spatial confidence information into a distribution measure, which can be further aggregated over the dataset for a dataset-level metric. The overall pipeline of our LIME-Eval is illustrated in Fig. [4](#S5.F4). After extracting background prediction $x_{bg}$ and classification prediction $x_{cls}$, the two feature maps are fused via $\sqrt{x_{cls}(1-x_{bg})}$. The energy is calculated as in Eq. ([6](#S5.E6)). Finally, we identify the image with the lowest energy value as the optimal one.

Figure: Figure 4: The pipeline of our proposed LIME-Eval.
Refer to caption: /html/2410.08810/assets/fig/pipeline.jpg

### 5.2 Experimental Validation

Correlation Studies on Synthesised Datasets. Having the evaluation function defined, it becomes feasible to assess images without relying on labels. To show that our proposed energy metric aligns with detection performance, we synthesized images from the validation set of MS-COCO with low-light-related distortion.

Figure: Figure 5: A visualization of synthetic setting. More details can be found in Appendix.
Refer to caption: /html/2410.08810/assets/fig/owl.png

To obtain a more accurate approximation of the mean Average Precision (mAP), we begin by pre-calibrating the images using a synthetic dataset derived from the validation split of the MS-COCO dataset. MS-COCO, a large-scale dataset for object detection, encompasses 80 object categories, providing a robust foundation for evaluating detection performance. The validation set of MS-COCO consists of 5,000 images, capturing a wide range of everyday scenes.

Inspired by typical low-light enhancers (Guo & Hu, ; Guo et al., ), which first perform exposure correction and then handle degradation, our synthesis pipeline is similarly divided into two primary phases to address two types of distortions: 1) *Exposure Shift*. The first phase focuses on the prevalent issues of overexposure and underexposure commonly observed in low-light enhancement outputs. We employ gamma correction to simulate these exposure shifts; and 2) *Degradation*. The second phase replicates degradations such as ineffective noise suppression, leading to either noise persistence or excessive image smoothing, as well as color distortions. To simulate these effects, we introduce impulse, shot, and Gaussian noises for the former, and employ Gaussian blur for the latter. Further, we adopt strategies to reduce saturation and brightness, mimicking color distortions.

A detailed depiction of the synthesizing process is illustrated in Fig. [5](#S5.F5). To quantitatively evaluate the performance of our energy metric, we employed two statistical measures: Pearson’s correlation coefficient ($\rho$) and Spearman’s rank correlation coefficient ($r$). Pearson’s correlation ($\rho$) measures the linear relationship between the energy metric values and detection performance scores, providing insight into how well the metric predicts actual performance improvements. Spearman’s rank correlation ($r$), on the other hand, assesses how well the relationship between the energy metric and detection performance follows a monotonic function. This is particularly useful for understanding the metric’s ability to rank enhancement methods accroding to their impact on detection performance, irrespective of the linearity of the relationship. A YOLOX-x model trained on the MS-COCO dataset is adopted as $f$ (aforementioned classifier). As can be seen from Fig. [6](#S5.F6), our proposed method shows a strong correlation with mAP ($r=0.881$, $\rho=0.847$), indicating that the proposed evaluation function aligns closely with actual mAP, even without the help of labels.

Figure: (a) $T=0.1$
Refer to caption: /html/2410.08810/assets/fig/plot_heat0_1.png

Consensus with Human-preference. We also employ user preference from LIME-Bench to benchmark the performance of the proposed method. As reported in Tab. [2](#S5.T2), our method exhibits a stronger correlation with human preferences compared to previous image quality assessment methods, demonstrating its superior alignment with perceptual quality judgments.

**Table 2: Spearman correlation comparison between IQA methods and our LIME-Eval. The best is in bold and the second-best is underlined.**
| Method | BRISQUE | NIQE | NIMA | LIQE | MUSIQ | ClipIQA | LIME-Eval |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Spearman $r$ | 0.321 | -0.250 | 0.457 | 0.225 | 0.143 | 0.307 | 0.593 |

**Table 3: Quantitative results on the LOL-v2 real dataset. We retrained Retinexformer based on the code released by the authors with their recommended training recipe and dataset split.**
|  | LPIPS$\downarrow$ | mAP$\uparrow$ | AP50$\uparrow$ |
| --- | --- | --- | --- |
| RetinexFormer | 0.1863 | 34.0 | 64.2 |
| RetinexFormer + LIME-Eval | 0.1625$(\mathbf{0.0238\downarrow})$ | 34.7 $\mathbf{(0.7\uparrow})$ | 64.9 $(\mathbf{0.7\uparrow})$ |

Figure: (a) YOLOX-Tiny
Refer to caption: /html/2410.08810/assets/fig/yoloxtslx.png

### 5.3 Backpropgation of Energy Helps Enhancers

Since our energy function is differentiable, it can serve as a loss function to provide additional regularization for low-light enhancers. We demonstrate this by integrating our energy function into the training process of Retinexformer.
As shown in Tab. [3](#S5.T3), the model enhanced with our energy function enjoys a favorable gains in both low-level metric(LPIPS) and downstream detection metrics (mAP and AP50). These results indicate that the energy function contributes effectively to regularization, aiding the enhancement process. Qualitative comparison can be found in the appendix.

#### 5.3.1 Ablation Study

The Effect of Hyper-parameter $T$. Given that our framework relies on a single hyper-parameter, $T$, we have conducted a series of experiments to assess its sensitivity and impact on performance. The experimental results are systematically presented in Fig. [6](#S5.F6). The findings show that our energy-based metric maintains a strong correlation across a range of values from 0.01 to 0.0001, indicating that the method stays stable over a broad spectrum of temperature settings.

The Effect of Model Size. We also explored the impact of model size on the performance of our framework by experimenting with different versions of the YOLOX architecture: YOLOX-Tiny, YOLOX-s, YOLOX-l, and YOLOX-x. This investigation aims to understand how the size of the base detector influences detection accuracy, processing speed, and overall system efficiency within our enhanced low-light image evaluation setup. The outcomes of these experiments, which detail the trade-offs associated with each model size, are documented in Fig. [7](#S5.F7). As we can observe from the figure, the larger the model, the stronger the correlation energy with the mAP will be. When we scale the model back to YOLOX-Tiny, the connection between energy and mAP vanishes.

## 6 Conclusion

In this study, we have provided a comprehensive evaluation of low-light image enhancement techniques, with a particular focus on the application of an energy-based model for assessment. Our investigation, rooted in the analysis of object detection on the ExDark dataset and utilizing the medium version of YOLOX as the base detector, has illuminated the limitations inherent in traditional evaluation methodologies that rely on retraining recognition models on enhanced images. We have demonstrated that such approaches can lead to overfitting, thus skewing the fairness and accuracy of evaluations.

By adopting an energy-based model for evaluation, we have introduced a novel framework that sidesteps the pitfalls of overfitting and offers a more equitable measure of an enhancement technique’s effectiveness. Our findings reveal that this method not only provides a more accurate reflection of an algorithm’s performance in enhancing low-light images but also presents a promising avenue for future research in image processing and evaluation. The implications of our study extend beyond the immediate scope of low-light image enhancement, suggesting a broader applicability of energy-based models in the evaluation of image processing techniques.

## Appendix A Appendix

### A.1 LIME-bench Details

Our LIME-bench collects data through an online user survey. A screenshot of our system can be found in Fig. Specifically, we randomly select an image, a particular attribute, and two enhancement methods (including the original input). Users are then asked to choose between four options: Image 1 is better, Image 2 is better, both are good, or both are bad. We adopt Elo rating system to obtain final rating for each method. For a pair of user preference, if user can tell which one is better, then we update the score with $k$ set to 16. However, when user voted for ”both are good/well”, we treat it as the two competitor both win/lose from the original input, since this requires 2 times of score update, we down-weighted the $k$ to 8 in this situation.

Figure: Figure 8: A screenshot of our online user survey system.
Refer to caption: /html/2410.08810/assets/fig/im.png

### A.2 Implementation Details

In this work, we use PyTorch to implement our LIME-eval framework. All of our experiments are carried out on NVIDIA RTX3090 GPUs. The detectors and enhancers are trained with the code and configuration (optimizer, learning rate random seeds, etc.) provided by the authors to provide a best-effort fair comparison, except for Tab. [1](#S3.T1), where we carefully tuned the parameters for the best performance since no existing training recipe for us to follow.

#### A.2.1 Data Synthesis

The data synthesis pipeline we have used comprises two types of distortions, the settings of which are as follows:

- 1.
Exposure Shifts
•
Under Exposure. Gamma correction with $\gamma=1.5,2$
•
Over Exposure. Gamma correction with $\gamma=0.75,0.5$
•
Original Exposure. (Gamma correction with $\gamma=1$)
- 2.
Degradation
•
Gaussian Blur with $\sigma_{s}=0.1,0.2,0.4,0.8,1.6$
•
Gaussian Noise under level $5,10,15,20,25$
•
Impulse Noise, amount $=0.01,0.025,0.05,0.075,0.1$
•
Shot Noise under level $60,45,30,20,12$
•
Brightness distortion. First, convert the image into HSV color space, then add $0.1,0.2,0.3,0.4,0.5$ to $V$.
•
Saturate distortion. First, convert the image into HSV color space, then scale component $S$ with $\alpha S+\beta$, where $\alpha=0.3,0.1,2,5,20$ and $\beta=0,0,0,0.1,0.2$

For every image, we first select a degradation and then perform an exposure shift. In this way, we generate 150 distorted datasets for correlation analysis.

### A.3 More Qualitative Comparisons

In this section, we first exhibited more comparison over existing methods on the ExDark dataset in Fig. [9](#A1.F9). As can be found in these Bread (Guo & Hu, ) presenting superior visual effects in most cases, the IAT (Cui et al., ) has the second-best performance where there exist artifacts in some cases. The ZeroDCE (Guo et al., ) has a good color restoration performance, but it suffers from unpleasant noise due to its non-denosing nature. The outputs of RetinexFormer (Cai et al., ) have artifacts in multiple cases.

Figure: (a) Input
Refer to caption: /html/2410.08810/assets/supp/input/2015_00033.jpg

Qualitative results for models equipped with our energy function is presented in Fig.[10](#A1.F10). The model equipped with the energy function method can produce more natural outputs. However, as shown in Fig.[11](#A1.F11), even equipped with our energy loss function, the model still failed to remove severe artifacts in some cases, including persistent checkerboard artifacts in the sky area, and the tendency for pixels in over-exposed areas to be out-of-bounds. Investigating this phenomenon and developing more sophisticated measures to alleviate it is out of the scope of this paper. Yet the case still demonstrates our ability to adjust images to a more natural exposure level.

Figure: (a) Input
Refer to caption: /html/2410.08810/assets/fig/input_2015_00017.jpg

Figure: (a) Input
Refer to caption: /html/2410.08810/assets/fig/input_2015_00276.jpg

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. Higher detection accuracy does not necessarily correlate with superior enhancement quality, which aligns with my understanding of low-light enhancement models. There exists a gap between low-level, human-visual-oriented perception and high-level, machine-vision-oriented objectives. Additionally, as noted in line 68, 'Is fine-tuning detectors a valid approach to evaluating enhancers?' this raises a valuable point. Relying on fine-tuning detectors to fully assess low-light detection performance can lead to overfitting and is not necessarily a scientifically rigorous approach.

2. The paper is well written, easy to understand, and authors conduct extensive experiments to validate their approach, offering thorough insights into the correlations between detector performance, human ratings, and LIME-Eval metrics.

3. LIME-Bench provides a structured way to incorporate human preferences into evaluation, bridging the gap between machine-based metrics and human perceptual quality. Which could also be helpful to future low-light vision research.
summary: The work: LIME-Eval: Rethinking Low-Light Image Enhancement Evaluation via Object Detection, critiques the standard practice of evaluating low-light image enhancement by retraining object detectors on enhanced images—a method prone to overfitting and reduced reliability. Instead, the authors propose LIME-Eval, an energy-based evaluation framework that eliminates the need for retraining detectors or relying on annotated data. They also introduce LIME-Bench, an online platform that collects human preferences on low-light enhancement to align automated metrics with human perception. Experiments on datasets like ExDark demonstrate LIME-Eval’s effectiveness in evaluating enhanced images in a way that closely mirrors human judgment.
weaknesses: 1. In the 'enhanced before data augmentation' setting (Table.1), is there a process like 'color jitter'? I think 'color jitter' could introduce significant interference, as it might substantially degrade the results of low-light enhancement if applied. Also the normalization of image maybe also affect the low-light enhancement models. Could the authors provide more detail on this part? 

2. The paper mainly uses ExDark and synthetic datasets, can this evaluation tool also been used in the low-light face detection dataset UG2+ DARK FACE ?

3. Some previous low-light object detection works, although have not used pre-defined enhancement modules, they remain closely related to this study. They did not use image enhancement models but instead employed other approaches, such as self-supervised learning and domain adaptation, to address the issue of low-light object detection. It would be beneficial for the authors to reference and discuss these works in the related works section (e.g., [1, 2, 3]).

[1]. Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection, ICCV 2021

[2]. Similarity min-max: Zero-shot day-night domain adaptation, ICCV 2023

[3]. Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation, CVPR 2024

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: 1. this paper addresses the challenges of evaluating low-light enhancement techniques via the proposed LIME-Bench online platform, collecting human preference low-light images for low-light enhancement techniques, and the LIME-Eval framework without fine-tuned detection models and annotation labels.

2. the paper demonstrates the uncorrelation of high detection performance to the enhancement quality of the enhanced images through comprehensive studies and analysis. This provides a scientific foundation to support their proposed LIME-Bench platform and LIME-Eval framework.

3. the paper addresses the challenge of evaluating low-light enhancement techniques without fine-tuning detection models and acquiring annotation labels through the energy-based modeling technique.
summary: This paper addresses the challenge of evaluating low-light image enhancement without paired ground-truth data, as current high-level vision task evaluations can often lead to overfitting and unreliable metrics. To improve this, the authors introduce LIME-Bench, an online platform that collects human preference data for low-light enhancement, creating a valuable dataset to correlate human perception with automated metrics. They further propose LIME-Eval, a novel evaluation framework using pre-trained detectors without retraining or annotation dependencies, to assess enhanced image quality. Through energy-based confidence map assessment, LIME-Eval effectively gauges enhancement accuracy. Extensive experiments validate LIME-Eval's effectiveness, and the code will be publicly available.
weaknesses: 1. the purpose of consdering background prediction in the LIME-Eval framework is vauge. It would be great for authors to describe the benefit of integrating background prediction into the energy-based evaluation function.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: See Weaknesses.
soundness: 1
strengths: In the background of low-light enhancement, the quality of the enhanced image may indeed be independent of the target detection performance. To verify this phenomenon, they established an online benchmarking platform, LIME - Bench, and developed a label-free assessment metric, LIME-Eval, for quality evaluation of image enhancement. This is a metric that considers both human vision and machine vision.
summary: The paper argues that using object detectors trained on enhanced low-light images for evaluation is prone to overfitting, reducing reliability. To address this, the authors introduce LIME-Bench, an online platform collecting human preferences to validate the link between human perception and automated metrics. They also present LIME-Eval, a framework using pre-trained detectors without annotations to evaluate image quality, avoiding retraining biases and dependency on dim image annotations.
weaknesses: I think the experiment lacks conviction.
1. Qualitative comparisons on samples from the ExDark dataset are inadequate and even wrong in my opinion. The reason for this is that the augmentation strategy is random and not a fair input to the target detector.
2. How is LIME-Eval integrated into Retinexformer? I don't see it in the appendix either, rather than you just stating the training details.
3. If LIME-Eva is integrated into the detection, will it participate in the calculation of the parameter gradient of the neural network? If it is involved, then I think that LIME-Eva just has the efficacy of a loss function, since the low-light enhanced image is then fed to the target detector for training, which is not referential in nature, as you conclude in Table I.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors need to solve the questions mentioned, particularly question 2, in the weakness part.
soundness: 2
strengths: This paper provides a systematic analysis of methods for measuring the quality of enhanced images in low-light enhancement tasks using object detection. It identifies the shortcomings of previous methods and introduces a new low-light dataset based on human preferences, analyzing the correlation between the evaluations of the object detector and human preferences. Finally, the authors propose a novel unlabeled image quality assessment method that addresses a significant challenge in the current field of low-light enhancement. Overall, this study holds substantial research significance in the domain.
summary: This paper reconsiders the evaluation of enhancement quality in low-light tasks, identifies the shortcomings of previous enhancement quality assessment methods, proposes a new dataset based on human preferences, and introduces a novel unlabeled image quality assessment method for enhanced images. The practical significance of the issues addressed in this study is commendable. However, the analysis of the problems is somewhat lacking, and the experimental section could be further improved, as detailed in the Weaknesses section.
weaknesses: 1. In Chapter 3, the introduction of the need to verify the issue of information loss due to quantization seems abrupt. The earlier sections do not mention the implementation of quantization strategies during the detector's testing process. If quantization was indeed utilized, please provide a brief overview of the quantization process.
2. In Chapter 5, the author fails to analyze how to achieve unlabeled evaluation metrics. The methods section merely outlines the computational processes related to detection-oriented energy-based modeling. The transition from problem introduction to method description is rather abrupt and lacks an analysis of this issue, such as: why is this approach capable of evaluating the quality of image enhancement in an unlabeled manner? Why is the image with the lowest energy value considered optimal? The author should provide relevant analytical descriptions.
3. It is necessary to standardize the meaning of variable x_bg. Why is it referred to as "object output" in line 402, while in line 415 it denotes "background prediction"? Additionally, the method of obtaining x_bg should be specified in this paper. If the network produces multiple object outputs, how should we derive x_bg?
4. In Chapter 5.2, Table 2 presents a comparison of the Spearman correlation coefficients for only IQA algorithm, which is insufficient. I recommend including Spearman correlation coefficients for several additional algorithms to provide a more convincing conclusion.
5. In Chapter 5.3, Table 3 presents the improvements of the method on the object detection metrics mAP and AP50. The author should also indicate whether there are improvements in other metrics (such as recall mAR and AR50). Furthermore, the author could include experimental analyses of additional object detection algorithms (e.g., YOLOX) to enhance the persuasiveness of the conclusions.
6. The article lacks citations for certain terms and methods, such as YOLOX, LOL dataset, and LPIPS. The author should include the relevant references in the text and check for any other missing citations.
7. This paper only discusses the application of detection-oriented energy-based modeling in low-light enhancement tasks. Is this evaluation strategy also applicable to other enhancement tasks (such as rain removal, fog removal, deblurring, etc.)? If possible, I suggest that the authors include a section reflecting on this aspect, such as adding it to the future work section.

Meta-review: The paper addresses the challenges in evaluating low-light image enhancement due to the lack of ground-truth references. It critiques a widely-used evaluation approach, which measures object detection accuracy on enhanced low-light images, arguing that it is prone to overfitting and unreliable. To address this, the authors propose a LIME-Bench, an online benchmarking platform designed to gather human preferences for low-light image enhancement. Besides, they propose a novel evaluation framework, LIME-Eval, that uses pre-trained object detectors without requiring annotations.

This paper receives significant dispersed scores among the reviews. The two reviewers, 3LRc and egde, largely appreciate the paper's contributions, particularly its emphasis on highlighting the distinction between low-light enhancement and detection. Besides, they all think this paper is well-written and well-organized.

The major negative comments come from Reviewer nSif, concerning center on the adequacy of the experiments, particularly the fairness of the augmentation strategy, the role of LIME-Eval in detection, and its integration with Retinexformer, which lacks clarity and rigor. These issues indicate that the methodology and experimental validation require further refinement, making the paper unsuitable for acceptance at this stage.

Decision: Reject