Title: CLODE: Continuous Exposure Learning for Low-Light Image Enhancement using Neural ODEs
Venue: NeurIPS 2024
Accepted: no
Keywords: Low-light Image enhancement

Image enhancement

Neural ODE

Unsupervised
Abstract: Low-light image enhancement poses a significant challenge due to the limited information captured by image sensors in low-light environments. 
Despite recent improvements in deep learning models, the lack of paired training datasets remains a significant obstacle. 
Therefore, unsupervised methods have emerged as a promising solution. 
In this work, we focus on the strength of curve-adjustment-based approaches to tackle unsupervised methods. 
The majority of existing unsupervised curve-adjustment approaches iteratively estimate higher order curve parameters to enhance the exposure of images while efficiently preserving the details of the images. 
However, the convergence of the enhancement procedure cannot be guaranteed, leading to sensitivity to the number of iterations and limited performance.
To address this problem, we consider the iterative curve-adjustment update process as a dynamic system and formulate it as a Neural Ordinary Differential Equations (NODE) for the first time, and this allows us to learn a continuous dynamics of the latent image. 
The strategy of utilizing NODE to leverage continuous dynamics in iterative methods enhances unsupervised learning and aids in achieving better convergence compared to discrete-space approaches. Consequently, we achieve state-of-the-art performance in unsupervised low-light image enhancement across various benchmark datasets.

Abstract

Low-light image enhancement poses a significant challenge due to the limited information captured by image sensors in low-light environments. Despite recent improvements in deep learning models, the lack of paired training datasets remains a significant obstacle. Therefore, unsupervised methods have emerged as a promising solution. In this work, we focus on the strength of curve-adjustment-based approaches to tackle unsupervised methods. The majority of existing unsupervised curve-adjustment approaches iteratively estimate higher order curve parameters to enhance the exposure of images while efficiently preserving the details of the images. However, the convergence of the enhancement procedure cannot be guaranteed, leading to sensitivity to the number of iterations and limited performance. To address this problem, we consider the iterative curve-adjustment update process as a dynamic system and formulate it as a Neural Ordinary Differential Equations (NODE) for the first time, and this allows us to learn a continuous dynamics of the latent image. The strategy of utilizing NODE to leverage continuous dynamics in iterative methods enhances unsupervised learning and aids in achieving better convergence compared to discrete-space approaches. Consequently, we achieve state-of-the-art performance in unsupervised low-light image enhancement across various benchmark datasets.

1 Introduction

Images taken in various low-light environments suffer from insufficient light, leading to the capture 20 of limited information by the camera's image sensor. Therefore, many studies have been conducted 21 to improve the quality of the low-light images and achieve images with optimal exposure levels. 22 In particular, recent supervision-based deep learning approaches [1, 2, 3] have shown remarkable 23 performance in enhancing low-light images. However, the process of collecting pairs of low-light 24 scenes and their corresponding ground-truth images for supervised learning is time consuming and 25 resource intensive. As a result, unsupervised approaches that rely solely on low-light images have 26 been proposed to address this problem. 27

Among many unsupervised low-light image enhancement approaches, curve-adjustment-based meth28

ods, conventionally used in photo editing software ( e.g ., Photoshop), have received much attention. 29

After the introduction of first learning-based curve-adjustment work by Yuan and Sun [7], iter30

ative curve-adjustment-based methods have been explored in various subsequent studies. These 31

unsupervised methods achieve enhancement without using the ground-truth images by fitting the 32

brightness values of pixels in the input image to specific curves. In addition, it is advantageous to 33

preserve local structural information adaptively by allowing efficient pixel-by-pixel computations. 34

For example, ZeroDCE [6, 8] introduced a fast and lightweight neural network to predict pixel-wise 35

curve parameter maps within a fixed iteration step. In addition, ReLLIE [9] produced more accurate 36

(a)

ZeroDCE

RetinexFormer

CLODE

(Ours)

(b)

Supervised

Unsupervised

22

CLODE

RetinexFormer

21

PSNR (dB)

20

LLFlow

19

URetinexNet

18

FourLLIE

PairLIE

Night-Enhancement

17

KinD

16

15

SCI

103

RUAS

104

ZeroDCE

105

RetinexNet

106

DRBN

MSEC

107

10#

Number of Parameters

image enhancement results by using reinforcement learning to predict the curve parameter map at 37 each iteration step, with users able to adjust the number of iterations. 38

In general, these curve-adjustment-based methods, which have fewer parameters, offer the advantage 39 of fast and efficient training and also demonstrate the effectiveness of using higher-order curves 40 for low-light image adjustment. However, conventional iterative approaches in discrete-space with 41 fixed update steps do not arrive at the optimal solution and cannot guarantee convergence of the 42 optimization. Therefore, we alleviate this problem in the discrete-space updating process of existing 43 methods. In doing so, we bring out the strengths of curve fitting methods by reformulating the 44 iterative update formula into ordinary differential equations, which allows the iterative approach to be 45 transformed from discrete-space to continuous-space and find input-specific higher-order curves until 46 convergence within a specified tolerance. To be specific, we present the Neural Ordinary Differential 47 Equations (NODE) model for the low-light enhancement task for the first time. By solving the 48 NODE problem using conventional ODE solvers, we obtain better approximate solutions to the 49 curve-adjustment problem, producing more accurate results than conventional results from iterative 50 updates in discrete-space by exploring the continuous exposure dynamics. In this work, we introduce 51 C ontinuous exposure learning for L ow-light image enhancement using neural O rdinary D ifferential 52 E quations ( CLODE ), which is the first dynamic system for low-light image enhancement. Our main 53 contributions can be summarized as follows: 54

55

56

CLODE is the first approach to formulate the higher-order curve estimation problem as a NODE problem, enabling effective and accurate solutions with standard ODE solvers.

57

58

59

60

61

62

63

64

By transforming the discrete update formula into NODE, which is solvable in continuousspace, we significantly enhance the unsupervised low-light image enhancement results across various benchmark datasets as shown in Fig. 1. This effectively bridges the performance gap between supervised and unsupervised approaches.

CLODE also offers user controllability without altering the network architecture, enabling users to manually adjust the desired level of exposure as needed.

2 Related works

2.1 Unsupervised Low-light Image Enhancement

Obtaining well-exposed ground-truth images paired with corresponding low-light images is inherently 65 challenging, which limits the use of supervised learning in low-light image enhancement. To address 66 this limitation, many unsupervised methods have been developed to tackle the problem. First, there 67 are some approaches [10, 11, 12, 13] that utilize the principles of retinex-theory. Among them, 68 PairLIE [13] utilizes retinex-theory to identify the reflectance and illumination, and employs gamma 69 correction with user-defined gamma values to enhance the illumination. In addition, UDCN [14] and 70 HEP [15] use histogram equalization results as a reference for exposure enhancement. Moreover, 71 recent approaches using GANs have shown remarkable improvements by additionally utilizing 72 unpaired images of normal exposed [16, 17]. Lastly, there are curve-adjustment-based methods [6, 8, 73 18, 9] that transform images through tone mapping. These methods have advanced the curve-fitting 74 techniques from traditional editing tools into deep learning-based approaches, enhancing images by 75 predicting the fitting curves pixel-by-pixel. By repeating the pixel-wise curve fitting and exposure 76 enhancement for a fixed number of iterations in discrete-space, these approaches aim to handle locally 77 varying exposure levels ( i.e ., single image with both underexposed and overexposed areas) in an 78 unsupervised manner. Our CLODE also follows this unsupervised curve-adjustment-based method 79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

and reformulates the curve-fitting problem into a neural ordinary differential equation (NODE). By solving the NODE problem using conventional ODE solvers, we increase the accuracy of curve fitting and thus significantly improve the performance of low-light image enhancement.

2.2 Neural Ordinary Differential Equations

An ordinary differential equation (ODE) is a fundamental concept in mathematics that describes how a function changes with respect to a single variable. It captures the relationship between a function and its derivatives, providing a powerful tool for modeling dynamic systems, such as Newton's Second Law of Motion. To effectively apply the strength of ordinary differential equations to the deep learning model, the concept of neural ordinary differential equations (NODE) is introduced in [19]. The use of NODE facilitates model definition and evaluation, highlighting its effectiveness in parameter efficiency, adaptive computation, and modeling continuous data. In order to effectively capture more complicated functions, the Augmented Neural ODE (ANODE) [20] has been introduced. Furthermore, for seamless continuous time-series modeling, Latent ODE [21] is proposed and recently, ClimODE [22] proposed a continuous-time NODE models for numerical weather prediction. To be specific, in the field of computer vision, the Vid-ODE approach [23] has been introduced to generate continuous-time videos. NODEO [24] has presented a versatile architecture tailored for deformable image registration, and a temporal deformation model using the capabilities of NODE has been developed in [25] to address the challenges associated with future prediction tasks in the context of 4D reconstruction. With advantages like continuous-space modeling, adaptive computation, and memory efficiency, NODE [19] is utilized in various deep learning tasks. However, it has not been extensively explored in the field of image restoration. While NODE-SR [26] has been introduced to address the arbitrary scale super-resolution problem, our methodology marks the first application in image exposure enhancement. In contrast to NODE-SR [26], which learns the continuous variation of the scaling factor for the arbitrary scale super-resolution problem, our CLODE learns the continuous variation of image exposure through curve-adjustment.

3 Proposed Method

3.1 Preliminary

In photo editing applications, the curve-adjustment method is often used to adjust the tone of input images and provides effective exposure control. While this method is useful for pixel-wise manipulation, it is not well suited for images that contain areas of extreme over- or under-exposure. Additionally, a notable drawback of this approach is its reliance on manual adjustments ( e.g ., the number of updates) by the user for each input image. This can be time-consuming and potentially less accurate in certain scenarios. To address this problem, Yuan and Sun [7] have proposed a solution that aims to mitigate the limitations of manual adjustments. They introduced an automated approach that involves estimating an image-specific S -shaped nonlinear tone curve (referred to as an S -curve) tailored to each input image. Specifically, for a given low-light image I 0 , where each pixel value is in the range [0, 1], the S -curve formula for the enhanced image I ′ 0 can be represented as follows:

where ϕ s and ϕ h represent parameters for the amount of shadow and highlight, respectively. The function P ∆ serves as an increasing function for the adjustment that manipulates the intensity of individual pixels within the input of the function.

While Eq.1 allows for adjusting the brightness of an entire image using a single global curve parameter, existing iterative curve-adjustments approaches [6, 8, 9, 27] operate on a pixel-wise basis of the input images. Furthermore, they introduce the necessity of higher-order curves, which enhances images by fitting higher-order curves for fixed iteration steps while using a deep learning model to predict curve parameters on a pixel-by-pixel basis. Specifically the update formula enhances an image I n at the n -th step to an image I n +1 at the next step as follows:

where A n ∈ R C × H × W represents a pixel-wise varying curve parameter map and C , H , and W represent the number of channels, height, and width of the image I n , and ⊗ operation denotes elementwise multiplication. Note that, the elements of A n corresponding to the curve parameters at each

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

𝐼

!

𝐼

"

𝐼

!

𝐼 "

!

𝒜

!

Eq. 9

𝑑𝐼

!

𝑑𝑡

𝑓

"

𝐼

"

. . .

𝐼

#

𝐼

#

=𝑂𝐷𝐸_𝑆𝑜𝑙𝑣𝑒𝑟(𝐼

$

,

0, 𝑇 , 𝑓

"

)

𝑡

𝑓

"

𝒇

𝜽

: ODEfunc

Noise Removal

Module (

𝑔

)

Curve Parameter

Estimation (

ℎ

)

Eq.5

Trajectory

of improvement

+

+

+

𝐼

#

=𝐼

$

+	5

$

#

𝑓

"

𝐼

!

, 𝑡 𝑑𝑡

. . .

𝑓

"

. . .

+

≈

𝑓

"

(b) ODEfunc

(a) Continuous update procedure of CLODE

pixel location are in the range [ -1 , 1] and determine the quadratic curve for the pixel-wise exposure adjustment during the enhancement process. Conventional curve-adjustment methods [6, 8, 9, 18] iteratively follow this process for N times, fitting an appropriate higher-order curve to produce the final well-exposed output image. On the contrary, our CLODE performs curve adjustment for image enhancement by reformulating Eq. 2 as an ordinary differential equation. This approach facilitates memory-efficient training and yields more accurate results through adaptive computation using modern ODE solvers.

3.2 Continuous Exposure Learning for Low-light Image Enhancement using Neural ODEs

Although conventional curve-adjustment-based iterative methods offer advantages in terms of lightweight network architecture and local robustness, these approaches cannot guarantee convergence of the update process. ZeroDCE [6] empirically determines the iteration number N and enhances low-light images by iterating the curve-adjustment formula 8 (= N ) times. While ReLLIE [9] provides users with optional flexibility, it requires manual selection of the value of N for each input image to further improve image quality. To tackle this challenge in optimization, we reformulate the curve-adjustment-based formula outlined in Eq. 2 as a Neural Ordinary Differential Equations (NODE). Then, we can solve the NODE with conventional ODE solvers ( e.g ., Euler, RK4, dopri5) which guarantees the convergence of loss within tolerances. Specifically, we reformulate the original curve-adjustment-based formula by introducing a continuous state t instead of using the discrete state n as follows:

where f θ is a neural network with trainable parameters θ that satisfies f θ ( I t , t ) = A t ⊗ I t ⊗ (1 -I t ) . Then, we can parameterize the derivative of the enhanced image during the update using the network f θ if the continuous update step is very small, and it is given by,

By transforming the original curve fitting problem into a NODE problem with an initial condition I 0 , we can estimate not only the derivative value of each state but also recover the enhanced image by solving the problem, and the initial value problem is given by,

where I T denotes the well-exposed image at the final state T . Finally, the low-light image enhancement process to output I T is accomplished by using the ODE solver as:

where ODE _ Solver denotes a conventional algorithm for solving the ordinary differential equations. 156 In our experiments, CLODE adopts the well-known dopri5 (Dormand-Prince 5th order Runge-Kutta) 157 as an adaptive ODE solver, that determines an input-specific number of iterations for each input and 158 dynamically adjusts the step size. Using the adaptive solver, we can adaptively compute the optimal 159 state for different exposure levels, thereby enabling a more accurate approximation of the solution. 160 This is in contrast to conventional methods, which use the same fixed number of iterations for all 161 input images and cannot guarantee optimality and convergence. To the best of our knowledge, our 162 approach is the first to define the low-light image enhancement problem as a novel NODE problem 163

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

3.2.1 ODE function (ODEfunc)

We can solve the NODE problem in Eq. 5 by integrating f θ over the time interval [0 , T ] with the given initial value I 0 (e.g., a low-light image). In practice, conventional ODE solvers are used to address this problem, iteratively enhancing the low-light images using Eq. 3. In Fig. 2(a), we illustrate the continuous update procedure of our CLODE approach. Notably, the ODE function (ODEfunc) f θ computes continuous dynamics of the latent image and is a key element in the update procedure. The detailed configuration of our ODEfunc f θ is shown in Fig. 2(b). To be specific, our ODEfunc includes Noise Removal ( g ) and the Curve Parameter Estimation ( h ) modules with trainable parameters, and outputs dI t dt , the continuous dynamics of I t . Please refer to Appendix A.1.2 for more details.

Noise Removal In the ODEfunc, we first employ a pre-processing step to eliminate the artifacts from I t and generate the denoised image ˜ I t in order to produce more accurate curve adjustment parameters A t . To minimize computational costs within the f θ , we employ a simple and lightweight three-layer convolutional neural network g as our Noise Removal module, expressed as follows:

The refined image ˜ I t is then used as the input to the subsequent Curve Parameter Estimation stage.

Curve Parameter Estimation Inspired by [7, 28], to enhance both under- and over- exposed areas, we not only use the denoised image ˜ I t and its inverted version (1 -˜ I t ) as inputs to the Curve Parameter Estimation module. The formulation is given by:

where A t represents the curve parameter map at t , and h represents the Curve Parameter Estimation module. For efficacy, this module is also a lightweight convolutional neural network. In particular, we apply layer normalization [29] to all intermediate features. Notably, the use of layer normalization enables CLODE to handle the diverse exposure ranges of input images. Furthermore, all convolutional layers within the Curve Parameter Estimation module h take the continuous state t as a conditional input, allowing for time-varying outputs during the integration interval [0 , T ] as in [19].

Continuous Dynamics Lastly, the derivative value of the one-step state at t is computed in our ODEfunc, and it is expressed as follows:

Notably, unlike conventional curve-adjustment-based update formulas that discretize update steps, our continuous dynamics allows the desired level of accuracy and produces more accurate solutions.

3.3 Inference Process of CLODE

Inference Process Given a low-light input image I 0 , CLODE undergoes successive image enhancement through f θ until convergence within the specified tolerance of the ODE solvers, resulting in a well-exposed image I T . Note that, the output image I T may contain some noise that is amplified during the image enhancement process. Therefore, we use the noise-free image ˜ I T as our final outcome by applying the Noise Removal module g .

User Controllable Design CLODE learns the low-light exposure adjustment mechanism in the continuous-space, and is trained to output I T by integrating the states from 0 to T in Eq. 5 using a fixed T . However, as shown in Fig. 3, users can manually adjust the integration interval by changing the final state value T at the test stage, allowing them to output images with the preferred exposure level and even produce images darker than the input. In practice, by controlling the final state from -( T +∆ t ) to ( T +∆ t ) , the exposure level of the output image can be easily controlled to provide a more user-friendly exposure level.

𝑰

𝟎

"

𝑰

𝑻

"

𝑰

#𝑻

"

𝑰

$(𝑻$𝚫𝒕)

"

𝑰

#(𝑻$𝚫𝒕)

…

…

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

3.4 Zero-Reference Loss Functions

To address the challenge posed by the lack of ground truth, we use five zero-reference loss functions for unsupervised training.

Spatial Consistency Loss While the given low-light input image I 0 is enhanced during the update procedure, maintaining spatial consistency in the pixel brightness order is crucial for preserving image details. Specifically, we measure the difference in spatial consistency between the input image I 0 and our prediction I T by comparing the differences in neighboring pixel values. Similar to [6], we compute the spatial consistency after applying 4-by-4 average pooling to both I 0 and I T , and the spatial consistency loss L spa is expressed as:

The 4-by-4 average pooling operation is denoted as m 4 ( · ) and Ω( i ) includes neighboring pixels in four directions (left, right, top, bottom) centered at position i . The normalization factor K denotes the number of pixels in the reduced image after the pooling operation, and K is given by H 4 × W 4 × C .

Exposure Loss To enforce a consistent exposure level across pixels, conventional unsupervised methods incorporate exposure guidance into the loss function [6]. Similarly, we introduce a desired exposure level parameter E and define the exposure loss L exp as:

In our experiments, we set the exposure level E to 0.6, which corresponds to the gray level in the RGB color space. To maintain the overall exposure level in the results, we minimize the difference between the pixel values of the predicted image I T and the desired exposure level E after performing a 16-by-16 average pooling operation m 16 ( · ) on the output image I T .

Color Constancy Loss In conventional zero-reference methods, two main approaches are used to enforce spatial color constancy: one based on the retinex-theory, and the other based on the GrayWorld hypothesis in [30]. In this work, the color constancy loss L col is based on the Gray-World hypothesis as in [6, 15], and the formulation is given by,

where R , G , and B are the mean pixel values of the red, green, and blue channels in the predicted image I T , respectively. We minimize the color constancy loss L col to correct the potential color deviations in the enhanced image.

Parameter Regularization Loss To prevent rapid changes of pixel values in nearby regions, we employ the spatial regularization to enforce smoothness among neighboring curve parameter values in A t , and the formulation is given by,

where the linear operations ∇ x and ∇ y compute the horizontal and vertical gradients from the parameter map A t , respectively. For better understanding, we represent T -1 as the stage before the final enhancement. We employ the parameter regularization loss at each update step ( e.g ., red points in Fig. 2 (a)) and accumulate the loss while solving the NODE problem.

Noise Removal Loss To estimate a spatially smooth A t regardless of the noise in the image I t , we use the Noise Removal module ( g ) to remove the noise. To train the Noise Removal module, we utilize a self-supervision-based loss L noise that follows the Noise2Noise approaches [31, 32, 33]. Specifically, we employ the loss introduced in Zeroshot-N2N [33]. Our L noise has two components at state t : the residual loss L t res and the consistency loss L t cons . We minimize these losses using two different down-samplers; D 1 and D 2 . Notably, D 1 and D 2 represent fixed 2D convolutional kernels:

convolutions with a stride of two. First, our L t res fits the noise within I t through a symmetric loss function similar to the approach in [34] and it yields:

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

Next, as in [33], L t cons ensures spatial consistency by maintaining similarity in noise distributions, even if the order of denoising and downsampling is altered. Specifically, L t cons also adopts a symmetric loss and is defined as at each update step ( e.g ., red points in Fig. 2 (a)):

Therefore, our final noise removal loss L noise can be represented accumulating during the update procedure as:

As with Eq. 13, we represent T -1 as the stage before the final enhancement. A more detailed description of the noise removal loss is provided in Appendix A.4.

Final Objective Function The final objective function to optimize is given as follows:

where w spa , w exp , w col , w param , and w noise are hyper-parameters used to control the relative significance of each associated loss during the training process.

4 Experiments

4.1 Implementation Details

Please refer to Appendix A.1 for more implementation details and training scheme. The code will be available upon acceptance.

4.2 Experimental Setup

In this work, we use the LOL [5] and SICE [35] Part1 datasets for training. The results of low-light image enhancement are evaluated on the LOL and LSRW [4] benchmark datasets. In addition, the SICE [35] Part2 dataset is used as a benchmark dataset for evaluation under various exposure conditions. SICE Part2 contains 229 image sequences with different exposure levels, and we use the entire sequences as the evaluation dataset. By default, each comparison model uses its official network weights. In cases where the official code is available but weights are not provided, the models are retrained using the official code and settings, except for ReLLIE [9]. We present the performance of ReLLIE on the LOL dataset as reported in their original manuscript.

4.3 Quantitative Comparisons

First, we quantitatively compare the performance of low-light image enhancement on different datasets. Notably, in the experimental results, CLODE represents our proposed method without requiring additional user input (by default), while CLODE † represents the result of adjusting the final state T to the user's preferred level, as introduced in Sec. 3.3.

In Table 1, we compare the low-light image enhancement performance on the LSRW [4] and LOL [5] benchmark datasets in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The term " GT Mean " refers to the evaluation method used by KinD [36] and LLFlow [1], which matches the average value of the output pixels to that of the ground truth pixels. CLODE and CLODE † outperform other unsupervised learning methods. Notably, CLODE † even surpasses the PSNR of state-of-the-art supervised learning methods by 0.73 dB, when averaging the results from the LSRW and LOL datasets in the rightmost columns, without using GT Mean . Moreover, two notable points can be highlighted in Table 1. First, the effectiveness of using NODE to compute accurate higher order curves is evident, as demonstrated by its superiority over curve-adjustment-based methods; ZeroDCE [6] and ReLLIE [9]. Second, unlike other models trained on the same training dataset (LOL), our model shows robust performance on both the LSRW and LOL test datasets, indicating that our model generalizes better than conventional approaches.

In Table 2, we demonstrate the robustness under various exposure conditions including both underand over- exposures, and evaluate the performance on SICE Part2 [35]. The results show that CLODE exhibits robust performance compared to other models, even under various exposure conditions. It outperforms other unsupervised learning methods, and even when compared to supervised learning

291

292

293

294

295

296

methods, CLODE † and CLODE achieve the best and second best results, respectively. Despite being an unsupervised method, CLODE narrows the performance gap with state-of-the-art supervised methods. Additionally, it operates robustly under challenging conditions such as various exposure conditions in SICE Part2, surpassing supervised approaches. These strengths distinguish CLODE from other unsupervised learning methods.

Input

Ground-Truth

Night-Enhance [17]

LLFlow [1]

RetinexFormer [2]

CLODE (Ours)

SCI [11]

ZeroDCE [6]

PairLIE [13]

4.4 Perceptual and Visual Comparisons

In Table 2, we also provide a perceptual comparison of the results with other methods. The evaluation 297 is conducted on SICE Part2, which includes a combination of underexposed and overexposed images. 298 To measure the perceptual quality, we adopt Learned Perceptual Image Patch Similarity (LPIPS) [43], 299 and non-reference metrics; natural image quality evaluator (NIQE) [44], blind/referenceless image 300 spatial quality evaluator (BRISQUE) [45], perception index (PI) [46], and Entropy [47]. In these 301 four aspects, both CLODE and CLODE † show outstanding performance compared to existing 302 unsupervised methods. The visual results are compared in Fig. 4. CLODE shows robust and natural 303 image enhancement results compared to other comparison methods, regardless of the exposure 304 conditions of the input image. 305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

4.5 Ablation Study

Effectiveness of NODE To validate the impact of NODE, we adjust the curves using the architecture of CLODE in both discrete (w/o NODE) and continuous (w/ NODE) spaces, and we compare the results in Table 3. In the discrete setting, similar to [6], curve parameters for fixed update steps [1, 5, 10, 20, 30] are estimated in parallel ((a1) - (e1)). In the continuous setting, however, curve parameters are estimated sequentially for non-fixed adaptive steps, up to a maximum of 30 steps ((f1)). Results in Table 3 demonstrate that the curve parameters produced during the sequential continuous update procedure are more accurate and verify superior performance of the proposed method over the update procedure in the conventional discrete setting. In addition, in Fig. 5, we visualize the trajectories of improvement by plotting PCA dimension reduction results of latent images during updates. We observe that when curve adjustments are made in continuous space by (f1), the trajectories converge more accurately at the final states compared to (e1). This demonstrates that using NODE to find the optimal state certainly contributes to image enhancement.

Effect of the Modules In Table 4, we conduct ablation experiments on the modules used in ODEfunc f θ . We verify the effects of the Noise Removal module g and the layer normalization (LN) in the Curve Parameter Estimation module h . Each module shows performance improvements compared to the baseline (a2). In particular, our final model (d2) achieves the largest performance gain in terms of PSNR/SSIM. Furthermore, case (c2), which includes layer normalization, has about a 4dB gain in PSNR compared to (a2), which does not include layer normalization. This shows that during the image enhancement process in NODE, it is essential to use layer normalization to normalize each state. The visual results can be seen in Fig. 8 of the Appendix.

5 Limitations

Table 5 shows the performance of PSNR/SSIM, number of parameters, and execution time measured in LSRW [4] using an NVIDIA RTX 4090. CLODE shows the advantage in model size compared to supervised methods. The iterative ODE solving process of CLODE takes longer than lightweight unsupervised models, but it exhibits faster speed and performance comparable to supervised methods. Additionally, a smaller version, CLODE-S in Appendix A.1.2 shows promising enhancement performance with competitive inference time comparable to those of unsupervised models.

6 Conclusions

In this work, we address the unsupervised low-light image enhancement problem by formulating it as a NODE problem. We introduce a novel iterative curve-adjustment approach with NODE, transforming discrete iterative problems into continuous ones. CLODE exhibits superior convergence compared to other unsupervised iterative methods, especially in diverse low-light and multi-exposure scenarios. In conclusion, our method effectively narrows the performance gap between unsupervised and supervised methods across various datasets.

Discrete (w/o NODE)

Input

output

Input

output

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

References

Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In AAAI , volume 36, 2022.

Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One-stage retinex-based transformer for low-light image enhancement. 2023.

Jinhui Hou, Zhiyu Zhu, Junhui Hou, Hui Liu, Huanqiang Zeng, and Hui Yuan. Global structureaware diffusion process for low-light image enhancement. NeurIPS , 2023.

Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han. R2rnet: Low-light image enhancement via real-low to real-normal network. Journal of Visual Communication and Image Representation , 90, 2023.

Wenhan Yang Jiaying Liu Chen Wei, Wenjing Wang. Deep retinex decomposition for low-light enhancement. In BMVC , 2018.

Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In CVPR , 2020.

Lu Yuan and Jian Sun. Automatic exposure correction of consumer photographs. In ECCV . Springer, 2012.

Chongyi Li, Chunle Guo, and Chen Change Loy. Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44, 2021.

Rongkai Zhang, Lanqing Guo, Siyu Huang, and Bihan Wen. Rellie: Deep reinforcement learning for customized low-light image enhancement. In ACMMM , 2021.

Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In CVPR , 2021.

Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In CVPR , 2022.

Zunjin Zhao, Bangshu Xiong, Lei Wang, Qiaofeng Ou, Lei Yu, and Fa Kuang. Retinexdip: A unified deep framework for low-light image enhancement. IEEE Transactions on Circuits and Systems for Video Technology , 32, 2021.

Zhenqi Fu, Yan Yang, Xiaotong Tu, Yue Huang, Xinghao Ding, and Kai-Kuang Ma. Learning a simple low-light image enhancer from paired low-light instances. In CVPR , 2023.

Qiuping Jiang, Yudong Mao, Runmin Cong, Wenqi Ren, Chao Huang, and Feng Shao. Unsupervised decomposition and correction network for low-light image enhancement. IEEE Transactions on Intelligent Transportation Systems , 23(10), 2022.

Feng Zhang, Yuanjie Shao, Yishi Sun, Kai Zhu, Changxin Gao, and Nong Sang. Unsupervised low-light image enhancement via histogram equalization prior. arXiv preprint arXiv:2112.01766 , 2021.

Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on image processing , 30, 2021.

Yeying Jin, Wenhan Yang, and Robby T Tan. Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. In ECCV , 2022.

Chongyi Li, Chunle Guo, Ruicheng Feng, Shangchen Zhou, and Chen Change Loy. Cudi: Curve 383 distillation for efficient and controllable exposure adjustment. arXiv preprint arXiv:2207.14273 , 384 2022. 385

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary 386 differential equations. NeurIPS , 31, 2018. 387

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. NeuraIPS , 32, 388 2019. 389

Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations 390 for irregularly-sampled time series. NeuraIPS , 32, 2019. 391

Yogesh Verma, Markus Heinonen, and Vikas Garg. Climode: Climate forecasting with physics392 informed neural odes. In ICLR , 2023. 393

Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jaegul Choo, Joonseok Lee, Sookyung Kim, and 394 Edward Choi. Vid-ode: Continuous-time video generation with neural ordinary differential 395 equation. In AAAI , volume 35, 2021. 396

397

398

399

Yifan Wu, Tom Z Jiahao, Jiancong Wang, Paul A Yushkevich, M Ani Hsieh, and James C Gee. Nodeo: A neural ordinary differential equation based optimization framework for deformable image registration. In CVPR , 2022.

400

401

Boyan Jiang, Yinda Zhang, Xingkui Wei, Xiangyang Xue, and Yanwei Fu. Learning compositional representation for 4d captures with neural ode. In CVPR , 2021.

Seobin Park and Tae Hyun Kim. Progressive image super-resolution via neural differential 402 equation. In ICASSP , 2022. 403

Jiancheng Huang, Yifan Liu, and Shifeng Chen. Bootstrap diffusion model curve estimation for 404 high resolution low-light image enhancement. In PRICAI . Springer, 2023. 405

Haoyuan Wang, Ke Xu, and Rynson WH Lau. Local color distributions prior for image 406 enhancement. In ECCV . Springer, 2022. 407

408

409

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

410

411

Gershon Buchsbaum. A spatial processor model for object colour perception. Journal of the Franklin institute , 310, 1980.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, 412 and Timo Aila. Noise2Noise: Learning image restoration without clean data. In ICML , 2018. 413

Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighbor: 414 Self-supervised denoising from single noisy images. In CVPR , 2021. 415

Youssef Mansour and Reinhard Heckel. Zero-shot noise2noise: Efficient image denoising 416 without any data. In CVPR , 2023. 417

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , 418 2021. 419

Jianrui Cai, Shuhang Gu, and Lei Zhang. Learning a deep single image contrast enhancer from 420 multi-exposure images. IEEE Transactions on Image Processing , 27, 2018. 421

Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: A practical low-light 422 image enhancer. In ACMMM , 2019. 423

Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. Uretinex424 net: Retinex-based deep unfolding network for low-light image enhancement. In CVPR , 2022. 425

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. Band representation426 based semi-supervised low-light image enhancement: Bridging the gap between signal fidelity 427 and perceptual quality. IEEE Transactions on image processing , 2021. 428

Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photographic 429 global tonal adjustment with a database of input / output image pairs. In CVPR , 2011. 430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

Wenhan Yang, Ye Yuan, Wenqi Ren, Jiaying Liu, Walter J Scheirer, Zhangyang Wang, Taiheng Zhang, Qiaoyong Zhong, Di Xie, Shiliang Pu, et al. Advancing image understanding in poor visibility environments: A collective benchmark study. IEEE Transactions on image processing , 29, 2020.

Jie Huang, Man Zhou, Yajing Liu, Mingde Yao, Feng Zhao, and Zhiwei Xiong. Exposureconsistency representation learning for exposure correction. In ACMMM , 2022.

Jie Huang, Yajing Liu, Feng Zhao, Keyu Yan, Jinghao Zhang, Yukun Huang, Man Zhou, and Zhiwei Xiong. Deep fourier-based exposure correction network with spatial-frequency interaction. In ECCV , 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a 'completely blind' image quality analyzer. IEEE Signal processing letters , 20, 2012.

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing , 21, 2012.

Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. The 2018 pirm challenge on perceptual image super-resolution. In ECCVW , 2018.

Xiaoqiao Chen, Qingyi Zhang, Manhui Lin, Guangyi Yang, and Chu He. No-reference color image quality assessment: From entropy to perceptual quality. EURASIP Journal on Image and Video Processing , 2019.

Ricky T. Q. Chen. torchdiffeq, 2018.

Mahmoud Afifi, Konstantinos G Derpanis, Bjorn Ommer, and Michael S Brown. Learning multi-scale photo exposure correction. In CVPR , 2021.

A Appendix 455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

A.1 Implement Details

The training set of images is resized to 128x128, we employ the Pytorch framework on NVIDIA A6000 GPU with a batch size of 8. The ADAM optimizer is used with default parameters and a fixed learning rate of 1 e -5 to optimize the parameters of our network. The weights for the loss function w col , w param , w spa , w exp and w noise are set to 20, 200, 1, 10 and 1 respectively, to balance the scale of losses. Furthermore, we adopt torchdiffeq [48] for Neural ODEs implementation. The training process is conducted for 100 epochs.

A.1.1 Implementation details of NODE

We utilize the adaptive ODE solver, dopri5 (Dormand-Prince Runge-Kutta of Order 5) for our work. The maximum allowed step for the adaptive solver is set to 30 . The relative and absolute tolerances for the error rate calculation are set uniformly to 1 e -5 . The adaptive solver uses the error rate to determine the steps. Also, the adaptive solver estimates an error rate for the current step t , and if the error exceeds the allowable tolerances, the step is re-done with a smaller step size. This process continues until the error becomes smaller than the provided tolerance. The error rate Γ t at the t -th step is computed as follows:

where atol is absolute tolerance, and rtol is relative tolerate, and the norm being used is a mixed L-infinity/RMS norm. We set both atol and rtol to 1e-5.

A.1.2 Details of the CLODE architecture

This section presents the architectural details of the CLODE network architecture, with a particular 474 focus on the ODEfunc module. The Noise Removal module g employs a simple and lightweight 475 three-layer convolutional network. In Curve Parameter Estimation module h , a shallow network 476 with two branches is utilized, wherein filters of varying sizes are employed at each branch to capture 477 image features across different filter scales. We also provide architectural details of CLODE-S as 478 mentioned in Sec. 5 of the main manuscript. This version omits the Noise Removal module for speed 479

and uses a 2-layer network with 1x1 convolutions.

𝐼

!

𝐼

"

𝐼

!

𝐼 "

!

𝒜

!

Eq. 9

𝑑𝐼

!

𝑑𝑡

𝑓

"

𝐼

"

. . .

𝐼

#

𝐼

#

=𝑂𝐷𝐸_𝑆𝑜𝑙𝑣𝑒𝑟(𝐼

$

,

0, 𝑇 , 𝑓

"

)

𝑡

𝑓

"

𝒇

𝜽

: ODEfunc

Noise Removal

Module (

𝑔

)

Curve Parameter

Estimation (

ℎ

)

Eq.5

Trajectory

of improvement

+

+

+

𝐼

#

=𝐼

$

+	5

$

#

𝑓

"

𝐼

!

, 𝑡 𝑑𝑡

. . .

𝑓

"

. . .

+

≈

𝑓

"

(b) ODEfunc

(a) Continuous update procedure of CLODE

A.2 Impact of Each Loss Functions

480

481

CLODE combines five non-reference loss functions to train NODE, producing optimal improvements. 482 We present ablation experiments for each loss function, and the results are presented in Table 6 and 483 Fig. 7. The results of each image ablation experiment demonstrate that appropriate improvement 484 results can only be obtained when using CLODE with all loss functions. The characteristics of the 485 loss function as observed in each ablation are as follows: ((a3) w/o L exp ) : Brightness improvement 486 is not achieved in low-exposure enhancement. ((b3) w/o L col ) : Severe color distortion occurs in 487 over-exposure enhancement, damaging structural details. ((c3) w/o L param ) : Structural distortion 488 occurs, creating artifacts. ((d3) w/o L spa ) : While showing better results than other experiments, it 489

(d3) w/o

ℒ

𝒔𝒑𝒂

Input

(b3) w/o

ℒ

𝒄𝒐𝒍

(a3) w/o

ℒ

(c3) w/o

ℒ

𝒑𝒂𝒓𝒂𝒎

(e3)

ℒ

+

ℒ

𝒄𝒐𝒍

+

ℒ

𝒑𝒂𝒓𝒂𝒎

+

ℒ

𝒔𝒑𝒂

Input

(e3) w/o

ℒ

𝒏𝒐𝒊𝒔𝒆

(f3) CLODE

口

occurs loss of structural details compared to (e3). ((e3) w/o L Noise ) : Compared to the proposed 490 version (f3), it produces improved results with noise present. 491

492

493

494

495

496

497

498

A.3 Visualization of curve parameter map A

We provide visual comparison results for the module ablation experiments in Sec. 4.5 of the main manuscript. In the visual results without noise removal module (c2), we can observe the noise in A . The enhanced result of (c2) using A with noise shows overall color discrepancy compared to the ground-truth, in contrast to the enhanced result of (d2) where the noise removal module are applied. The enhanced result of (d2) shows robust color similarity with the ground-truth image. We can confirm that removing noise for A is important for curve-adjustment-based method.

Input

Ground-Truth

(d2)

𝒜

(c2)

𝒜

(c2) enhanced

(d2) enhanced

499

500

501

502

503

504

505

506

507

A.4 Background of Noise Removal Loss

In Sec.3.4 we provide information about the zero-reference loss functions that we used. Unlike the others, the Noise Removal Loss ( L noise ) requires more explanation due to its unfamiliarity in the field of low-light enhancement, so we provide additional explanation for it.

A.4.1 Noise2Noise background

In supervised denoising studies, neural networks are aimed at denoising the noisy image y to the clean image x . Since the noisy y is an addition of the clean image x and the noise e , the network is trained to map the noise e which is called Noise2Clean (N2C) method. If the network parameter is ϕ N 2 C , the object function of the supervised denoising method with the network g ϕ can be written as:

Denoising networks can also be trained to output the noisy image y 2 from the noisy input image y 1 508 that comes from the same clean image x . This noise-to-noise manner can be achieved by assuming 509 that the noise has a mean of zero as introduced in Noise2Noise (N2N) [31]. This is the objective 510 function for the N2N network parameter ϕ N 2 N : 511

The N2N manner shows close performance compare to N2C manner with sufficient training data since 512 the objective functions of N2C and N2N are aimed on the same network parameter. If y a = x + e a , 513 y b = x + e b , and the mean value of e a and e b are zero, the proof is as follows: 514

515

516

517

518

519

520

521

522

By Eq. 21 we can confirm that the object of ϕ N 2 C and ϕ N 2 N is the identical one.

A.4.2 Zeroshot Noise2Noise method

In spite of N2N approaches, it is hard to obtain two different noisy images from the same clean scene. To address this hurdle, the Neighbor2Neighbor [32] method is proposed. This allows a pair of noisy images to be augmented from a single noisy image coming from the same clean image. In Zeroshot-N2N [33], which is adopted in our proposed method, Neighbor2Neighbor is achieved by using two different 2D convolutional kernels ( D 1 and D 2 ) on noisy images. If the noisy image is y , a pair of down-sampled images y 1 , y 2 can be represented as:

For a noisy image y with a size of H × W × C , the size of y 1 and y 2 is H 2 × W 2 × C . With 523 downsampled images y 1 and y 2 , the loss optimizes g ϕ to fit the noise as: 524

530

531

532

533

534

535

536

537

538

539

A.4.3 More Quantitative Results

We present the comparison results for non-reference metrics, which we did not include in Table 1. Table 7 demonstrates that CLODE outperforms other unsupervised methods in terms of perceptual quality. Notably, it demonstrates competitive results in terms of BRISQUE and PI, even when compared to state-of-the-art supervised methods.

A.4.4 Comparison with other iterative methods 540

Fig. 9 shows the changes in performance over steps of each curve-adjustment-based method. Each 541 comparison method is retrained for 10 steps in the official code provided by the author. To fix the 542 number of steps in CLODE to 10, we replace CLODE's ODE solver with the Euler method, and 543

Zeroshot-N2N [33] emphasizes that residual learning, a symmetry loss, and an additional coherence525 enhancing term are critical for good performance. Zeroshot-N2N proposes two different loss functions, 526 the residual loss L res and the consistency loss L cons . First, the residual loss optimizes the network 527 g ϕ to fit the noise instead of image. The loss then becomes as: 528

To fit the noise in y 1 and y 2 both, a symmetric loss [34] is applied as: 529

Second, the method constrain consistency by making denoised output of the downsampled image and downsampled result of the denoised image like:

Same as Eq. 25, with the adoption of a symmetric manner, the consistency loss is represented as:

The noise removal loss function L noise in Zeroshot-N2N becomes the sum of Eq. 25 and Eq. 27, expressed as:

referred to it as CLODEEuler . The results show that even within the same number of steps, CLODE544 Euler performs better than other curve adjustment-based methods. Furthermore, the proposed version, 545 CLODE, demonstrates higher performance compared to other methods in most iterative steps. 546

In case of ReLLIE [9], it exhibits a decline in performance after 7 steps, emphasizing the need 547 for careful selection of the number of iterative steps itself to achieve optimal result, this makes the 548 method impractical to use. 549

0.0

0.2

0.4

0.6

0.8

1.0

Step (Scaled to 0-1)

8

10

12

14

16

18

20

PSNR (dB)

Comparison with curve-adjustment-based methods

ZeroDCE (10 steps)

ZeroDCE++ (10 steps)

CLODE-Euler (10 steps)

CLODE (Adaptive steps)

A.5 More visual results

We show additional results for CLODE enhancement that we did not show in the main manuscript due to lack of space. We present additional visual comparison results for PairLIE [13] and NightEnhancement [17], which demonstrated the best quantitative performance among the unsupervised methods in Table 1 of the main manuscript, except for our proposed method (CLODE), in Fig. 10. CLODE shows the most robust enhancement results across various image exposure conditions.

Fig. 11, Fig. 12, Fig. 13 and Fig. 14 show the results for CLODE and CLODE † on LOL [5] and SICE [35] validation dataset. Additionally, Fig. 15 shows the visual results with different exposures for photos extracted from MSEC [49] and the internet (Filckr: CC BY-NC 2.0).

Input

(b)

CLODE

(a)

550

551

552

553

554

555

556

557

558

Ground-Truth

CLODE

Input

CLODE

†

Ground-Truth

CLODE

Input

CLODE

†

EE60

0933

EE60

0933

4

0833

EE8n

Ground-Truth

Input

CLODE

CLODE

†

OYOY

Ground-Truth

Input

CLODE

CLODE

†

CLODE

Input

CLODE

Input

By julochka (Flickr: CC BY-NC 2.0)

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We presented the contributions and effects of our method in the abstract and introduction, and demonstrated them through experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: There is a "Limitation" section containing information about the execution speed of our method.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our paper reports the theoretical approaches of the NODE reformulation process for curve adjustment equations.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper includes main experimental results as well as ablation experiment results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In our paper, we include the rationale behind NODE reformulation along with the workflow, and provide the network architecture in the appendix. This ensures reproducibility, and we will also provide the code upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We included details about the dataset used for training in the main manuscript, while other hyperparameters, weights, etc., are documented in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide detailed information on low-light performance validation, the impact of main contributions, and performance validation through ablation studies.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information about the GPU resources used and the execution time. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the ethical guidelines (Code of Ethics) and ensured compliance.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our proposed model does not include a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We will accurately specify the original owner's license and share the code upon acceptance. Additionally, the source and the original owner's name for the single image used in the appendix have been indicated on the image.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

868

869

870

871

872

873

874

875

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper formulates the higher-order curve estimation problem as a NODE problem, enabling effective and accurate solutions with standard ODE solvers.

STRENGTHS:
This paper is well-written and structurally organised.

WEAKNESSES:
Reference formats are not consistent.

QUESTIONS:
Why choose E=0.6 in Eq.(11)?
The setting of hyper-parameters in Eq.(17) should be mentioned.
What's the challenge of applying ODE to continuous exposure learning?
The background of the proposed method in Figure 4 is quite different from GT, why?

LIMITATIONS:
As described above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly addressed the problem of insufficient data for low-light enhancement. Specifically, it proposed CLODE , which employs Neural Ordinary Differential Equations to learn the continuous dynamics of the latent image for the first time. The experiments demonstrate the CLODE performs better than other unsupervised learning methods.\n\nSTRENGTHS:\n+ This is the first attempt to formulate the higher-order curve estimation problem as a NODE problem.\n+ CLODE can offer user controllability, enabling users to manually adjust exposure.\n\nWEAKNESSES:\n- Details of User Controllable Design. Despite the better result with use control, detail of the users is missing. For example, the number of volunteers, and whether they are banned from the ground truth image before they adjust the output image. Also, involving human feedback bring much more time in the inference stage.\n- In Sec. 3.3 Inference Process, the relationship between the output image IT and noise-free image is questioned. Each iteration includes a noise removal module, yet the output image still contains some noise, contradicting the expectation of a noise-free result in the model.\n- Experimental Setup: The experimental setting described in [1] seems more suitable for unsupervised methods. Using only a single dataset for training in this study does not adequately reflect the advantages of the proposed method. A specific analysis comparing and justifying the differences in experimental setups is necessary.\n- Model Iteration Selection in \"CLODE+\" (Table 2): The manual operation required to select the iteration step raises concerns. How is this value determined to ensure suitable results? This approach appears more suited to image retouching tasks than enhancement.\n- Concern about the fair comparison with previous methods. This paper uses 5 different losses. I wonder whether only part of them is used in previous methods, are the proposed method align with previous methods? For example, some Retinex-based method does not explicitly consider the impact of noise, and they do not have Noise Removal process. Does CLODE still outperform other methods without noise removal? More ablation experiments are needed for thorough explanation.\n- Effectiveness of Noise Removal Module: In the first toy scene in Figure 4, as well as Figures 7 and 8, there is noticeable noise residue and some degree of color distortion, which casts doubt on the effectiveness of CLODE and its noise removal module for low-light enhancement.\n- More explanation of the superiority of CLODE. Can author provide clearer explanation of the mechanism? For example, in Figure 9 of Supp material, is the better results comes from the more iterations, or more iterations at the early stage, where the estimation is harder?\n\n[1] Learning a Simple Low-light Image Enhancer from Paired Low-light Instances\n\nQUESTIONS:\n- In tab. 3, can the continuous method be considered to have an early stop mechanism? While discrete methods adjust in every iteration as discrete value, the continuous method adjusts as a continuous value. Further analysis and additional visual results comparing these methods across iterations are anticipated.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This manuscript introduces CLODE, which learns low-light image enhancement using neural ordinary differential equations (NODE). The key innovation lies in formulating the higher-order curve estimation problem as a NODE problem. Experimental results show that the proposed approach outperforms state-of-the-art unsupervised counterparts across several benchmarks.

STRENGTHS:
1.	The paper is easy to follow.
2.	Using neural ordinary differential equations to address the iterative curve-adjustment update process shows better performance.

WEAKNESSES:
1.	The novelty is limited, and the technical contribution is incremental. Apart from formulating the curve estimation as a NODE problem, the paper lacks innovation，which is the main reason why I gave this paper a lower score.
2.	More strong supervised baselines should be included for reference. Comparing only a few relatively weak baselines can lead to a misunderstanding of the current gap between supervised and unsupervised methods. 
3.	Additionally, the authors should report some perceptual metrics for better comparison.
4.	The writing and the presentation need improvement.

QUESTIONS:
Please refer to `Weakness'.

LIMITATIONS:
The authors have discussed the limitations of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper This paper proposes an ODE-based  method to tackle low-light image enhancement problems. The motivation of the paper is inspired by the observation that the conventional discrete iterative approaches set fixed update steps. It does not only miss the optimal solution and also does not guarantee the convergence.  Hence, the proposed method takes the iterative curve-adjustment approach and formulates them into solving neural ordinary differential equations. This method is used to work with unsupervised learning to estimate the higher order curve parameters to reconstruct image structure details. Comprehensive experiments demonstrate that the proposed method outperforms the baseline methods on LOL and SICE  benchmarking datasets.\n\nSTRENGTHS:\nStrengths\n\n1. This paper proposes a novel method that integrates the neural networks into an ODE optimization framework. The neural network is playing an adaptive set of updatable parameters. \n2. Comprehensive experiments show that the proposed method outperforms the baseline methods in the task of low-light image enhancement. \n3. The motivation of this paper is strong and solid. It is inspired by the drawbacks of the existing methods and tackle the problems directly in the proposed method.\n4. This paper addresses the limitation of the proposed method.\n\nWEAKNESSES:\nWeakness\n1. Based on the visual comparison in Figure 4, the proposed method tends to produce over-exposed areas for highlight regions.  \n2. The processing speed of the proposed method is one of the limitations.\n\nQUESTIONS:\nThis paper is a novel low-light image enhancement work with solid equation derivation to support the its objective. The authors are expected to address the questions raised in the weakness sections in the rebuttal period.\n\nLIMITATIONS:\nThe limitation is included in the main manuscript. The processing speed (inference speed) of the proposed method is slow compared to dedicated supervised DL methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Neural ODE method for curve-adjustment-based low light image enhancement methods to achieve better results which are often sub-optimal for fixed discrete step methods. Specifically, the proposed method reformulates the curve-adjustment-based from the discrete version into the ODE problem by introducing a continuous state. An ODE solver is adopted for the optimization to find the optimal step for the enhancement. Additionally, a simple denosier and a curve parameter estimation module are proposed for noise removal and parameter estimation, respectively. Extensive experiments are conducted to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. Turn the discrete curve-adjustment method into a NODE problem, benefited from the optimization to search for the optimal step.\n2. User control support during inference is good for the application of the proposed method.\n3. The proposed method seems to have good performance over other competitors.\n\nWEAKNESSES:\n1. The proposed method faces color casts, which is obvious in almost all qualitative results, even with a color constraint in loss functions.\n2. The proposed method proposes to denoiser and curve parameter estimator in the NODE framework, however, generalize the method to existing curve-ajustment-based method seems to be a more attractive solution.\n3. The denoiser seems to be weak since there is so much noise left for the qualitative results.\n\nQUESTIONS:\n1. What leads to the color cast, even with color constraints applied. And it seems that the steps increases, the color shifts become stronger. Comparisons with user-controlled result are also evident for the color shifts. An evaluation on color consistency is necessary for this issue.\n2. Is there any potential to generalize such a method to previous methods, like turn previous methods into NODE and find optimal steps for them?\n3. In Table 3 in the main paper and Figure 9 in the supp., the comparisons between discrete methods and the proposed method show good advantage of the proposed method over previous methods. But what is the statistics for a dataset or some test examples, like the most common optimal steps for inextreme cases? It may help to get interesting findings.\n4. Why is the denoiser so weak? Can it be improved by use existing modules?\n\nLIMITATIONS:\nYes, it is discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}