Title: Night-to-Day Translation via Illumination Degradation Disentanglement
Venue: NeurIPS 2024
Accepted: no
Keywords: Image Translation

Nighttime Image Rendering.
Abstract: Night-to-Day translation (Night2Day) aims to achieve day-like nighttime vision. However, processing images with complex degradations in the night using unpaired data still remains a challenge in this field. Previous methods that uniformly mitigate these degradations have proven inadequate in simultaneously restoring daytime domain information and preserving underlying semantics. In this paper, we recognize the different degradation patterns in nighttime images and propose N2D3 (Night to Day via Degradation Disentanglement). It comprises a degradation disentanglement module and a degradation-aware contrastive learning module. Firstly, we extract physical priors from the photometric model derived from the Kubelka-Munk theory. Subsequently, under the guidance of physical priors, we design a disentanglement module to discriminate among different illumination degradation regions. Finally, we introduce degradation-aware contrastive learning to preserve semantic consistency across distinct degradation regions. Our method is evaluated on two public datasets, \textbf{with a significant improvement of 5.4 FID on BDD100K and 10.3 FID on Alderley

Abstract

Night-to-Day translation (Night2Day) aims to achieve day-like vision for nighttime scenes. However, processing night images with complex degradations remains a significant challenge under unpaired conditions. Previous methods that uniformly mitigate these degradations have proven inadequate in simultaneously restoring daytime domain information and preserving underlying semantics. In this paper, we propose N2D3 ( N ight-toD ay via D egradation D isentanglement) to identify different degradation patterns in nighttime images. Specifically, our method comprises a degradation disentanglement module and a degradation-aware contrastive learning module. Firstly, we extract physical priors from a photometric model based on Kubelka-Munk theory. Then, guided by these physical priors, we design a disentanglement module to discriminate among different illumination degradation regions. Finally, we introduce the degradation-aware contrastive learning strategy to preserve semantic consistency across distinct degradation regions. Our method is evaluated on two public datasets, demonstrating a significant improvement of 5.4 FID on BDD100K and 10.3 FID on Alderley .

1 Introduction 16

Nighttime images often suffer from severe information loss, posing significant challenges to both 17

human visual recognition and computer vision tasks including detection, segmentation, etc. [14]. 18

In contrast, daylight images exhibit rich content and intricate details. Achieving day-like nighttime 19

vision remains a primary objective in nighttime perception, sparking numerous pioneering works [30]. 20

Night-to-Day image translation (Night2Day) offers a comprehensive solution to achieve day-like 21

vision at night. The primary goal is to transform images from nighttime to daytime while maintaining 22

their underlying semantic structure. However, achieving this goal is challenging. It requires to process 23

24

complex degraded images using unpaired data, which raises additional difficulties compared to other

image translation tasks. 25

Recently, explorations have been made in Night2Day. Early approaches, such as ToDayGAN, 26

demonstrated the effectiveness of cycle-consistent learning in maintaining semantic structure [1]. 27

Subsequent methods incorporated auxiliary structure regularization techniques, including perceptual 28

loss and uncertainty regularization, to better preserve the original structure [33, 18]. Furthermore, 29

some methods utilized daytime images with nearby GPS locations to aid in coarse structure regular30

ization [26]. However, these methods often neglect the complex degradations at nighttime, applying 31

structure regularization uniformly and resulting in severe artifacts. To address this issue, more recent 32

approaches adopt auxiliary human annotations to maintain semantic consistency, such as segmenta33

tion maps and bounding boxes [16, 22]. Despite their potential, these methods are labor-intensive 34

and challenging, especially since many nighttime scenes are beyond human cognition. 35

Physical

Prior

w Disentangled Regularization

w/o Disentangled Regularization

(c) Visual Comparison

(b) Disentangled Regularization

(a) Disentanglement Process

Darkness

Well-lit

Generated Patches

High-light

Light Effects

The critical limitation of the aforementioned methods is the disregard for complex degraded regions. 36

Specifically, different regions in nighttime images possess varying characteristics, such as extreme 37

darkness, well-lit regions, light effects, etc . Treating all these degraded regions equally could adversely 38

impact the results. As illustrated in Figure 1, our key insight emphasizes that nighttime images suffer 39

40

from various degradations, necessitating customizing restoration for different degradation types.

Intuitively, we manage to disentangle nighttime images into patches according to the recognized 41

degradation type and learn individual restoration patterns for them to enhance the overall performance. 42

Motivated by this point, we propose N2D3 ( N ight to D ay via D egradation D isentanglement), which 43 utilizes Generative Adversarial Networks (GANs) to bridge the domain gap between nighttime and 44 daytime in a degradation-aware manner, as illustrated in Figure 2. There are two modules in N2D3, 45 including physical-informed degradation disentanglement and degradation-aware contrastive learning, 46 which are employed to preserve the semantic structure of nighttime images. In the disentanglement 47 of nighttime degradation, a photometric model tailored to nighttime scenes is conducted to extract 48 physical priors. Subsequently, the illuminance and physical priors are integrated to disentangle 49 regions into darkness, well-lit, high-light, and light effects. Building on this, degradation-aware 50 contrastive learning is designed to constrain the similarity of the source and generated images in 51 different regions. It comprises disentanglement-guided sampling and reweighting strategies. The 52 sampling strategy mines valuable anchors and hard negative examples, while the reweighting process 53 assigns their weights. They enhance vanilla contrastive learning by prioritizing valuable patches with 54 appropriate attention. Ultimately, our method yields highly faithful results that are visually pleasing 55 and beneficial for downstream vision tasks including keypoint matching and semantic segmentation. 56

57

58

59

60

61

62

63

64

65

66

Our contributions are summarized as follows:

We propose the N2D3 translation method based on the illumination degradation disentanglement module, which enables degradation-aware restoration of nighttime images.

We present a novel degradation-aware contrastive learning module to preserve the semantic structure of generated results. The core design incorporates disentanglement-guided sampling and reweighting strategies, which greatly enhance the performance of vanilla contrastive learning.

Experimental results on two public datasets underscore the significance of considering distinct degradation types in nighttime scenes. Our method achieves state-of-the-art performance in visual effects and downstream tasks.

2 Related Work

Unpaired Image-to-Image Translation. Unpaired image-to-image translation addresses the chal67

lenge of lacking paired data, providing an effective self-supervised learning strategy. To overcome the 68 efficiency limitations of traditional cycle-consistency learning, Park et al. , first introduces contrastive 69 learning to this domain, achieving efficient one-sided learning[20]. Following this work, several stud70 ies have improved the contrastive learning by generating hard negative examples [24], re-weighting 71 positive-negative pairs [31], and selecting key samples [9]. Furthermore, other constraints, such as 72 density [27] and path length [28], have been explored in unpaired image translation. However, all 73 these works neglect physical priors in the nighttime, leading to suboptimal results in Night2Day. 74

Physical

Prior

Darkness

Light Effect

Well-lit

𝑥1 𝑦 1

𝑥1 𝑦 2

𝑥1 𝑦 3

𝑥1 𝑦

4

𝑥1 𝑦 5

𝑥1 𝑦 6

𝑥2𝑦1

𝑥2𝑦2

𝑥2𝑦3

𝑥2𝑦 4

𝑥2𝑦5

𝑥2𝑦6

𝑥3𝑦1

𝑥3𝑦2

𝑥3𝑦3

𝑥3𝑦 4

𝑥3𝑦5

𝑥3𝑦6

𝑥4𝑦1

𝑥4𝑦2

𝑥4𝑦3

𝑥4𝑦 4

𝑥4𝑦5

𝑥5𝑦1

𝑥5𝑦2

𝑥5𝑦3

𝑥5𝑦 4

𝑥5𝑦5

𝑥5𝑦6

𝑥6𝑦1

𝑥6𝑦2

𝑥6𝑦3

𝑥6𝑦 4

𝑥6𝑦5

𝑥6𝑦6

𝑥1 𝑦 1

𝑤12

0

0

0

0

𝑤21

𝑥2𝑦2

0

0

0

0

0

0

𝑥3𝑦3

𝑤34

0

0

0

0

𝑤43

𝑥4𝑦 4

0

0

0

0

0

0

𝑥5𝑦5

𝑤56

0

0

0

0

𝑤65

𝑥6𝑦6

min

wij ,i,j∈[1,N] [෍

i

N

෍

j

N

wij

⋅ e

xi y i/β

]

i=1

N

wij = 1, ෍

j=1

N

wij = 1

𝑦1

𝑦2

𝑦4

𝑦5

𝑦6

𝑥1

𝑥2

𝑥3

𝑥4

𝑥5

𝑥6

…

…

…

…

…

…

…

…

(a) Inference Phase

GAN

Loss

Deg. Aware

Reweighting

Optimal Transport

Block Diagonal SM

Similarity Matrix(SM)

Deg. Aware

Sampling

Disentangled Map

Disentangled Categories

Generator

Shared

Encoder

Deg. Aware

Contrastive

Loss

Nighttime Domain Translation. Domain translation techniques have been applied to address adverse 75 nighttime conditions. An early contribution is made by Anoosheh et al. , which demonstrates the 76 effectiveness of cycle-consistent learning in Night2Day[1]. Following this, many works incorporate 77 different modules into cycle-consistent learning to enhance structural modeling capabilities. Zheng et 78 al. incorporate a fork-shaped encoder to enhance visual perceptual quality[33]. AUGAN employs 79 uncertainty estimation to mine useful features in nighttime images[18]. Fan et al. explore inter80 frequency relation knowledge to streamline the Night2Day process[5]. Xia et al. utilize nearby GPS 81 locations to form paired night and daytime images, providing weak supervision[26]. Some other 82 studies incorporate human annotations to impose structural constraints, overlooking the practical 83 difficulty of acquiring such annotations at nighttime with multiple degradations [11][16] [22]. To 84 address the concerns of the aforementioned methods, the proposed N2D3 explores patch-wise 85 contrastive learning with physical guidance, so as to achieve degradation-aware Night2Day. N2D3 is 86 free of human annotations and offers comprehensive structural modeling to provide faithful translation 87 results. 88

89

3 Methods

Given nighttime image I ∈ N and daytime image I ∈ D , the goal of Night2Day is to translate 90

N D images from nighttime to daytime while preserving content semantic consistency. This involves the 91 construction of a mapping function F with parameters θ , which can be formulated as F θ : I N → I D . 92 Our method N2D3 is illustrated in Figure 2. To train a generator for Night2Day, we employ GANs as 93 the overall learning framework to bridge the domain gap between nighttime and daytime. Our core 94 design, consisting of the degradation disentanglement module and the degradation-aware contrastive 95 learning module, aims to preserve the structure from the source images and suppress artifacts. 96

In this section, we first introduce physical priors in the nighttime environment, and then describe 97

the degradation disentanglement module and the degradation-aware contrastive learning module, 98

respectively. 99

100

101

102

103

104

105

106

3.1 Physical Priors for Nighttime Environment

The illumination degradations at night are primarily categorized as darkness, well-lit regions, highlight regions, and light effects. As shown in Figure 3, well-lit represents the diffused reflectance under normal light, while the light effects denote phenomena such as flare, glow, and specular reflections. Intuitively, these regions can be disentangled through the analysis of illumination distribution. Among these degradation types, darkness and high-light are directly correlated with illuminance and can be effectively disentangled through illumination estimation.

As a common practice, we estimate the illuminance map L by utilizing the maximum RGB channel 107 of image I N as L = max c ∈ R,G,B I c N . Then k-nearest neighbors [4] is employed to acquire three 108 clusters representing darkness, well-lit, and high-light regions. These clusters are aggregated as 109 masks M d , M n , M h . However, the challenge arises with light effects that are mainly related to 110

132

the illumination. Light effects regions tend to intertwine with well-lit regions when using only the 111 illumination map, as they often share similar illumination densities. To disentangle light effects from 112 well-lit regions, we need to introduce additional physical priors. 113

114

115

116

117

118

119

120

121

To extract the physical priors for disentangling light effects, we develop a photometric model derived from Kubelka-Munk theory [17]. This model characterizes the spectrum of light E reflected from an object as follows:

here x represents the horizontal component for analysis, while the analysis of the vertical component y is the same as the horizontal component. λ corresponds to the wavelength of light. e ( λ, x ) signifies the spectrum, representing the illumination density and color. ρ f stands for the Fresnel reflectance coefficient. R ∞ is the material reflectivity function, formulated as follows at a specific location x = x 0 :

where k ( λ ) and s ( λ ) denote the absorption and scattering coefficients, respectively. This formulation 122 implies that for any local pixels, the material reflectivity is determined if the material is given. 123 Assuming C is the material distribution function, which describes the material type varying across 124 locations, the material reflectivity R ∞ can be formulated as: 125

Since the mixture of light effects and well-lit regions has been obtained previously, the core of 126 disentangling light effects from well-lit regions lies in separating the illumination e ( λ, x ) and re127 flectance components R ( λ ) C ( x ) . Note that the Fresnel reflectance coefficient ρ f ( x ) approaches 0 in 128 reflectance-dominating well-lit regions, while ρ f ( x ) approaches 1 in illumination-dominating light 129 effects regions. According to Equation (1), the photometric model for the mixture of light effects and 130 well-lit regions is formulated as: 131

where Ω denotes the reflectance-dominating well-lit regions.

Subsequently, we observe that the following color invariant response to the regions with high color 133 saturation, which is suitable to extract the illumination: 134

This invariant has the following characteristics: 135

Equation (5) to Equation (6) demonstrate that the invariant N λ m x n captures the features only related 136 to illumination e ( λ, x ) . Consequently, we assert that N λ m x n functions as a light effects detector 137 because light effects are mainly related to the illumination. It allows us to design the illumination 138 disentanglement module based on this physical prior. 139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

3.2 Degradation Disentanglement Module

In this subsection, we will elucidate how to incorporate the invariant for extracting light effects into the disentanglement in computation. As common practice, the following second and third-order components, both horizontally and vertically, are taken into account in the practical calculation of the final invariant, which is denoted as N :

here N λx and N λλx can be computed through E ( λ, x ) by simplifying Equation (5). The calculation of N λy and N λλy are the same. Specifically,

where E x and E λ denote the partial derivatives of x and λ .

To compute each component in the invariant N , we develop a computation scheme starting with the estimation of E and its partial derivatives E λ and E λλ using the Gaussian color model:

where x, y are pixel locations of the image. Then, the spatial derivatives E x and E y are calculated by convolving E with Gaussian derivative kernel g and standard deviation σ :

where t denotes the index of the horizontal component x and Z represents set of integers. The spatial derivatives for E λx and E λλx are obtained by applying Equation (10) to E λ and E λλ . Then invariant N can be obtained following Equation (8) and Equation (7).

To extract the light effects, ReLU and normalization functions are first applied to filter out minor disturbances. Then, by filtering invariant N with the well-lit mask M n , we obtain the light effects from the well-lit regions. The operations above can be formulated as:

while the well-lit mask are refined: M n ← M n -M le .

With the initial disentanglement in Section 3.1, we obtain the final disentanglement: M d , M n , M h and M le . All the masks are stacked to obtain the disentanglement map. Through the employment of the aforementioned techniques and processes, we successfully achieve the disentanglement of various degradation regions.

3.3 Degradation-Aware Contrastive Learning

For unpaired image translation, contrastive learning has validated its effectiveness for the preservation of content. It targets to maximize the mutual information between patches in the same spatial location from the generated image and the source image as below:

v is the anchor that denotes the patch from the generated image. The positive example v + corresponds 167 to the source image patch with the same location as the anchor v . The negative examples v -represent 168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

patches with locations distinct from that of the anchor v . Q denotes the total number of negative examples. In our work, the key insight of degradation-aware contrastive learning lies in two folds: (1) How to sample the anchor, positive, and negative examples. (2) How to manage the focus on different negative examples.

Degradation-Aware Sampling. In this paper, N2D3 selects the anchor, positive, and negative patches under the guidance of the disentanglement results. Initially, based on the disentanglement mask obtained in the Section 3.2, we compute the patch count for different degradation types, denoting as K s , s ∈ [1 , 4] . Then, within each degradation region, the anchors v are randomly selected from the patches of generated daytime images I N→D . The positive examples v + are sampled from the same locations with the anchors in the source nighttime images I N , and the negative examples v -are randomly selected from other locations of I N . For each anchor, there is one corresponding positive example and K s negative examples. Subsequently, the sample set with the same degradation type will be assigned weights and the contrastive loss will be computed in the following steps.

Degradation-Aware Reweighting. Despite the careful selection of anchor, positive, and negative examples, the importance of anchor-negative pairs still differs within the same degradation. A known principle of designing contrastive learning is that the hard anchor-negative pairs ( i.e ., the pairs with high similarity) should assign higher attention. Thus, weighted contrastive learning can be formulated as:

w n denotes the weight of the n -th anchor-negative pairs.

The contrastive objective is depicted in the Similarity Matrix in Figure 2. The patches in different regions are obviously easy examples. We suppress their weights to 0, which transforms the similarity matrix into a blocked diagonal matrix with diag ( A 1 , . . . , A 4 ) . Within each degradation matrix A s , s ∈ [1 , 4] , a soft reweighting strategy is implemented. Specifically, for each anchor-negative pair, we apply optimal transport to yield an optimal transport plan, serving as a reweighting matrix associated with the disentangled results. It can adaptively optimize and avoid manual design. The reweight matrix for each degradation type is formulated as:

̸

The aforementioned operations transform the contrastive objective to the Block Diagonal Similarity 195 Matrix depicted in Figure 2. As a common practice, our degradation-aware contrastive loss is applied 196 to the S layers of the CNN feature extractor, formulated as: 197

3.4 Other Regularizations 198

As a common practice, GANs are employed to bridge the domain gap between daytime and nighttime. 199 The adversarial loss is formulated as: 200

where D denotes the discriminator network. The final loss function is formatted as : 201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

4 Experiments

4.1 Experimental Settings

Datasets. Experiments are conducted on the two public datasets BDD100K [29] and Alderley [19]. Alderley dataset consists of images captured along the same route twice: once on a sunny day and another time during a stormy rainy night. The nighttime images in this dataset are often blurry due to the rainy conditions, which makes Night2Day challenging. BDD100K dataset is a large-scale highresolution autonomous driving dataset. It comprises 100,000 video clips under various conditions. For each video, a keyframe is selected and meticulously annotated with details. We reorganized this dataset based on its annotations, resulting in 27,971 night images for training and 3,929 night images for evaluation.

Evaluation Metric. Following common practice, we utilize the Fréchet Inception Distance (FID) scores [7] to assess whether the generated images align with the target distribution. This assessment helps determine if a model effectively transforms images from the night domain to the day domain. Additionally, we seek to understand the extent to which the generated daytime images maintain structural consistency compared to the original inputs. To measure this, we employ SIFT scores, mIoU scores and LPIPS distance [32].

DownStream Vision Task. Two downstream tasks are conducted. In the Alderley dataset, GPS annotations indicate the locations of two images, one in the nighttime and the other in the daytime, as the same. We calculate the number of SIFT-detected key points between the generated daytime images and their corresponding daytime images to measure if the two images represent the same location. The BDD100K dataset includes 329 night images with semantic annotations. We employ Deeplabv3 pretrained on the Cityscapes dataset as the semantic segmentation model [2], then perform inference on our generated daytime images without any additional training and compute the mIoU (mean Intersection over Union).

4.2 Results on Alderley

We first apply Night2Day on the Alderley dataset, a challenging collection of nighttime images captured on rainy nights. In Figure 4, we present a visual comparison of the results. CycleGAN [34] and CUT [20] manage to preserve the general structural information of the entire image but often lose many fine details. ToDayGAN [1], ForkGAN [33], Decent [27], and Santa [28] tend to miss important elements such as cars in their results.

In Table 1, thirteen translation methods and three enhancement methods are compared, considering both visual effects and keypoint matching metrics. Our method showcases an improvement of 10.3

234

235

236

237

238

239

240

Real Night

CycleGAN

ForkGAN

ToDayGAN

N2D3 (ours)

Decent

Santa

CUT

Real Night

CUT

ForkGAN

N2D3 (ours)

CycleGAN

Decent

ToDayGAN

Santa

in FID scores and 4.52 in SIFT scores compared to the previous state-of-the-art. This suggests that N2D3 successfully achieves photorealistic daytime image generation, underscoring its potential for robotic localization applications. The qualitative comparison results are demonstrated in Figure 4. In conclusion, N2D3 achieves top scores in both FID and LPIPS metrics, demonstrating its superiority in the Night2Day task. N2D3 excels in generating photorealistic daytime images while effectively preserving structures, even in challenging scenarios such as rainy nights in the Alderley.

4.3 Results on BDD100K

We conducted experiments on a larger-scale dataset, BDD100K, focusing on more general night 241 scenes. The qualitative results can be found in Figure 5. CycleGAN, ToDayGAN, and CUT succeed 242 in preserving the structure in well-lit regions. ForkGAN, Santa, and Decent demonstrate poor 243 performance in such challenging scenes. Regretfully, none of them excel in handling light effects and 244 exhibit weak performance in maintaining global structures. With a customized design specifically 245 addressing light effects, our method successfully preserves the structure in all regions. 246

247

248

249

250

The quantitative results are presented in Table 1. As the scale of the dataset increases, all the compared methods show an improvement in their performance. Notably, N2D3 demonstrates the best performance with a significant improvement of 5.4 in FID scores , showcasing its ability to handle a broader range of nighttime scenes and establishing itself as the most advanced method in this domain.

251

252

253

254

255

256

257

258

259

260

261

We also investigate the potential of Night2Day in enhancing downstream vision tasks in nighttime environments using the BDD100K dataset. The quantitative results are summarized in Table 1. The enhancement methods demonstrate a slight improvement in segmentation results, while some image-to-image translation methods have a negative impact on performance. N2D3 exhibits the best performance in enhancing nighttime semantic segmentation with a remarkable improvement of 5.95 in mIoU compared to inferring the segmentation model directly on nighttime images.

In conclusion, N2D3 achieves top scores in both FID and LPIPS metrics, establishing itself as the most advanced method for the Night2Day task. It excels in generating photorealistic daytime images while preserving local and global structures. Moreover, the substantial improvement in nighttime semantic segmentation highlights its benefits for downstream tasks and its potential for wide-ranging applications.

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

50.8

45.4

31.5

35.4

36.7

65.0

59.4

50.9

58.2

53.9

20

30

40

50

60

70

64

128

256

512

1024

FID

num samping

BDD100K

0.582

0.579

0.466

0.508

0.491

0.681

0.673

0.650

0.685

0.739

0.4

0.5

0.6

0.7

0.8

64

128

256

512

1024

LPIPS

num samping

BDD100K

(a) Ablation in FID

(b) Ablation in LPIPS

4.4 Ablation Study

Ablation on the main component of degradation-aware contrastive learning. The core design of the degradation-aware contrastive learning module relies on two main components: (a) degradationaware sampling, and (b) degradation-aware reweighting. As shown in Table 2, when degradationaware sampling is exclusively activated, there is a noticeable decrease in FID on both datasets compared to the baseline (no components activated). Notably, the combination of degradation-aware sampling and reweighting achieves the lowest FID on both BDD100K and Alderley, indicating the effectiveness of degradation-aware sampling in conjunction with degradation-aware reweighting.

Ablation on the number of patches in the degradation-aware sampling. To explore the impact of the number of sampling patches in our method, we conduct an ablation study on the number of sampling patches with settings of 64, 128, 256, 512, and 1024 for degradation-aware sampling. The FID and LPIPS scores are evaluated, as shown in Figure 6. The optimal performance is achieved with 256 patches, and increasing the number of sampling patches beyond this point leads to a degradation in performance.

Ablation on the type of the invariant in disentanglement. To explore different invariants for obtaining degradation-disentangled prototypes, we conduct an ablation study on the type of invariant. As shown in Table 2, when L is enabled, the FID decreases from 55.5 to 49.1 on BDD100K and from 64.7 to 62.9 on Alderley. This suggests that incorporating illuminance maps helps in reducing the perceptual gap between generated and source nighttime images. When N is activated, there is a consistent improvement in FID on both datasets, indicating that considering physical priors invariant contributes to more realistic image generation. The combination of both illuminance map and physical prior invariant results in the lowest FID on both datasets, showcasing the complementary nature of these degradation types in improving contrastive learning.

5 Conclusion

This paper introduces a novel solution for the Night2Day image translation task, focusing on translating nighttime images to their corresponding daytime counterparts while preserving semantic consistency. To achieve this objective, the proposed method begins by disentangling the degradation presented in nighttime images, which is the key insight of our method. To achieve this, we contribute a degradation disentanglement module and a degradation-aware contrastive learning module. Our method outperforms the existing state-of-the-art, which shows the effectiveness of N2D3 and the superiority of the insight to disentangle the degradation.

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

References

Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, and Luc Van Gool. Night-to-day image translation for retrieval-based localization. In 2019 International Conference on Robotics and Automation (ICRA) , pages 5958-5964. IEEE, 2019.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition , pages 8789-8797, 2018.

T. Cover and P. Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory , 13(1):21-27, 1967.

Zhentao Fan, Xianhao Wu, Xiang Chen, and Yufeng Li. Learning to see in nighttime driving scenes with inter-frequency priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4217-4224, 2023.

Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proc. IEEE Conference on Computer Vision and Pattern Recognition , pages 1780-1789, 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30, 2017.

Jinhui Hou, Zhiyu Zhu, Junhui Hou, Hui Liu, Huanqiang Zeng, and Hui Yuan. Global structureaware diffusion process for low-light image enhancement. Advances in Neural Information Processing Systems , 36, 2024.

Xueqi Hu, Xinyue Zhou, Qiusheng Huang, Zhengyi Shi, Li Sun, and Qingli Li. Qs-attn: Queryselected attention for contrastive learning in i2i translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18291-18300, 2022.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei Efros. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition , pages 1125-1134, 2017.

Somi Jeong, Youngjung Kim, Eungbean Lee, and Kwanghoon Sohn. Memory-guided unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6558-6567, 2021.

Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, and Shuaicheng Liu. Low-light image enhancement with wavelet-based diffusion models. ACM Transactions on Graphics (TOG) , 42(6):1-14, 2023.

Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing , 30:2340-2349, 2021.

Mikhail Kennerley, Jian-Gang Wang, Bharadwaj Veeravalli, and Robby T Tan. 2pcnet: Twophase consistency training for day-to-night unsupervised domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11484-11493, 2023.

Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint arXiv:1907.10830 , 2019.

Soohyun Kim, Jongbeom Baek, Jihye Park, Gyeongnyeon Kim, and Seungryong Kim. In339 staformer: Instance-aware image-to-image translation with transformer. In Proceedings of 340 the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18321-18331, 341 2022. 342

343

344

Paul Kubelka. Ein beitrag zur optik der farbanstriche (contribution to the optic of paint). Zeitschrift fur technische Physik , 12:593-601, 1931.

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

Jeong-gi Kwak, Youngsaeng Jin, Yuanming Li, Dongsik Yoon, Donghyeon Kim, and Hanseok Ko. Adverse weather image translation with asymmetric and uncertainty-aware gan. arXiv preprint arXiv:2112.04283 , 2021.

Michael J. Milford and Gordon. F. Wyeth. Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights. In 2012 IEEE International Conference on Robotics and Automation , pages 1643-1649, 2012.

Taesung Park, Alexei Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision , pages 319-345, 2020.

Aashish Sharma and Robby T Tan. Nighttime visibility enhancement by increasing the dynamic range and suppression of light effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11977-11986, 2021.

Seokbeom Song, Suhyeon Lee, Hongje Seong, Kyoungwon Min, and Euntai Kim. Shunit: Style harmonization for unpaired image-to-image translation. Proceedings of the AAAI Conference on Artificial Intelligence , 37(2):2292-2302, Jun. 2023.

Tao Wang, Kaihao Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu. Ultrahigh-definition low-light image enhancement: A benchmark and transformer-based method. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2654-2662, 2023.

Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1402014029, 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Youya Xia, Josephine Monica, Wei-Lun Chao, Bharath Hariharan, Kilian Q Weinberger, and Mark Campbell. Image-to-image translation for autonomous driving from coarsely-aligned image pairs. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7756-7762. IEEE, 2023.

Shaoan Xie, Qirong Ho, and Kun Zhang. Unsupervised image-to-image translation with density changing regularization. In Advances in Neural Information Processing Systems , 2022.

Shaoan Xie, Yanwu Xu, Mingming Gong, and Kun Zhang. Unpaired image-to-image translation with shortest path regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10177-10187, June 2023.

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2636-2645, 2020.

Zhenjie Yu, Shuang Li, Yirui Shen, Chi Harold Liu, and Shuigen Wang. On the difficulty of unpaired infrared-to-visible video translation: Fine-grained content-rich patches transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1631-1640, June 2023.

Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, and Shijian Lu. Modulated contrast 388 for versatile image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision 389 and Pattern Recognition (CVPR) , pages 18280-18290, June 2022. 390

391

392

393

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

394

395

Ziqiang Zheng, Yang Wu, Xinran Han, and Jianbo Shi. Forkgan: Seeing into the rainy night. In European conference on computer vision , pages 155-170. Springer, 2020.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image transla396 tion using cycle-consistent adversarial networks. In Proc. IEEE International Conference on 397 Computer Vision , pages 2223-2232, 2017. 398

399

400

401

402

403

404

405

406

407

408

A Overview

This supplementary material is organized as follows. Appendix B provides additional details about the proof that the invariant N λ m x n is exclusively related to the illumination. Appendix C outlines the limitations and failure case of N2D3. Appendix D illustrates the implementation details, including N2D3 and other methods used in the experiments. Appendix E presents additional visualization results.

B More Proof Details

We provide a detailed proof process to demonstrate how the invariant N λ m x n is exclusively related to the illumination and can function as the light effect detector. First, consider the following equations, corresponding to Equation (5) in the main paper:

by applying the additivity of linear differential operators, the first term represents the invariants only 409 related to the illumination. The second term can be simplified by applying the chain rule as follows: 410

Finally, we conclude that the invariant N λ m x n is exclusively related to the illumination and can be 411 formulated as follows: 412

C Limitations and Failure Case 413

Despite the superior performance of N2D3 in Night2Day, it still exhibits certain limitations. On the 414 one hand, this work focuses solely on addressing light degradation, while nighttime environments 415 encompass various other types of degradation, including blur caused by rain, motion, and other 416

w/o N

w/o reweighting

N2D3 (ours)

Real Night

factors. Our method currently struggles to handle these situations effectively. On the other hand, the 417 limitations of visible imaging in night vision arise from the scarcity of photos captured in low-light 418 conditions, as illustrated by the failure cases presented inFigure 7. Future advancements in night 419 vision will likely incorporate additional modalities, such as infrared images, radar, and other sensor 420 data, to overcome these challenges and improve performance. 421

422

423

424

425

426

427

428

429

D Implementation Details

Training Details. We adopt the resnet 9blocks , a ResNetbased model with nine residual blocks, as the backbone for generator G . Additionally, we utilize the patch-wise discriminator D following PatchGAN[10]. To conduct degradation-aware contrastive learning on multiple layers, we extract features from 5 layers of the generator G encoder, as done in [20]. These layers include RGB pixels, the first and second downsampling convolution, and the first and fifth residual block. For the features of each layer, we apply a 2-layer MLP to acquire final 256-dimensional features. These features are then utilized in our degradation-aware contrastive learning.

All the comparison methods are reproduced using their released source code with default settings. 430

Training procedures are consistent across all methods. All models are trained using the Adaptive 431 Moment Estimation optimizer with an initial learning rate of 10 -4 , a momentum of 0.9, and weight 432 decay of 10 -4 . For the BDD100K dataset, training consists of 10 epochs with the initial learning 433 rate, followed by another 10 epochs with a decreased learning rate using the polynomial annealing 434 procedure with a power of 0.9. On the Alderley dataset, given the limited training data compared 435

to BDD100K, we extend the training to 20 epochs with the initial learning rate and an additional 436

CUT

N2D3 (ours)

Decent

Santa

Real Night

CycleGAN

ForkGAN

ToDayGAN

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

20 epochs with the decayed learning rate. All the experiments are run on a single A100 GPU with 80GB of memory. Training our method with a smaller patch size and batch size on a device with less memory is feasible.

Evaluation Details. In the evaluation, we compute the Fréchet Inception Distance (FID) [7], Structural Similarity Index (SSIM) [25], and Learned Perceptual Image Patch Similarity (LPIPS) [32] scores on 256 × 512 images. Partial FID scores are provided by ForkGAN [33], and all SSIM and LPIPS scores are reproduced by us.

Semantic segmentation evaluation are conducted as follows. First, we use Deeplabv3 pretrained on the Cityscapes dataset as the semantic segmentation model [2]. The model is provided by https://github.com/open-mmlab/mmsegmentation with an R-18-D8 backbone and trained at a resolution of 512 × 1024 . Second, we perform 512 × 1024 Night2Day translation to obtain the generation results. Finally, we infer the semantic segmentation on the generated daytime images.

E More Visualization Results

More Ablation Visualization Results. We provide ablation visualization results on both Alderley and BDD100K in Figure 9. The complete method is presented along with ablation studies on the invariant N and without degradation-aware reweighting. All the modules contribute to improving the ability to maintain semantic consistency.

More Disentanglement Results. We provide additional disentanglement results in Figure 8. Our disentanglement methods offer a comprehensive representation of different illumination degradation types in various nighttime scenes.

More Qualitative Comparison. We present more qualitative comparisons in Figure 10 and Figure 11 alongside other methods.Our method demonstrates visually pleasing results under various nighttime conditions.

Real Night

CycleGAN

ToDayGAN

ForkGAN

CUT

N2D3 (ours)

Decent

Santa

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim our main contribution as N2D3, which achieves SOTA performance by bridging the domain gap between nighttime and daytime in a degradation-aware manner.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss our limitation in degradations beyond light and low-light image scarcity in the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and complete proofs in both Section 3.1 and Appendix B .

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the information needed to reproduce the main experimental results is included in the Section 3 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code will be released latter.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training details and dataset information are provided in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive. We report our results using a fixed random seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the compute resources in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impacts are discussed in the manuscript and appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our model does not have such risks, and all the datasets used in the experiments are open-source benchmarks in this field.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code and data are properly credited, and the license and terms of use are explicitly mentioned and properly documented.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code introduced in the paper is well-documented, and the documentation is provided alongside it.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

770

771

772

773

774

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach, namely N2D3, for night-to-day image translation. Specifically, the proposed pipeline involves two stages: illumination degradation disentanglement and degradation-aware contrastive learning. The first stage decomposes an image into darkness, well-lit areas, light effects, and highlight regions. The second stage applies contrastive learning to these four types of nighttime degradations. Extensive experiments conducted on the BDD100K and Alderley datasets demonstrate that N2D3 outperforms existing methods.\n\nSTRENGTHS:\n- The paper addresses the critical problem of night-to-day image translation in computer vision.\n- The authors provide comprehensive experimental validation of the proposed method.\n\nWEAKNESSES:\nThe reviewer has raised this paper for an ethics review due to a significant omission of a key citation. In Section 3.1, the authors introduce a color invariant term for light effect detection. However, this term was originally derived by Geusebroek et al. in their paper *Color Invariance* [1]. The authors devote an entire page to deriving the invariant term without appropriately citing the original work, which violates academic integrity. The authors should explain why this citation is missing, as it does not seem to be an unintentional oversight. This intended missing reference also made Eq. (1)-(5) lack logical coherence and hard to follow.\n\n[1] Color Invariance. J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, H. Geerts. IEEE TPAMI, 2001.\n\n**Note that although the reviewer has raised the ethics review flag, the reviewer’s rating does not take this into account.** \n\nIn addition to the missing citation, the reviewer has concerns about the technical soundness of the paper. Specifically, why are four types of degradation considered? Since the disentanglement of well-lit and light effect regions is the paper’s main contribution, ablation studies using only three types of degradation (darkness, well-lit, and highlight) should be provided.\n\nBesides, the paper’s citation style is inconsistent. For instance, citations for the same conference sometimes include the abbreviation and publisher while others do not (e.g., [1], [19], and [26]). Additionally, some citations include the month of the conference while others do not (e.g., [24], [28]), and some contain volume information while others do not (e.g., [22], [23]). Ensuring consistent citation formatting would enhance the paper’s overall presentation quality.\n\nQUESTIONS:\nBeyond concerns regarding missing citations and technical soundness, the reviewer has the following questions:\n\n- Given the prevalence of large models, why not approach the night-to-day translation task using diffusion models? For instance, a paper [2] addresses day-to-night/fog/snow translation on BDD100K; could this framework be adapted to handle night-to-day tasks? The reviewer does not require quantitative or qualitative results but would appreciate the authors’ insights on this matter.\n\n- What is the rationale behind categorizing degradation into four types? The paper mentions that light effects involve phenomena like flare, glow, and specular reflections. Could degradation be categorized into more types? While a physics-based approach may not apply here, could a segmentation model be trained for this purpose (the reviewer acknowledges the lack of labeled data for training such a model, but there might exist other possible solutions)?\n\n[2] Greenberg et al. S2ST: Image-to-Image Translation in the Seed Space of Latent Diffusion. In CVPR, 2023.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of the work, and this paper does not have any negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework N2D3 for solving night to day image translation problem. Their framework consists of a physics-based disentanglement module and a contrastive learning module for preserving semantic consistency. Their method shows improved performance in terms of FID and downstream task performance on BDD100K and Alderley dataset.\n\nSTRENGTHS:\n- Using the Kubelka-Munk theory for different degradation types and applying a patch-based image translation is a novel method. \n- The figures are well-made. For instance, the visualization in Fig. 1 and Fig. 2 are intuitive and helpful for understanding the whole architecture.\n- Quantitative evaluation results are convincing, showing the effectiveness of the proposed framework in terms of various metrics.\n\nWEAKNESSES:\n- Clarity of the method sections can be improved. For instance, including more rigorous definitions or visualizations of what well-lit and different light effects mean and provide a motivation why it is helpful to disentangle those illumination causes separately. \n- The authors can also add proper citations to previous work when they mention “by common practice”, for instance in line 107, line 142 and line 196.\n\nQUESTIONS:\n- Why is illuminance computed by the maximum of RGB channel (line 107)? I think this is indeed not a common practice.\n- Could you provide some ablation visualizations with and without applying the disentanglement method in 3.2? For instance, the 3 clusters with initial disentanglement in 3.1 and the four clusters afterwards. \n- Since the evaluation metrics are performed on segmentation tasks, can you provide some visual examples of the segmented regions of those previous methods and your method?\n\nLIMITATIONS:\nAs the authors already mentioned in the appendix, the current physics-aware degradation disentanglement module is designed mostly for illumination related effects and does not handle other types of degradation such as raindrops. I wonder how the authors think the framework could benefit or inspire other types of adverse weather image restoration tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive solution for Night2Day image translation by leveraging physical priors, photometric modeling, and contrastive learning, leading to state-of-the-art performance in visual quality and downstream vision tasks.\n\nSTRENGTHS:\nThe authors develop a photometric model based on Kubelka-Munk theory to extract physical priors from nighttime images. This model helps to disentangle different types of illumination degradations by analyzing the illumination distribution.\nOverall, the paper presents a novel approach to handling nighttime image translation by considering the unique challenges posed by varying degradations and employing both physical modeling and advanced learning strategies to address these challenges effectively.\n\nWEAKNESSES:\n1. The writing is difficult to understand. The explanations and derivations for Eqs (1) to (5) lack logical coherence and necessary references, making them hard to follow.   The derivations for Eqs (7) to (9) also lack supporting references, casting doubt on their validity. \n2. The motivation for DAR is unclear. Please explain the motivation behind it.\n3. There is no baseline network, making it difficult to determine the performance gain for the specific module. \n4. The ablation experiments lack in-depth analysis. For instance, there are no ablation experiments to verify the impact of introducing four regions for disentanglement versus three regions (e.g., excluding the light effects region).\n5.There is a need to compare with more recent methods for unpaired image-to-image translation, such as COCO-FUNIT, StegoGAN, GP-UNIT, etc. Please check the reference [5], as it does not seem to be published in CVPR.\n\nQUESTIONS:\n1. The explanations for Eqs (1) to (5) lack logical coherence and necessary references, making them hard to follow. What is the \"color invariant response\" in Eq (5), and why is it used to extract illuminance? The derivations and calculations for Eqs (7) to (9) also lack supporting references, casting doubt on their validity. Why is there a need to refine Mn?\n2. How is the reweighting matrix obtained through optimal transport? Are there any references that have implemented this approach? \n3. Contrastive learning has been widely used in image translation. The paper lacks ablation experiments to support the performance gains from the reweighting operation.\n\nLIMITATIONS:\nThe motivation behind some methodological choices, is not clearly explained.\nThe ablation experiments are insufficient and lack depth. \nThe paper lacks a discussion on the computational complexity and resource consumption of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to address the night-to-day translation problem in which its learning basically can be briefly described by two steps: 1) illumination distribution as well as the physic priors built upon the Kubelka-Munk photometric model are firstly adopted to separate/disentangle the image regions into four degradation categories, i.e. darkness, well-lit, light effects, and high-light, in which such illumination degradation disentanglement is the main contribution of the proposed method; 2) the degradation-aware contrastive learning module is applied to maximize the mutual information between patches in the same spatial location from the generated image and the source image, where the anchor and its corresponding negative patches should be from the same degradation category (i.e. degradation-aware sampling) and the weights for each negative patch are determined by similar matrix obtained from the optimal transport computation (i.e. degradation-aware reweighting). Moreover, the GAN-based objective function is employed to bridge the domain gap between (generated) daytime and nighttime images. The translated images (from nighttime to daytime) are shown to have better quantitative and qualitative performance (in terms of FID) for aligning with the real nighttime image distribution.\n\nSTRENGTHS:\n+ In addition to provide better translation performance (both quantitative and qualitative), the translated images produced by the proposed method are shown to have better structural similarity with respect to the corresponding daytime images (evaluated in Alderley dataset)  in comparison to the baselines. Moreover, the typical semantic segmentation model (pretrained on typical daytime dataset, i.e. Cityscapes) applied on the translated images (produced by the proposed method) leads to better segmentation results in comparison to being applied on the images generated by the baselines (i.e. indirect evidence showing that the images generated by the proposed better follows the daytime image distribution which the semantic segmentation model is trained on). \n+ The ablation study does demonstrate the contribution of illumination degradation disentanglement for separating the image patches into four different degradation categories.\n\nWEAKNESSES:\n- Although experimentally shown to be effective, the mechanism and the basic ideas behind leveraging the illuminance distribution as well as the physic priors for realizing disentanglement of four degradation categories (i.e. darkness, well-lit, light effects, and high-light) are not well explained, in which the physical meanings for Eq.1 to Eq.11 are hard to understand and follow. Basically, as such illumination degradation disentanglement is the main contribution of the proposed method, the description should be more self-contained and explanatory. \n- As the illumination degradation disentanglement plays a key role in the proposed method, it would be great to have the robustness analysis on such disentanglement if possible (i.e. how accurate is the disentanglement, is there any related dataset we could apply such analysis?) and how would the translation model learning be affected once there are erroneous disentanglement?\n\nQUESTIONS:\nThough the contribution and the novelty of the proposed illumination degradation disentanglement is clearly recognizable from the quantitative and qualitative results, better description upon its mechanism and basic ideas would be much appreciated. Moreover, the further analysis upon its accuracy/robustness would be also better to have.\n\nLIMITATIONS:\nno potential negative societal impact is found.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}