# Daydirex: Translating Nighttime Scenes towards Daytime Driving Experience at Night
## 1 Introduction  
Imagine that you are riding in an autonomous car and passing through the Amalfi Coast (Figure 1), one of the world's most scenic roads in Italy listed in the UNESCO World Heritage. You do not have to worry about driving your car, but could enjoy the breathtaking view through the window of your car. Unfortunately, however, it is night. You can see only the tiny part of such a beautiful scene. What if your car window is a magic window, which transforms dark views outside of the window into bright ones as you can see in the daytime? To realize such a window, one of the important requirements is that the stream of transformed images displayed on the window should be of high quality so that users perceive it as real scenes in the day. This problem can be seen as a problem of image-to-image translation focusing on nighttime to daytime image translation. Image-to-image translation based on Generative Adversarial Networks (GANs) has been widely studied in the deep learning field. Among others, there have been research works for unsupervised image-to-image translation including CycleGAN [1], UNIT [2], MUNIT [3], EGSC-IT [4]. They are advantageous since they do not require paired training images to learn the mapping between two domains of images. Existing image translation models have been applied to diverse applications such as style transfer, season transfer, and photo enhancement, and they have shown impressive translation results. However, night-to-day image translation is still a challenging task. In many cases, too dark areas of the night images inevitably exist, which often causes poor quality of translated images. From our preliminary study with CycleGAN, we find that the CycleGAN model hardly provides quality results for dark areas in nighttime images. The areas rarely have information on geographical features,objects, or structures that actually exist in those areas. It is quite difficult, if not impossible, to translate such dark areas by using input images only. To address the problem, we propose a new approach, i.e., supplementing the missing information in dark areas of input image frames by using existing daytime images corresponding to the input images, which has not been studied yet. This is possible for our target scenario since there are readily available daytime street view images in several cities and rural areas worldwide from street view services. We briefly introduce the problem and our initial idea in [5]. However, it is not trivial to realize the approach. In this paper, we present Daydriex, a processing pipeline to generate enhanced daytime translation focusing on road views to study the feasibility of the proposed approach. Its unique feature is to employ a data supplement processing step that supplements hint information for the dark areas of nighttime images. This processing step retrieves a daytime image from a street view service for an input nighttime image and augments the input image with hint information from the street view image. A resulting image from the data supplement processing step is used as an input of the daytime image generation. Such supplemented input images improve the quality of translation results. For daytime image generation, we adopt an existing image-to-image translation model. To realize the idea, we address several issues which we discuss in detail in Section 4. To evaluate Daydriex compared to an existing translation model, we conduct an experiment that assesses the reality of Daydriex outputs in two aspects. First, we use the Frechet Inception Distance (FID) [6], which is one of key metrics widely used to measure GAN performance. Second, we conduct a user perceptual study with 30 people. We show that the results by Daydriex achieves lower FID scores and higher user perception scores compared to those by CycleGAN only. We also present average computation time taken to run main operations of Daydriex. The rest of this paper is organized as follows. Section 2 discusses related work. Section 3 presents a motivational study.