Package: n2dit
Night-to-Day Image-translation SystemIntroductionAs mentioned in the following paper, brightness is a very important and sensitive factor in image processing.Night surveillance is a challenging task because of low brightness, low contrast, low Signal to Noise Ratio (SNR) and low appearance information. [1]Inhomogeneous illumination induces large contrast variations in images. Detection is diﬃcult in low-contrast regions and may result in loss of color information and in confusing foreground and background regions. [2]In addition, in the object detection field, brightness is a factor that greatly affects the performance like mAP, and the accuracy of object detection decreases at night.
With this in mind, the project proposes a Night-to-Day Image-translation System(hereinafter referred to as "n2dit") that transforms night images to day images.
The system supports improving the performance of systems that require object recognition, such as ADAS, black boxes, and computer vision.CycleGAN, an unsupervised learning-based adversary generation model, was used and Alpha blending was introduced to improve the detection rate of the transformed image.
Through this, we proved that, there was a improvement in mAP(mean Average Precision) compared to the existing CLAHE(Contrast Limited Adaptive Histogram Equalization) algorithm.ImplementationCycleGANThe project started with CycleGAN[3]. CycleGAN is created by adding cycle-consistency loss to GAN. Cycle-consistency loss helps CycleGAN to only transform the original image to a Fake to the level that can be recovered to Real. In addition, CycleGAN has the great advantage of using a training set that does not require pairing. Thus, allowing CycleGAN to learn domain-to-domain image-translation.While following the original implementation, we found out that if the depth of the discriminator is less than a certain amount, generator no longer learns after a certain period of time. Therefore, we tried to find the appropriate number of convolutional layers in the discriminator to improve performance through equivalent competitive learning of the generator and discriminator.Alpha Blending[5]Alpha blending is also used in 2D computer graphics to put rasterized foreground elements over a background.
We experimentally found that the method of displaying a mixture by overlaying the output image to the input image showed higher object detection rates than the conversion method through CycleGAN.Failed ExperimentsPerceptual LossIn Ledig et al.[4], the authors propose a perceptual loss function which consists of an adversarial loss and a content loss.
We use this in the objective function to improve the resolution of the fake image generated by CycleGAN.Adding CLAHE-Sobel LossBefore we applied the perceptual loss, we experimented with adding the following loss to increase the resolution of the generated images:
For the mapping functionG: X → Y,F: Y → Xand CLAHEC, sobel filterSSo the full objective is:The results during the learning process seemed quite plausible, but the test scores did not improve significantly.Resize ConvolutionWe found that the checkerboard-like pattern that occurs during the generator's upsampling process reduces the object detection rate and causes low resolution.
We found the Bilinear-Resize Convolution[6], which is an alternative to the existing upsampling method, “transposed convolution”.
The method resizes the image using the nearest-neighbor interpolation and then upsamples it through the convolutional layer, which improve the “uneven overlap” such as a checkerboard pattern.Test ProcessPreparing DatasetExtract night and day images from the labels in theBerKeley DeepDrive dataset, redefine the class name in the label as appropriate for the darknet setup, and create the correct label by calculating the width and height from the x1, x2, y1, and y2 coordinate.Model TrainingTrain our model with the training dataset prepared using the various options.Image-translationConvert the night image in the test dataset into a day image with a trained model.Object DetectionCalculate mAP of real and transformed images using Darknet with COCO-Yolo v3 pre-trained model.
There are 80 classes in the COCO dataset, but we limit it to three classes(person, car, bus) for the characteristics of the BDD dataset.Refine ModelRepeat training by adjusting network or parameters according to whether object detection is improved.ResultsResult ImagesThe image translation results of all experiments, including the failed ones, are as follows.Object Detection PerformanceThe following table is the result of object detection of 5,000 night images from a BerKeley DeepDrive Dataset with COCO-Yolo v3 weight.
The number of classes and objects in a dataset is as follows: (Person: 3611, Car: 45694, Bus: 378)
When the generator uses 9 resnet blocks, the discriminator uses 6 convolution layers, and alpha blending is applied, the highest object detection rate is shown in the following table.OriginalCLAHEAlpha Blending(Real:0.8, fake:0.2 ratio)The discriminator has 6 convolution layerswith Alpha BlendingPerson(AP)29.7531.9030.7130.98Car(AP)42.0944.4144.5446.19Bus(AP)25.4426.3728.3428.05mAP32.4334.2334.5335.07mAP(Balanced)41.0743.3643.4144.95Diff-2.302.353.88ConclusionThe Night-to-Day Image-translation System adds the characteristics of the target domain, which the original domain did not have, to the original domain for image transformation. Night images with low brightness are converted to day images with high brightness, which is a major factor in improving the performance of object detection.
In addition, problems such as noise, low resolution, generated from a bad weather environment or a sensor of image processing devices, etc. that degrade the performance of object detection can be solved by learning them into the original domain and converting them to have desired characteristics.
Therefore, the system is considered to be highly utilized in the field of computer vision, and it is possible to apply a target transformation in image processing of various fields as well as night and day transformation.Reference[1]Object detection and tracking for night surveillance based on salient contrast analysis[2]NightOwls: A pedestrians at night dataset[3]Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[4]Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[5]Alpha blending in Wikipedia[6]Deconvolution and Checkerboard ArtifactsSystem RequirementsHardwareNVIDIA GPUs 8G10.00 GB of available disk spaceSoftwareLinuxUbuntu 18.04NVIDIA driver - version for CUDACUDA - 10.0cuDNN - 7.6.2.24-1Python 3.6.8pip - 19.2.3(python 3.6)tensorflow-gpu - 1.14matplotlib - 3.1.1pillow - 6.2.1opencv-python - 3.4.2.17DockerNVIDIA driver - version for CUDADocker Engine - 19.03.4(Option 1) tensorflow/tensorflow:1.14.0-gpu-py3(Digest:e72e66b3dcb9)matplotlib - 3.1.1pillow - 6.2.1opencv-python - 3.4.2.17(Option 2) n2dit Docker ImageSee below.InstallationUse pip for Installing(Optional) To use the GUI, checkout the gui branch with the following command.$ git checkout guiInstall the list of requirements with the following command.$ pip install -r requirements.txtInstall n2dit with the following command from the directory path where the setup.py file is located.$ pip install .(Optional) If you are using Anaconda, follow these steps.Remove opencv-python from requirements.txt.Create and activate the environment you want to use.Install opencv-python with the following command.$ conda install -c menpo opencvUse the following command as in the two steps above.$ conda install --file requirements.txt
$ pip install .DockerBuild n2dit docker image(Optional) To use the GUI, checkout the gui branch with the following command.Build the docker image by running the following command where the Dockerfile is located:$ docker build -t <image-name>:<tag> --target release .Docker HubYou can get the docker image that have been built from the following repository.deeva2019/n2dit:<version>versionCLI version: X.X.XGUI version: gui-X.X.XExcecutionn2ditTrainingSet options via n2dit command options or configuration files.Sample configuration files are located in the config directory.[cyclegan]
dirA = ./datasets/day
dirB = ./datasets/night
results_dir = ./results
load_size = 286
crop_size = 256
channels = 3
shuffle = yes
pool_size = 50
exp_name = test
continue = no
epoch = 0
niter = 100
niter_decay = 100
disp_loss_freq = 10
disp_summary_freq = 100
learning_rate = 0.0002
lambda_A = 10.0
lambda_B = 10.0
lambda_idt = 0.5
upsample = resize_convRun with the following command.$ n2dit cyclegan train -C config/sample_train.cfgTestConfig option values through the n2dit command options or sample configuration file as shown above.Run with the following command.$ n2dit cyclegan test -C config/sample_test.cfgGuide to using dockerCreate a container with the following command to use the host's GPUs.$ docker run --gpus all <image-name>:<tag> <n2dit options>The file used by n2dit corresponds to the following options.dirA: Source domain image directory you want to transformdirB: Target domain image directoryresults_dir: Common results directory used for training and testing(Optional) config_file: Configuration fileSo unless you use n2dit inside the container, use the option v to bind the host volume to the container as follows.$ docker run --gpus all -v /home/deeva/datasets/A/:/work/A\
                        -v /home/deeva/datasets/B/:/work/B\
                        -v /home/deeva/results/:/work/results\
                        n2dit:latest cyclegan train --dirA /work/A --dirB /work/B --results_dir /work/resultsn2dit GUI(Optional) To use the host's GUI inside the container, The script uses a docker option that exposes xhost to render to the display by reading and writing X11 Unix sockets.To use GUI's, use the following command. However, This approach is very vulnerable.$ xhost +local:rootWhen you are finished with the container's GUI, run the following command.$ xhost -local:rootUse easily.