L2BGAN: An image enhancement model for image quality improvement and image analysis tasks without paired supervision

1 INTRODUCTION . Image enhancement is a prerequisite for many computer vision based image understanding tasks . In particular , it is very crucial to enhance low light or dark images to obtain images which not only have better image aesthetic quality but can also be suitably processed for object detection and face detection tasks . The task by itself is one of the earliest studied domains in computer vision , but the continuous evolution of computational resources and deep learning architectures has generated a paradigm shift towards the latter approach . A thorough literature review shows three main ways in which the problem is approached : histogram equalization ( Ibrahim & Pik Kong , 2007 ; Nakai et al. , 2013 ) , Retinex-based Guo et al . ( 2017 ) ; Wang et al . ( 2016 ) ; Fu et al . ( 2016 ) , and machine learning based ( Lv et al. , 2018 ; Wang et al. , 2019 ; Jiang et al. , 2021 ; Guo et al. , 2020 ) . The primary challenge while dealing with low light image enhancement tasks is that there are many noise sources in the acquisition of poorly lit scenes . These include readout , photon shot , dark current , and fixed pattern noises , in addition to photon response non-uniformities . The noise level increases while treating lightness and contrast of low light images , more so in case of compressed-dynamics images . Applying a denoising filter prior to light enhancement will result in blurring , while the reverse causes noise amplification as seen in Fig . 1 . Hence , dealing with denoising and low-light enhancement problems simultaneously using a learning based approach seems to be the optimal choice . The success of deep learning enhancement models depend on the availability of large scale anno- tated data . For the present problem , there is a requirement of paired images such that the low-light image serves as input while its brighter counterpart serves as the target image . The Adobe 5k dataset ( Vladimir et al. , 2011 ) serves as a benchmark used by many researchers for this task . This dataset provides an original low light image , retouched by five different photographic experts , one of which is used as the target of supervised training . Some researchers have also used a synthetic paired set for the same by intentionally transforming a bright image to generate its dark counterpart . This transformation being global , its suitability for challenging real time images is arguable . Ideally , a training procedure without the requirement of paired supervision is the most appropriate solution . Very few works have been reported on low light image enhancement using unpaired images ( Jiang et al. , 2021 ; Ni et al. , 2020 ) . Moreover , even though image to image style translation without paired supervision is a popular approach Anoosheh et al . ( 2019 ) ; Fu et al . ( 2019 ) ; Zhu et al . ( 2017b ) , not many have considered the mapping of low light images to bright light images as a style transfer problem . In this work , we consider the low light image enhancement task as an image style transfer problem . We train a deep neural architecture using unsupervised image pairs . This promotes the use of available real time dark and bright images without the need of pairwise annotations or synthetic treatment . We resort to GANS for training using two encoders and two decoders . We use three discriminators to learn color , texture , and edges separately . The unsupervised image to image translation problem is challenging as it aims to recover a joint probability given a marginal probability . For this purpose , one of the most common approaches is to add a cycle consistency constraint along with the adversarial losses . This bijective constraint becomes restrictive in cases as the task at hand , where the low light image domain may contain less information compared to its bright counterpart . To handle this situation we use a geometric consistency constraint which ensures that an image from one domain and its geometric transformed version generate the same image in the other domain . Additionally , a contextual loss constraint ensures that the context and semantics of the images are preserved . This paper has the following main contributions : ( i ) it poses the low to bright light enhancement problem as an unpaired image translation problem ; ( ii ) it uses a geometric and an illumination consistency constraint in the images ; ( iii ) it uses a contextual loss to measure semantic similarity ; ( iv ) it uses multiscale color , texture and edge discriminators per domain ; ( v ) it demonstrates the effectiveness of the approach for image understanding tasks . While none of such items are individually novel , exploiting their ensemble effect in this task is indeed a significant novel contribution . Our paper is organized in the following manner . We provide a related background of the task in Sec . 2 . The proposed technique is discussed in details in Sec . 3 . Experimental results and conclusion is provided in Secs . 4 and 5 respectively . 2 RELATED BACKGROUND . Histogram based enhancement aims to improve the image quality by modifying the image histogram . This can result in overstretched contrast which lacks in naturalness . Using this framework different techniques have been developed . BPDHE ( Ibrahim & Pik Kong , 2007 ) uses an HE on a dynamic range expanded version of sub-histograms obtained by the local maxima of the input image histogram . FHSABP ( Wang et al. , 2008 ) solves a convex optimization problem to obtain the flattest target histogram with the brightness preservation constraint . Due to their brightness preservation , ( Ibrahim & Pik Kong , 2007 ) and ( Wang et al. , 2008 ) are not suitable to tackle the low light image enhancement , but prevent overstretching . Morover , ( Ibrahim & Pik Kong , 2007 ) and ( Wang et al. , 2008 ) do not consider relationships between adjacent pixels , whereas methods such as HMF ( Arici et al. , 2009 ) , CEBGA ( Hashemi et al. , 2009 ) and DHECI ( Nakai et al. , 2013 ) do . In HMF ( Arici et al. , 2009 ) the target histogram is obtained via a parametric optimization problem , involving the local variance of pixels . In CEBGA ( Hashemi et al. , 2009 ) a genetic algorithm is adopted to enhance the contrast of the images modifying the histogram , using as a fitness function the number of edges in the enhanced image . DHECI ( Nakai et al. , 2013 ) performs a histogram equalization on the differential intensity histogram and the differential saturation histogram from the HSI color space . More advanced techniques which embed contextual information are CVC ( Celik & Tjahjadi , 2011 ) and LDR ( Lee et al. , 2013 ) , which use 2D histograms . Retinex based methods use the assumption that an image can be pixel-wise decomposed into reflectance and illumination . In ( Li et al. , 2011 ) a classic approach is proposed , where the lightness is first decomposed in reflex lightness and ambient illumination . Reflectance is then extracted from reflex lightness , while the ambient illumination is log-transformed and used for the output image . A widely used framework for the Retinex based enhancement methods is the fusion based framework , as in FEMWII Fu et al . ( 2016 ) and FMSBIE Wang et al . ( 2016 ) . Both estimate the illumination using the pixel-wise maximum in the RGB color space , which is used to obtain the reflectance . Three different modified illuminations are then obtained through different techniques to improve bright- ness and contrast . These illumination maps are then fused with a multiscale approach . Multiscale techniques are also used to extract reflectance ( Liu et al. , 2016 ) Another approach for the illumination estimation is proposed in LIME ( Guo et al. , 2017 ) , where an optimization problem is used to obtain a smooth , structure-preserving illumination map , which is then enhanced through a gamma correction . As the noise is critical in poorly illuminated images in ( Guo et al. , 2017 ) a denoising techinque is also proposed : the output reflectance is the linear combination of the original one and a BM3D-filtered one , using the illumination as the weight to prevent the oversmoothing of bright areas . In ( Li et al. , 2011 ) , Fu et al . ( 2016 ) , Wang et al . ( 2016 ) , ( Liu et al. , 2016 ) , and ( Guo et al. , 2017 ) reflectance is obtained by the estimated illumination . More advanced techniques , such as SRIE Fu et al . ( 2016 ) and JIEPMR Cai et al . ( 2017 ) , jointly decompose reflectance and illumination . While the Retinex theory has proved to be suited for low light image enhancement , this approach suffers from color distortion and hand-crafted illumination manipulation . Learning based methods can overcome these problems , while also providing better denoising , which is crucial in low light images . Data learning methods include a variety of methods from deep learning , such as autoencoders , Deep CNNs and GANs , all of which have proved to be suitable techniques to enhance image quality and brightness . In LLNet ( Lore et al. , 2017 ) , GLADNet ( Wang et al. , 2018 ) and LLED-Net ( Li et al. , 2020 ) , denoising autoencoders are used . Despite the good results , the simplicity of these methods limits their capability compared to more complex learning methods . In ( Ren et al. , 2019 ) multiple autoencoders are used to work separately on the content and on the edges of the image while also using CNN , RNN and a more advanced loss function , including a discriminator and a VGG-16 net . An autoencoder is also used for DeepUPE ( Wang et al. , 2019 ) to learn a image-to-illumination mapping used for a Retinex-fashioned enhancing , while considering reconstruction loss , color loss and smoothness of the illuminance . In ( Jiang & Zheng , 2019 ) , an end-to-end U-Net is used , with a 3D convolution in order to deal with videos , preventing flickering in consecutive frames , and using the GRBG components from the camera sensor as an input . In MBLLEN ( Lv et al. , 2018 ) , a CNN is applied to the input image . Then , the output of each layer is fed to independent autoencoders . The obtained output image is a weighted sum with learnable weights of the autoencoders output . Considering strutural , contextual and regional loss for the training . While autoencoders and U-Net are convenient for image-to-image nets , methods such as Tao et al . ( 2017 ) and ( Ignatov et al. , 2017 ) use only deep neural networks . In particular , these use residual convolutional neural networks to mitigate the vanishing gradient problem in deep networks . In ( Ignatov et al. , 2017 ) , the CNN enhances images from low-end devices to professional DSLR quality , using a dataset with multiple acquisition of the same scene from different devices for the training , which is difficult to obtain . Tao et al . ( 2017 ) uses special-designed residual CNN modules to create a deep CNN , but is then trained using only the SSIM . Apart from ( Wang et al. , 2019 ) , also RetinexNet ( Wei et al. , 2018 ) and Kind++ ( Zhang et al. , 2021 ) augment the Retinex theory , outperforming classic Retinex based methods . Both ( Wei et al. , 2018 ) and ( Zhang et al. , 2021 ) use a CNN to decompose the reflectance and illumination components . In both cases , denoising take advantage of the illumination map , as in ( Guo et al. , 2017 ) , to improve the result over plain denoising . In ( Wei et al. , 2018 ) , the BM3D is used for denoising whereas in ( Zhang et al. , 2021 ) a specific CNN with multi-scale illumination consideration is proposed . For illumination adjustment ( Wei et al. , 2018 ) uses an encoder decoder structure while ( Zhang et al. , 2021 ) adopt a CNN with a free parameter to modify the adjustment ratio . A common critical aspect about the methods considered above is data availability . Since these methods are supervised , they require paired low/normal light images of the same scene . A widely used way to obtain them is to artificially darken normal light images , but this impacts the naturalness of the results . Recently , methods such as ( Wei et al. , 2018 ) and ( Zhang et al. , 2021 ) used datasets with images of the same scenes with different exposures . This proved to be effective , but such datasets are scarce and tedious to create . To overcome the data availability issue , several unsupervised and unpaired supervised methods have been developed . ZDCE ( Guo et al. , 2020 ) is an unsupervised low-light image enhancer which uses a CNN to obtain pixel-wise enhancement curves to adjust the input image , using non-reference loss functions for the training . EGAN ( Jiang et al. , 2021 ) and UEGAN ( Ni et al. , 2020 ) use a GAN to obtain a light enhancement without paired supervision , using a decoder-encoder for the generation . In ( Jiang et al. , 2021 ) a local and a global discriminator are used while in ( Ni et al. , 2020 ) only one multi-scale discriminator is used . The promising results of ( Jiang et al. , 2021 ) and ( Ni et al. , 2020 ) , together with the availability of unpaired low/normal light images , makes GANs an appealing approach for low light image enhancement techniques . Our contribution is similar to EGAN and UEGAN as it uses a GAN architecture with unpaired supervision . Different from these , we do not use a global attention . We rather use separate discriminators for color , texture and edge . Further we make use of cycle consistency , geometric and illumination consistency with contextual loss instead of perceptual losses .

Summary: The authors propose a new image enhancement specific for low light images. They exploit the concepts of geometric and lighting consistency together with a contextual loss criterion.  Extensive experiments are performed on benchmark datasets to compare their results both visually and objectively.