Learning an adaptive model for extreme low‐light raw image processing

IET Image ProcessingVolume 14, Issue 14 p. 3433-3443 Research ArticleFree Access Learning an adaptive model for extreme low-light raw image processing Qingxu Fu, Qingxu Fu Control and Simulation Center, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this authorXiaoguang Di, Corresponding Author Xiaoguang Di dixiaoguang@hit.edu.cn orcid.org/0000-0002-5709-6862 Control and Simulation Center, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this authorYu Zhang, Yu Zhang orcid.org/0000-0002-5848-8892 National Key Laboratory of Tunable Laser Technology, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this author Qingxu Fu, Qingxu Fu Control and Simulation Center, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this authorXiaoguang Di, Corresponding Author Xiaoguang Di dixiaoguang@hit.edu.cn orcid.org/0000-0002-5709-6862 Control and Simulation Center, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this authorYu Zhang, Yu Zhang orcid.org/0000-0002-5848-8892 National Key Laboratory of Tunable Laser Technology, Harbin Institute of Technology, Harbin, 150080 People's Republic of ChinaSearch for more papers by this author First published: 22 October 2020 https://doi.org/10.1049/iet-ipr.2020.0100Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Low-light images suffer from severe noise and low illumination. In this work, the authors propose an adaptive low-light raw image enhancement network to avoid parameter-handcrafting in current deep learning models and to improve image quality. The proposed method can be divided into two sub-models: brightness prediction and exposure shifting (ES). The former is designed to control the brightness of the resulting image by estimating a guideline exposure time . The latter learns to approximate an exposure-shifting operator ES, converting a low-light image with real exposure time to a noise-free image with guideline exposure time . Additionally, structural similarity loss and image enhancement vector are introduced to promote image quality, and a new campus image data set (CID) for training the proposed model is proposed to overcome the limitations of the existing data sets. In quantitative tests, it is shown that the proposed method has the lowest noise level estimation score compared with the state-of-the-art low-light algorithms, suggesting a superior denoising performance. Furthermore, those tests illustrate that the proposed method is able to adaptively control the global image brightness according to the content of the image scene. Lastly, the potential application in video processing is briefly discussed. 1 Introduction Images play an irreplaceable role in industry, military and entertainment. As the art of light, how to obtain better images in the low-light environment has been extensively studied in the literature. Some advanced photography equipment can help but at an expensive cost. On the other hand, there are a few limitations to post-process low-light images. However, it is challenging due to problems such as noise, colour distortion and others. Exposure time, ISO (which measures the sensitivity of the image sensor) and aperture are known as three pillars of photography. With extended exposure time, the low-light problem can be easily solved, but it is not realistic because if the camera is not fixed or the scene contains moving objects. Higher ISO introduces more noise and is not always available for mobile devices. As a flexible solution, low-light image enhancement provides an alternative for imaging in the low-light environment. In general, current low-light image enhancement method can be classified as follows: Classic low-light image enhancement methods, with no application of convolutional neural networks CNNs. Histogram equalisation (HE) [1] and gamma correction [2] provide simple solution for low-light image processing. Based on Retinex theory [3], single-scale retinex (SSR) [4], multi-scale retinex (MSR) [5], SRIE [6] and LIME [7] were proposed, enhancing low-light image by estimating the illumination map and reflectance map. Dehazing [8] methods can also be applied to this subject, regarding low-light images as inversed hazed image. With the rapid development of the deep neural network, researchers have combined classic low-light image enhancement methods with CNNs. Chen et al. [9] proposed Retinex-Net to decompose low-light image into illumination and reflectance and use BM3D [10] for denoising. Those methods assume low-light images are free of noise or noise is already removed by additional denoising process. Therefore, when applying those methods on real-world low-light images, an additional denoising process is required. However, most low-light images suffer from severe noise, traditional denoising methods like BM3D are not able to provide satisfactory results. To be rid of image noise, researchers also explored the method of deep learning [11–14] in denoising topics. There are some low-light images which are barely visible before post-processing, as they are captured in extremely dark environments or with very short exposure time. In this work, those images are named as extreme low-light images. While the classic methods are unable to tackle the severe noise and serious colour distortion in those images, it is then found that with the help of deep learning and raw-format image data sets, we can still obtain satisfactory results. Chen et al. proposed an end-to-end solution for raw low-light image enhancement [15], dealing with the low-light problem and denoising in a single model. However, extra brightness amplification ratio is needed and requires manual parameter tuning for different scenes, gaining excellent denoising effects but losing the flexibility and adaptability of classic methods. Currently, all data sets available for low-light enhancement models chose a longer-exposed image as the ground-truth image in training, which was heavily dependent on the data set collector's experience to select exposure parameters such as exposure time, ISO and white balance mode. Thus these parameters were usually not optimal, considering the content of the scene, illumination of the environment and others, which can lead to an undesired trained model. It is hard for deep CNNs to learn how we determine the best exposure parameters for the ground-truth images because this can be very subjective and lacks specific standards. It can lead to more problems if there is more than one data set collector. On the other hand, making a baseline for the ground-truth selection is equally difficult. Nevertheless, it is possible to predict how an image varies when exposure parameters change. We name this process exposure shifting (ES). With a learned ES model, it becomes possible to reversely study what good exposure parameters (e.g. exposure time) should be suitable for each low-light image, utilising the characteristics of the back-propagation algorithm. The contributions of our work are summarised as follows: (i) We presented campus image data set (CID), which contains a variety of scenes captured by multi-level exposure. CID provides powerful support for the training of our data-driven low-light enhancement model. (ii) We proposed an adaptive two-stage low-light image enhancement model, which provides state-of-the-art low-light noise suppression as well as adaptive global brightness control effect. In stage one, a BPN was introduced to estimate a proper exposure time based on the content of images as well as exchangeable image file (Exif) metadata. BPN was trained to provide adaptive image brightness control during image enhancement and to get rid of the external parameters required in the ES stage. In stage two, an Exposure Shifting Network (ESN) was proposed to estimate a longer-exposed version of the image with target exposure time estimated by BPN. Moreover, ESN also completes image denoising and ends up with the final resulting RGB image. The rest of the article is organised as follows. In Section 2, previous research works are provided. In Section 3, we demonstrate the importance and advantages of our CID data set. In Section 4, our model and its training details are provided, as well as the design concept of the model. The proposed model is compared with other state-of-the-art methods in Section 5, from multiple perspectives including denoising, adaptive brightness control performance and computational cost. The possible extension in video processing is briefly discussed here. In Section 6, the advantages and the limitations of our model are concluded, and we also present the future expectations of this work. 2 Related works The method we proposed was inspired not only by the application of deep neural networks but also by the classic methods based on the HE model, dehazing model and Retinex model. 2.1 HE methods HE method is used extensively to process digital images. The basic idea is to improve image visibility by changing the image histogram into a uniform distribution, which effectively increases image entropy for low-light images. In [16] it is argued that a colour-invariant representation can be obtained by applying HE. However, HE tends to cause noise amplification, details disappearance and colour distortion in low-light image enhancement. On the account of the drawbacks of HE, a number of improvements have been put forward. Adaptive HE (AHE) [1] and its variation contrast-limited AHE [17] perform the HE considering not only the global histogram, but also the neighbours of each pixel. Hue-preserving colour image enhancement [18] works on contrast-enhancing and hue-preserving, [19, 20] on preserving the original image luminance. 2.2 Dehazing and retinex model-based methods A number of low-light enhancement methods are based on the dehazing and Retinex model. It is observed that low-light images and inversed haze images share many similarities. For this reason, Dong et al. [8] presented a method and achieved the low-light image enhancement in which the low-light images is inversed, dehazed and then inversed back again. This method was studied further by [21, 22]. On the other hand, the Retinex model [3] revealed that a natural image is made up of the illumination map and the reflectance map. With Retinex decomposition and the assumption that the illumination map is smooth, researchers have proposed SSR [4] and MSR [4]. Based on image fusion, adjustments can be applied to the illumination map to improve the performance [6]. Wang et al. [23] focused on preserving natural characteristics with a bright-pass filter. In [24], a variational Retinex model was proposed and the illumination estimation problem was formulated as a quadratic programming optimisation problem. In [25] illumination map is estimated considering local structure and [26] focused on revealing the structure details from the reflectance map. A recent study shows that the Retinex model can combine with the atmospheric scattering model [27]. More refined models were presented by [28, 29] based on variational Retinex theory. However, noise modelling is not well established in those algorithms. In [8] it is assumed that noise is insignificant and can be removed before or after the Dehazing stage. In addition, the Retinex-based models rely on classic noise reduction algorithms (e.g. BM3D) to reduce noise found in the reflectance map. Therefore, it can achieve good results in mild low-light images but tends to amplify noise when processing severe or extreme low-light images. Besides, the variational Retinex model is very time-consuming as it takes many iterations to solve the optimisation objectives, thus it is hard to achieve real-time processing. 2.3 Data-driven methods Image-denoising and low-light enhancement can be realised with the data-driven approach, separately or integrally. Noise reduction was the subject of extensive studies. Burger et al. [11] discovered that CNNs could compete with the state-of-the-art classic denoising algorithm BM3D. The works on date-driven denoising have been extended by [13, 14, 30]. Although these works were not targeted at low-light image processing, they provided some references for the research of other image processing studies. The first application of the data-driven method is LLNET [12], which utilised a deep auto-encoder to learn from synthetic low-light image data sets and the denoising procedure is integrated into the low-light enhancement process. Also, it was demonstrated that Retinex based methods can be further improved by deep learning [9, 31], but those works still have undesirable denoising performance because synthetic data sets are used and they cannot reflect the characteristics of real low-light images. To solve the drawbacks of synthetic data sets, a large multi-exposure image data set was proposed in [32]. It was found by Chen et al. [15] that models utilising raw-format images can provide much better results in low-light enhancement. In the learning-to-See-In-the-Dark data set (SID) model proposed in [15], it suggested that the state-of-the-art results can be achieved by using raw-format paired images and training in an end-to-end way. However, since the paired images have different exposure time, a ratio was introduced as an additional input to bridge the gap. Consequently, if the SID model is applied to images out of the data set, manual adjustment is required because there are no paired images to indicate the ratio, which results in many drawbacks in practical applications. 3 Data set For extreme low-light image enhancement, there are few optional data sets. The Google HDR+ data set [33] and Darmstadt noise data set [34] avoid the disadvantages of synthetic data sets, but they were mainly collected in environments with sufficient illumination. The RENOIR [35] and SID [15] provides real low-light image pairs, which is captured by carefully selecting the ISO and exposure time for each scene. The exclusively dark data set [36] is another low-light data set with object-level annotations, which is targeting at object recognition. However, these data sets cannot meet the requirements of our study. In our work, in order to adaptively enhance images with different noise levels, it is needed to consider how the exposure parameters have an impact on low-light images. More specifically, as the environment illumination grows lower, low-light images are gradually overwhelmed by noise and invalid pixels. Although it is not flexible to manipulate the environment illumination, the camera exposure time setting can be easily changed to simulate this illumination shift. Therefore, it is expected that there is a data set containing, instead of just some image pairs, a number of multi-exposed image series, in each of which there are low-light images of different levels and one reference image captured in the same scene. Overall, there are three conditions to be met for a satisfactory data set: • Real scenes: Unlike synthetic images, photos captured in real scenes reflect much more diverse noise and distortion pattern • Multiple exposure levels: Using a number of multi-exposed image series to train an adaptive model capable of enhancing low-light images of different levels. • Raw-format images: Most data captured by the camera sensor is lost in format conversion, That lost information is essential for extreme low-light image enhancement. Based on the above description, we proposed our CID. There are about 200 image groups in CID, each of which consists of eight raw images with different exposure time, shot continuously in the same scene using a tripod (Fig. 1). To be specific, for every group, an upper limit of exposure time was selected to make sure that the first image in the sequence was slightly overexposed, then a lower limit was chosen to capture an extreme low-light image as the last image in the sequence, and the gap between the limits was filled up by another six images. Finally, the ground-truth image for each group was manually selected, and invalid images that had nothing but noise due to extreme short exposure were tagged and excluded. A demonstration of scenes in CID is presented in Fig. 2. Fig. 1Open in figure viewerPowerPoint Demonstration of a CID image group Fig. 2Open in figure viewerPowerPoint Several scenes in CID data sets 4 Method 4.1 Method overview Most of the existing methods are unable to suppress the severe noise present in extreme low-light images. On the other hand, recent studies, e.g. [15] have concentrated on the extreme low-light enhancement with raw-format images and real-scene data sets in an end-to-end way, but unlike the classic methods, the brightness of the enhanced image cannot be automatically controlled in this framework, more specifically, an external ratio has to be manually adjusted when the input image has no paired image as the reference. It is our aim to find a solution to enhance extreme low-light images, combining the advantages of adaptive brightness control and superior denoising performance. It is found that the problem of low-light enhancement can be considered as a special case of changing the time of exposure. With and representing exposure time, the low-light image can be interpreted as a normal image corrupted by a hypothetical exposure-shifting operator ES, which only alters the exposure time but keeps the aperture and ISO unchanged (1)In this process, more stochastic noise is generated by operator ES because , corresponding to the low peak signal-to-noise ratio (PSNR) value in low-light images. Contrariwise, the corrupted low-light image can also be restored by the same operator ES: (2)The noise is reduced because . Thus the normal image is reconstructed. The latter process, named as ES, can enhance low-light images as well as suppress image noise. More importantly, it can be learned in a data-driven and end-to-end way, thus it is expected to have superior denoising performance. However, a proper guideline exposure time always has to be provided in the reconstruction . To make our model adaptive, a brightness prediction (BP) procedure is introduced to achieve guideline time estimation. In brief, the extreme low-light enhancement process is split into two procedures. First, for a raw-format low-light image (with real exposure time ), an optimal guideline exposure time is estimated via a sub-model called BPN. Second, the final RGB-format enhanced image is obtained by another sub-model, namely ESN, using estimated guideline time to control the image brightness. In addition, besides the image itself, some extra data are involved in both BPN and ESN, such as white balance index, ISO, and . The image-enhancing vector (IEV) is put forward to encode the involved extra data. 4.2 Image enhancing vector Raw image array is the original data captured by the camera. Other key parameters, such as camera white balance, ISO, exposure time, time and date, are stored together with the raw image array as Exif metadata. In HDR imaging [37], exposure time and ISO information have played an essential role in calculating the response curve and LDR to HDR conversion. Although it is not necessary to explicitly train the network to learn the camera response curve, feeding these additional parameters into the network as input avoids underfitting caused by insufficient features. IEV is proposed to introduce extra features into the convolution neural networks. In this paper, IEV consists of ISO , white balance indices for four raw image channels , exposure time of the low-light image and guideline exposure time . Moreover, it is possible to append other Exif metadata into IEV to improve network performance when necessary, e.g. aperture and focal length. The IEV is used as network input in both BPN (for estimation) and ESN (for ES), but a difference exists. As it is shown in Fig. 3, in ES procedure the IEV can be written as (3)where are the white balace indices of four raw image channels and is the ISO value (controlling the camera sensitivity). Fig. 3Open in figure viewerPowerPoint Proposed framework. For a underexposed low-light raw image with guideline exposure time already known, the ESN is applied to complete the estimation of the RGB result image. In a more general case, BPN is utilised to give a proper estimation for the uncertain parameter with raw image array and available Exif metadata In BP procedure the in IEV is removed. It is renamed as partial IEV (pIEV) to indicate the difference (4) IEV and pIEV are fed into the CNN as additional channels of the raw image array. For example, the first extra channel provided by IEV is a uniform image filled up with pixel value . 4.3 Exposure shifting ES is the second procedure in our model but is trained first. The U-NET [38] is selected as the structure of the ESN, Compared with the fully convolutional network, the U-NET architecture reduced the usage of GPU memory, thus it can easily process images with high resolution. Recall that the basic unit of our CID is an image group, which consists of 8 raw-format images with different exposure time but captured in the same scene. Let be the paired raw image extracted from one group, in which is the pre-selected ground-truth image and is a randomly selected low-light image. More specifically, is randomly chosen within this image group, with and over-exposed images excluded (Fig. 1). Moreover, the exposure time of is denoted as . The corresponding RGB-format image of is denoted as and its exposure time as . It is expected for the ESN to learn the exposure-shifting operator ES from a large number of paired images. Specifically, the ESN is required to estimate an image that resembles the as closely as possible. In this way ESN successfully changes the raw-format low-light image into an RGB-format longer-exposed version of itself. More importantly, the serious noise of can also be removed. This procedure can be written as (5)where ESN is denoted as , its parameters as . Note that since the BP procedure has not yet involved, the guideline exposure time is replace by in IEV . The enhanced result image is represented by . Two metrics to guide the training process of the ESN model are mean square error (MAE) and multi-scale structural similarity (SSIM) [39]. The former is considered as a simple but effective indicator, evaluating the general similarity of the estimated image and the groud-truth image. AndIn addition, the later is a perceptual metric that is more sensitive to visible structures. Both of them are full-reference metrics. Let be the MAE loss and be the SSIM loss. The width and length of the image are denoted as m and n, respectively. With subscript omitted, the MAE loss and SSIM loss are defined as: (6)The final MAE loss is the average of the channel MAE loss, for simplicity, the image channel is not presented in the equation. The calculation of the multi-scale SSIM can be refered in [39]. With subscript omitted, the loss function is defined as the linear combination of and (7)where is a constant and . The influence of will be discussed in the Experiments section. The parameters of the ESN network are learned by minimising over K pairs of images in the training set (8) represents the optimised ESN parameters. The obtained sub-model is an approximation to the exposure-shifting operator ES. However, for a low-light image out of the training set, a ground-truth counterpart does not exist, thus the guideline time in IEV is absent, which leads to the BP sub-model and guideline time estimation. 4.4 Brightness prediction The BP procedure provides the estimation of guideline exposure time (9)Where and is BPN and its parameters respectively. The pIEV is represented by . It is the first enhancement procedure in our model but trained after the ESN. As it is illustrated in Fig. 4, The BPN is a relatively small network, which consists of multiple convolution layers and then ends up with several fully connected layers. Fig. 4Open in figure viewerPowerPoint Components of BPN Owing to the existence of fully connected layers, the width and length of the input image should be uniformly changed into . With the estimated obtained in this BP procedure, the item in IEV in the ES procedure can be replaced by . Different from derived in (5), the new final result image can be formulated as (10) Then an approach to train the BPN is required. To begin with, the purpose of BPN is to control the brightness of the final result image, which is achieved by adjusting the guiding exposure time in the ES procedure. Since there are image pairs and their exposure time available, intuitively it seems that the BPN can simply be learned from in an end-to-end way, with as the input and as the label. However it cannot be achieved, the reason for which will be discussed later in the next subsection. The proposed BPN training approach and the loss function are as follows. We define the brightness of a single-pixel Br as the average pixel value of all colour channels. Given a certain dark scene for image capturing and with ISO, aperture settings and the scene all fixed, then the brightness of a pixel is only determined by exposure time. Let be the pixel irradiance in this scene, when the exposure time t in the camera settings increases, the pixel exposure increases as well [40]: (11)Thus the pixel becomes brighter. It is feasible to design loss function based on the pixel brightness. It should be noted that the relationship between pixel brightness and Exposure is not linear, instead, it can be typically described by an S-shaped curve because the pixel brightness is limited within (or for 8-bit image storage) and the camera sensor is less sensitive to the change of t when the pixel value is close to 0 or 1. In other words, as the exposure time t changes, some pixels are not sensitive and there are little changes in the pixel brightness compared with some other pixels. For example, when photographing a street at night, pixels revealing the lamp bulbs are always saturated unless the t is extremely small, and pixels representing the dark sky are always close to zero unless it is affected by noise. If those pixels are involved in BPN training, they can cause gradient vanishing in the backpropagation algorithm because and as it is revealed in the S-shaped curve, is close to 0. On the other hand, when pixel brightness Br is around the reaches its maximum. The pixels meeting this condition can be collected into an area of interest (AoI), which identifies pixels that are sensitive to the process of ES. First, in the ground-truth image , an AoI weight map can be obtained by filtering out pixels with brightness near . More specifically, the ground-truth image is converted into a single channel grayscale image , then a Gaussian culve with mean and variance is applied on each pixel of (12)Then is normalised, the result is the AoI weight map . The value at position (i,j) is denoted as subscripts, namely and (13)Combining the (12) and (13), they can be simplified by the softmax function (14) Lastly, the loss function is designed to check if the final result image has the same AoI. Having the same AoI means that in this AoI area and share similar brightness. Moreover, recall that pixels outside AoI are not sensitive to the change of exposure time t or the ES process, thus it can be derived that the overall image brightness of resembles that of . In this way, BPN completed the adaptive BP objective. Then, the enhanced image is converted into a grayscale image . When the AoI of and overlaps exactly, the following loss function reaches its minimum: (15)where is another variance constant. The image value at position (i,j) is represented by i,j subscripts. Finally, BPN is trained by optimising over K pairs of images in the training set (16) 4.5 Method summary and discussion In Algorithm 1 (see Fig. 5), the training process of our model is summarised, together with the method to evaluate and apply this model. Fig. 5Open in figure viewerPowerPoint Algorithm 1: Model training and evaluation As mentioned earlier in the last section, training BPN with input and label is straightforward but not feasible. First let be the hypothetical optimal BPN model, then is the optimal estimation of guideline exposure time. For a pair of image and time label , the can be considered as one sampling from a Gaussian distribution: . The variance in this process is too large, the reason are: • In every CID image group, the ground-truth image is chosen manually and is dominated by subjective human judgments. • The ground-truth image is selected from only 8 candidate images, but the optimal exists somewhere in the time continuum . Furthermore, the number of samples is very limited since each image group can only provide one sample. As a result, this straightforward approach cannot