# Image Demosaicing Based on Generative Adversarial Network

Authors: Jingrui Luo; Jie Wang
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Digital cameras with a single sensor use a color filter array (CFA) that captures only one color component in each pixel. Therefore, noise and artifacts will be generated when reconstructing the color image, which reduces the resolution of the image. In this paper, we proposed an image demosaicing method based on generative adversarial network (GAN) to obtain high-quality color images. The proposed network does not need any initial interpolation process in the data preparation phase, which can greatly reduce the computational complexity. The generator of the GAN is designed using the U-net to directly generate the demosaicing images. The dense residual network is used for the discriminator to improve the discriminant ability of the network. We compared the proposed method with several interpolation-based algorithms and the DnCNN. Results from the comparative experiments proved that the proposed method can more effectively eliminate the image artifacts and can better recover the color image.


---

## Body

## 1. Introduction

Images are widely used in people’s daily life. Compared to analog images, digital images are more superior in their higher resolution and easier storage, and they are more suitable for computer processing. With the development of computer technology, the digital imaging technology has attracted lots of attention and digital cameras have gradually become the mainstream imaging equipment that is widely used in intelligent transportation [1, 2], medical imaging [3, 4], remote sensing technology [5, 6] and other fields. In our daily life, the digital color images are most commonly used, which include three color components, that is, red, green, and blue, in each pixel. Ideally, digital cameras with three sensors can get full-color images with each sensor capturing one color component, and the three components are combined together into a color image. However, in practice, the arrangement of the three color sensors will affect the subsequent color synthesis, and cameras with three sensors are usually expensive and relatively large. Therefore, most digital cameras use single sensor with a color filter array (CFA) placed in front of the sensor. The obtained CFA image needs to be processed to acquire the full-color image and this process is known as image demosaicing [7]. As only one color component is captured for each pixel in the CFA, without image demosaicing, the CFA image can only reflect the general outline of the scenery instead of the complete color information, which consequently affects subsequent image processing [8].The CFA image demosaicing is essentially an ill-posed inverse problem [9]. The methods for image demosaicing generally include interpolation-based algorithms and learning-based algorithms. Generally, image demosaicing using interpolation methods can achieve high accuracy for smooth areas with approximately same colors and gradient brightness. For the color images, the red, green, and blue components occupy different color channels, respectively. When the high-frequency signals change (high-frequency single/information refers to the region with strong color variation, such as edges and angles), there may be spatial offsets in each color channel. Therefore, the reconstructed images may display color artifacts and zippering when doing interpolation [7]. In addition, some traditional interpolation-based methods ignore the correlation among different color channels, which results in unsmooth images [8]. On the whole, the interpolation-based algorithms still have some limitations for image demosaicing, especially at the high-frequency areas.In recent years, neural networks have been rapidly developed and widely used in image processing, such as image classification [10, 11], motion recognition [12, 13], and image super-resolution [14, 15]. Recently, the generative adversarial network (GAN) [16] has been proposed and rapidly attracts attention of many researchers. Ledig et al. [17] proposed a super-resolution generative adversarial network (SRGAN), which used a deep residual network for the training and can well recover the image textures from greatly downsampled images. Inspired by super-resolution image reconstruction and the conditional generative adversarial network (CGAN), Kupyn et al. [18] applied the CGAN to image deblurring and effectively restored clear images. Pan et al. [19] proposed a physics model constrained learning algorithm so that it can guide the estimation of the specific task in the conventional GAN framework, which can directly solve image restoration problems (such as image deblurring and image denoising).GAN has been used and played important roles in several areas; however, it has not been used for image demosaicing. In this paper, we proposed a novel learning-based image demosaicing method using GAN to improve the ability for color image recovery. Our contributions are as follows:(1)
We proposed a CFA image demosaicing method based on GAN(2)
We carefully designed each part for the GAN model(3)
We introduced long jump connections for the improved U-net [20] model to design the generator(4)
We used the dense residual network, which includes dense residual blocks with long jump links and dense connections for the discriminator(5)
We combined the adversarial loss, the feature loss, and the pixel loss together to further strengthen the network performanceIn the experimental section, we show the performance of our method using some comparative experiments. The results prove that the proposed method can more effectively remove artifacts and recover the full-color image, especially for some high-frequency areas such as edges and angles.


## 2. Related Works

### 2.1. Interpolation-Based Algorithms

There are many interpolation-based methods for image demosaicing. Linear interpolation algorithm is the simplest one. However, this method often causes artifacts and blurring at the image edges [21]. The bilinear interpolation algorithm [22] estimates the unknown pixels from their adjacent pixels. This method often causes color distortion in the reconstructed image. Malvar et al. [23] proposed a high-quality linear (HQL) interpolation algorithm, which can greatly reduce the computational complexity. However, the artifacts still occur at high-frequency components of the image. In order to further reduce the artifacts, different interpolation techniques were proposed.Within the gradient-based schemes, Hamilton and Adam [24] proposed the Hamilton–Adam algorithm, which uses the second derivative of the sampled color channels when doing interpolation. Therefore, this method considers the correlation among different color channels and significantly improves the image details. Mukherjee et al. [25] proposed a two-line (TL) interpolation algorithm, which used the homogeneity of the cross-ratios of different spectral components around a small neighborhood to interpolate the pixels lying in the low gradient directions, so as to produce high-quality images.Within the directional interpolation schemes, Chung and Chan [26] used the prior decision in the horizontal interpolation and the vertical interpolation and got the interpolation result according to the trend of the image edges. This method is prone to producing false colors at tiny edges, especially when the edges are not in the horizontal or vertical directions. Zhang et al. [27] proposed a local directional interpolation and nonlocal adaptive thresholding (LDI-NAT) algorithm. This method used the nonlocal redundancy of the image to improve the local color reproduction and can better reconstruct the edges and reduce color artifacts.Within the residual interpolation schemes, Kiku et al. [28] proposed a minimized-Laplacian residual interpolation (MLRI) algorithm. This method estimated the tentative pixel values by minimizing the Laplacian energy of the residuals, which can effectively reduce the color artifacts. Monno et al. [29] proposed an adaptive residual interpolation (ARI) algorithm, which adaptively selects a suitable iteration number and combines two different types of residual interpolation algorithms at each pixel. Kiku et al. [30] incorporated the residual interpolation algorithm into the gradient-based threshold free (RI-GBTF) algorithm, and the interpolation accuracy is greatly improved. Besides, L. Zhang and D. Zhang [21] proposed a joint demosaicing-zooming scheme. This method used the correlation of the hyperspectral spatial for the CFA image to calculate the color difference, so as to restore the three color components, which can effectively eliminate color artifacts.


### 2.2. Learning-Based Algorithms

Recently, neural networks have also been used for image demosaicing. Prakash et al. [31] used a denoising convolution neural network (DnCNN) to perform demosaicing and denoising independently, which effectively suppressed the noise and artifacts. Tan et al. [32] used the deep residual network for image demosaicing and image denoising, which also effectively obtained high-resolution color images. Shopovska et al. [33] proposed an improved residual U-net and used it for image demosaicing, which achieved high-quality reconstructed color images for different CFA patterns. Generally, the learning-based strategies can achieve better performance compared to the traditional interpolation-based methods. However, higher-resolution and clearer recovered color images are the constant pursuit for image demosaicing; that is why we are trying GAN for this task.


## 2.1. Interpolation-Based Algorithms

There are many interpolation-based methods for image demosaicing. Linear interpolation algorithm is the simplest one. However, this method often causes artifacts and blurring at the image edges [21]. The bilinear interpolation algorithm [22] estimates the unknown pixels from their adjacent pixels. This method often causes color distortion in the reconstructed image. Malvar et al. [23] proposed a high-quality linear (HQL) interpolation algorithm, which can greatly reduce the computational complexity. However, the artifacts still occur at high-frequency components of the image. In order to further reduce the artifacts, different interpolation techniques were proposed.Within the gradient-based schemes, Hamilton and Adam [24] proposed the Hamilton–Adam algorithm, which uses the second derivative of the sampled color channels when doing interpolation. Therefore, this method considers the correlation among different color channels and significantly improves the image details. Mukherjee et al. [25] proposed a two-line (TL) interpolation algorithm, which used the homogeneity of the cross-ratios of different spectral components around a small neighborhood to interpolate the pixels lying in the low gradient directions, so as to produce high-quality images.Within the directional interpolation schemes, Chung and Chan [26] used the prior decision in the horizontal interpolation and the vertical interpolation and got the interpolation result according to the trend of the image edges. This method is prone to producing false colors at tiny edges, especially when the edges are not in the horizontal or vertical directions. Zhang et al. [27] proposed a local directional interpolation and nonlocal adaptive thresholding (LDI-NAT) algorithm. This method used the nonlocal redundancy of the image to improve the local color reproduction and can better reconstruct the edges and reduce color artifacts.Within the residual interpolation schemes, Kiku et al. [28] proposed a minimized-Laplacian residual interpolation (MLRI) algorithm. This method estimated the tentative pixel values by minimizing the Laplacian energy of the residuals, which can effectively reduce the color artifacts. Monno et al. [29] proposed an adaptive residual interpolation (ARI) algorithm, which adaptively selects a suitable iteration number and combines two different types of residual interpolation algorithms at each pixel. Kiku et al. [30] incorporated the residual interpolation algorithm into the gradient-based threshold free (RI-GBTF) algorithm, and the interpolation accuracy is greatly improved. Besides, L. Zhang and D. Zhang [21] proposed a joint demosaicing-zooming scheme. This method used the correlation of the hyperspectral spatial for the CFA image to calculate the color difference, so as to restore the three color components, which can effectively eliminate color artifacts.


## 2.2. Learning-Based Algorithms

Recently, neural networks have also been used for image demosaicing. Prakash et al. [31] used a denoising convolution neural network (DnCNN) to perform demosaicing and denoising independently, which effectively suppressed the noise and artifacts. Tan et al. [32] used the deep residual network for image demosaicing and image denoising, which also effectively obtained high-resolution color images. Shopovska et al. [33] proposed an improved residual U-net and used it for image demosaicing, which achieved high-quality reconstructed color images for different CFA patterns. Generally, the learning-based strategies can achieve better performance compared to the traditional interpolation-based methods. However, higher-resolution and clearer recovered color images are the constant pursuit for image demosaicing; that is why we are trying GAN for this task.


## 3. Problem Formulation

### 3.1. CFA Image

To obtain a color image with detailed description of the natural image, the best solution is to use three sensors to accept the red, green, and blue components for each pixel, respectively. Then the color image can be synthesized by combining the three color components. Considering the cost and volume, most digital cameras use a single image sensor for the image acquisition systems. The image acquisition of the camera with single sensor is shown in Figure1. The CFA is set before the sensor. For common CFA, such as the Bayer pattern [34] that is used in this work, the light reaching the sensor mainly consists of the red, green, and blue components. Within the CFA, each pixel only accepts one color component. As shown in Figure 1, the obtained Bayer pattern image can only estimate the approximate gray outline of the scenery instead of the complete color information. The color arrangement of Bayer pattern can be clearly seen from the local zoomed in area. In the Bayer pattern, a set of red and green filters and a set of green and blue filters are alternately used. The number of green pixels is 1/2 of the total number of pixels, while the numbers of red and blue pixels are both 1/4 of the total number of pixels. As only one color component is captured for each pixel, the other two color components need to be recovered according to the color information from adjacent pixels; then a full-color image is obtained from the CFA image. This processing is called image demosaicing.Figure 1
Image acquisition of the digital cameras with single sensor.


### 3.2. Theory of GAN

GAN is a kind of probabilistic generative network, which was first introduced by Goodfellow et al. [16] into the deep learning field. The general architecture of GAN is shown in Figure 2. GAN uses G to perform inverse transformation sampling of the probability distribution and capture the distribution of the ground truth data x. Based on noise data z which obeys a certain distribution (such as Gaussian distribution), G will generate a fake sample Gz similar to x. The output of D represents the probability of the incoming data. Thus, if the input is x, the output Dx is a large probability value; otherwise it outputs a small probability. The GAN’s training process is to maximize the discrimination accuracy by training D, as well as to minimize the difference between the generated sample and the real sample by training G. Thus, the training for G and D is a min-max game problem. The performances of G and D are improved by alternative optimization. Finally, G and D reach Nash equilibrium, so that the data distribution synthesized by G is similar to that of the ground truth data x. The loss function of the above process is defined as(1)minGmaxDVD,G=Ex∼PdataxlogDx+Ez∼Pnoisezlog1−DGz,where VD,G represents the value function [16]. x∼Pdatax represents ground truth data x obeying a real data distribution Pdatax, and z∼Pnoisez represents noise data z obeying a simulated distribution Pnoisez (such as the Gaussian distribution). Dx and DGz are the classification outputs of D for the ground truth data x and the generated data Gz, respectively. E means expectation.Figure 2
The general architecture of GAN.


## 3.1. CFA Image

To obtain a color image with detailed description of the natural image, the best solution is to use three sensors to accept the red, green, and blue components for each pixel, respectively. Then the color image can be synthesized by combining the three color components. Considering the cost and volume, most digital cameras use a single image sensor for the image acquisition systems. The image acquisition of the camera with single sensor is shown in Figure1. The CFA is set before the sensor. For common CFA, such as the Bayer pattern [34] that is used in this work, the light reaching the sensor mainly consists of the red, green, and blue components. Within the CFA, each pixel only accepts one color component. As shown in Figure 1, the obtained Bayer pattern image can only estimate the approximate gray outline of the scenery instead of the complete color information. The color arrangement of Bayer pattern can be clearly seen from the local zoomed in area. In the Bayer pattern, a set of red and green filters and a set of green and blue filters are alternately used. The number of green pixels is 1/2 of the total number of pixels, while the numbers of red and blue pixels are both 1/4 of the total number of pixels. As only one color component is captured for each pixel, the other two color components need to be recovered according to the color information from adjacent pixels; then a full-color image is obtained from the CFA image. This processing is called image demosaicing.Figure 1
Image acquisition of the digital cameras with single sensor.


## 3.2. Theory of GAN

GAN is a kind of probabilistic generative network, which was first introduced by Goodfellow et al. [16] into the deep learning field. The general architecture of GAN is shown in Figure 2. GAN uses G to perform inverse transformation sampling of the probability distribution and capture the distribution of the ground truth data x. Based on noise data z which obeys a certain distribution (such as Gaussian distribution), G will generate a fake sample Gz similar to x. The output of D represents the probability of the incoming data. Thus, if the input is x, the output Dx is a large probability value; otherwise it outputs a small probability. The GAN’s training process is to maximize the discrimination accuracy by training D, as well as to minimize the difference between the generated sample and the real sample by training G. Thus, the training for G and D is a min-max game problem. The performances of G and D are improved by alternative optimization. Finally, G and D reach Nash equilibrium, so that the data distribution synthesized by G is similar to that of the ground truth data x. The loss function of the above process is defined as(1)minGmaxDVD,G=Ex∼PdataxlogDx+Ez∼Pnoisezlog1−DGz,where VD,G represents the value function [16]. x∼Pdatax represents ground truth data x obeying a real data distribution Pdatax, and z∼Pnoisez represents noise data z obeying a simulated distribution Pnoisez (such as the Gaussian distribution). Dx and DGz are the classification outputs of D for the ground truth data x and the generated data Gz, respectively. E means expectation.Figure 2
The general architecture of GAN.


## 4. The Proposed Method

In this section, we propose an effective demosaicing algorithm based on GAN. The whole process is shown in Figure3. The proposed algorithm first extracts the red, green, and blue components from the original CFA image to form the 3-channeled split CFA image. Then the extracted green component is further separated into two channels to form the 4-channeled split CFA image. Subsequently, the algorithm extracts only the pixel values that are not 0 to compress the 4-channeled split CFA image. The compressed 4-channeled image is taken as the input of G in GAN. The output of G is the interpolated 3-channel full-color image. The output images from G and the ground truth images are then inserted into D. The parameters of G are optimized according to the output of D. We designed the architectures for G and D and trained the database through an end-to-end trainable neural network. In addition, the algorithm combined the adversarial loss, pixel loss, and feature loss to design the generator loss function in order to further improve the network performance [35]. In the following, we give a detailed introduction to different parts of the network.Figure 3
The process of the proposed CFA image demosaicing.


### 4.1. Generator

The purpose of the generatorG is to convert the 4-channeled compressed CFA images to the 3-channeled output full-color images. The structure of G is shown in Figure 4. We used the improved U-net [20] model for G. Overall, the generator G consists of an encoder (the first half) and a decoder (the second half), which is shown in Figure 4. One layer in the encoder and the corresponding layer in the decoder form a U-shaped symmetric layer. The long jump links within each symmetric layer in the U-net model can reduce the information redundancy. Besides, we remove the pooling layer in the U-net, which can avoid the loss of useful information in the feature maps and increase the stability of the training process.Figure 4
The structure of the generator.The encoder is mainly based on the downsampling operation (i.e., convolution operation). It can analyze the input data to obtain the most significant features and provide feature mappings to its corresponding layer in the decoder. The activation function of the encoder is a leaky rectified linear unit (LReLU), which is defined as(2)LReLUxe=max0,xe+κ×min0,xe,where κ is a positive constant (κ∈0,1). xe represents the input vectors for a specific layer of the encoder. In our experiments, we set κ as 0.1.The decoder is mainly based on the upsampling operation (i.e., deconvolution operation) to restore the full-color images. The activation function of the decoder is a standard rectifier linear unit (ReLU), which is defined as(3)ReLUxd=max0,xd,where xd represents the input vectors for a specific layer of the decoder.Particularly for the final layer of the decoder, the activation function is the tanh activation function, which is defined as(4)tanhxf=exf−e−xfexf+e−xf,where xf represents the input vectors for the final layer of the decoder.In order to accelerate the convergence and improve the network performance, we introduce the batch normalization (BN) operation after each convolution and deconvolution operation to slow down the transfer of internal covariates and reduce the sensitivity of the network to initialization weights [36].Detailed parameters for the convolution and deconvolution layers are shown in Table1.Table 1
The parameters for the generator.
ConvolutionEncoderKernel3 × 34 × 44 × 44 × 44 × 44 × 44 × 44 × 4Stride1 × 12 × 22 × 22 × 22 × 22 × 22 × 22 × 2Channel163264128256256512512DeconvolutionDecoderKernel4 × 44 × 44 × 44 × 44 × 44 × 44 × 43 × 3Stride2 × 22 × 22 × 22 × 22 × 22 × 22 × 21 × 1Channel51251225625612864323


### 4.2. Discriminator

We used a dense residual network, which is inspired by the ResNet [36], for the discriminator D. The ResNet is formed by stacking multiple consecutive residual blocks (RB). In order to improve the network performance and solve the problem of gradient disappearance and gradient dispersion during the network training, we used an improved residual dense block (RDB). The structure of D is shown in Figure 5. The long jump connection after each RDB helps to transfer the output of this RDB to the final convolution layer. Within each RDB, there are several units with each unit consisting of the ReLU activation function, the convolution layer, and the BN operation. There are dense connections with different distances among these units. The output from the final convolution layer is mapped into 0 or 1 using the sigmoid activate function. The sigmoid function performs a probability analysis that can normalize the discriminant result, which is defined as(5)sigmoidxs=11+e−xs,where xs represents the input vectors for the sigmoid function.Figure 5
The structure of the discriminator.For the convolution layers inD, the kernel size is set as 3 × 3, the stride size is set as 1 × 1, and an output channel is 64.


### 4.3. Loss Function

Denote the ground truth imagesxi,i=1,2,…,N, where N represents the number of images. After a series of operations, the CFA images are transformed into the corresponding 4-channeled compressed CFA images, which are denoted as yi,i=1,2,…,N and are regarded as the input of G. According to the loss function inspired by Alsaiari et al. [35], we combined the adversarial loss, the feature loss, and the pixel loss together with appropriate weights to work as the final loss function for the generator. The adversarial loss function (La) is expressed as(6)La=1N∑i=1Nlog1−DGyi,where yi represents the i−th 4-channeled compressed CFA image. The 3-channeled color images are produced using Equation (6) to fool the generator G.The feature loss function (Lf) is defined as(7)Lf=1N∑i=1NFGyi−Fxi2,where F· represents the feature mapping matrix extracted from the pretrained VGG network [35]. ·2represents the L2 norm. Using Equation (7), we can extract the image features and restore the image details by comparing feature data between the generated image Gyi and the ground truth image xi.The pixel loss (pixel-to-pixel Euclidean distance) function (Lp) is defined as(8)Lp=1N∑i=1NGyi−xi2+λ∇Gyi2,where ∇Gyi2 is the regularization item, with λ representing the regularization weight. Using Equation (8), we can correctly restore the image information by comparing each pixel between the generated image Gyi and the ground truth image xi.We combineLa, Lf, and Lp together with appropriate weights to form the final loss function for the generator, which is defined as(9)LG=αLa+βLf+γLp,where α, β, and γ represent the predefined positive weights according to the empirical values [35].According to Equation (1), the discriminator D uses the following equation to update the parameters:(10)LD=maxD1N∑i=1NlogDxi+log1−DGyi.For the ground truth imagexi, the probability of the output Dxi is close to 1. For the generated image Gyi, the probability of the output DGyi is close to 0.Based on the above strategy, the generatorG and the discriminator D will be alternately optimized.Based on the above introduction, we give the whole pipeline in Figure6 to clearly describe the proposed method. The real scenery is captured by the camera and converted to the CFA image. The obtained CFA image is then further converted to the 4-channeled compressed CFA images, which is inserted into the generator designed by the U-net model. The output from the generator and the ground truth image are then inputted into the discriminator that is designed by the dense residual network. The generator will finally give near-real demosaiced image through the network training.Figure 6
The pipeline of the proposed method.


## 4.1. Generator

The purpose of the generatorG is to convert the 4-channeled compressed CFA images to the 3-channeled output full-color images. The structure of G is shown in Figure 4. We used the improved U-net [20] model for G. Overall, the generator G consists of an encoder (the first half) and a decoder (the second half), which is shown in Figure 4. One layer in the encoder and the corresponding layer in the decoder form a U-shaped symmetric layer. The long jump links within each symmetric layer in the U-net model can reduce the information redundancy. Besides, we remove the pooling layer in the U-net, which can avoid the loss of useful information in the feature maps and increase the stability of the training process.Figure 4
The structure of the generator.The encoder is mainly based on the downsampling operation (i.e., convolution operation). It can analyze the input data to obtain the most significant features and provide feature mappings to its corresponding layer in the decoder. The activation function of the encoder is a leaky rectified linear unit (LReLU), which is defined as(2)LReLUxe=max0,xe+κ×min0,xe,where κ is a positive constant (κ∈0,1). xe represents the input vectors for a specific layer of the encoder. In our experiments, we set κ as 0.1.The decoder is mainly based on the upsampling operation (i.e., deconvolution operation) to restore the full-color images. The activation function of the decoder is a standard rectifier linear unit (ReLU), which is defined as(3)ReLUxd=max0,xd,where xd represents the input vectors for a specific layer of the decoder.Particularly for the final layer of the decoder, the activation function is the tanh activation function, which is defined as(4)tanhxf=exf−e−xfexf+e−xf,where xf represents the input vectors for the final layer of the decoder.In order to accelerate the convergence and improve the network performance, we introduce the batch normalization (BN) operation after each convolution and deconvolution operation to slow down the transfer of internal covariates and reduce the sensitivity of the network to initialization weights [36].Detailed parameters for the convolution and deconvolution layers are shown in Table1.Table 1
The parameters for the generator.
ConvolutionEncoderKernel3 × 34 × 44 × 44 × 44 × 44 × 44 × 44 × 4Stride1 × 12 × 22 × 22 × 22 × 22 × 22 × 22 × 2Channel163264128256256512512DeconvolutionDecoderKernel4 × 44 × 44 × 44 × 44 × 44 × 44 × 43 × 3Stride2 × 22 × 22 × 22 × 22 × 22 × 22 × 21 × 1Channel51251225625612864323


## 4.2. Discriminator

We used a dense residual network, which is inspired by the ResNet [36], for the discriminator D. The ResNet is formed by stacking multiple consecutive residual blocks (RB). In order to improve the network performance and solve the problem of gradient disappearance and gradient dispersion during the network training, we used an improved residual dense block (RDB). The structure of D is shown in Figure 5. The long jump connection after each RDB helps to transfer the output of this RDB to the final convolution layer. Within each RDB, there are several units with each unit consisting of the ReLU activation function, the convolution layer, and the BN operation. There are dense connections with different distances among these units. The output from the final convolution layer is mapped into 0 or 1 using the sigmoid activate function. The sigmoid function performs a probability analysis that can normalize the discriminant result, which is defined as(5)sigmoidxs=11+e−xs,where xs represents the input vectors for the sigmoid function.Figure 5
The structure of the discriminator.For the convolution layers inD, the kernel size is set as 3 × 3, the stride size is set as 1 × 1, and an output channel is 64.


## 4.3. Loss Function

Denote the ground truth imagesxi,i=1,2,…,N, where N represents the number of images. After a series of operations, the CFA images are transformed into the corresponding 4-channeled compressed CFA images, which are denoted as yi,i=1,2,…,N and are regarded as the input of G. According to the loss function inspired by Alsaiari et al. [35], we combined the adversarial loss, the feature loss, and the pixel loss together with appropriate weights to work as the final loss function for the generator. The adversarial loss function (La) is expressed as(6)La=1N∑i=1Nlog1−DGyi,where yi represents the i−th 4-channeled compressed CFA image. The 3-channeled color images are produced using Equation (6) to fool the generator G.The feature loss function (Lf) is defined as(7)Lf=1N∑i=1NFGyi−Fxi2,where F· represents the feature mapping matrix extracted from the pretrained VGG network [35]. ·2represents the L2 norm. Using Equation (7), we can extract the image features and restore the image details by comparing feature data between the generated image Gyi and the ground truth image xi.The pixel loss (pixel-to-pixel Euclidean distance) function (Lp) is defined as(8)Lp=1N∑i=1NGyi−xi2+λ∇Gyi2,where ∇Gyi2 is the regularization item, with λ representing the regularization weight. Using Equation (8), we can correctly restore the image information by comparing each pixel between the generated image Gyi and the ground truth image xi.We combineLa, Lf, and Lp together with appropriate weights to form the final loss function for the generator, which is defined as(9)LG=αLa+βLf+γLp,where α, β, and γ represent the predefined positive weights according to the empirical values [35].According to Equation (1), the discriminator D uses the following equation to update the parameters:(10)LD=maxD1N∑i=1NlogDxi+log1−DGyi.For the ground truth imagexi, the probability of the output Dxi is close to 1. For the generated image Gyi, the probability of the output DGyi is close to 0.Based on the above strategy, the generatorG and the discriminator D will be alternately optimized.Based on the above introduction, we give the whole pipeline in Figure6 to clearly describe the proposed method. The real scenery is captured by the camera and converted to the CFA image. The obtained CFA image is then further converted to the 4-channeled compressed CFA images, which is inserted into the generator designed by the U-net model. The output from the generator and the ground truth image are then inputted into the discriminator that is designed by the dense residual network. The generator will finally give near-real demosaiced image through the network training.Figure 6
The pipeline of the proposed method.


## 5. Experiments

In this section, we demonstrate the performance of the proposed network with numerical experiments. The network training is carried out under the TensorFlow environment, which is installed on a PC with Nvidia GeForce® MX250 GPU and Intel Core i5-8265U CPU. The training sets are created beforehand and then uploaded into TensorFlow.


### 5.1. Training Details

The training database used in this paper is from the Waterloo Exploration database (WED) [37], which contains 4744 pristine natural images. We first randomly selected 400 images to create the training set. For the training set, we used data augmentation operations such as cropping and rotations to increase the number of images. To be more specific, we first scaled down each selected image by 1, 0.9, 0.8, and 0.7 times and then used a sliding window to crop the scaled images into patches with a size of 40×40 pixels. The sliding step-lengths in the horizontal and vertical directions are both 20 pixels. Subsequently, the obtained patches are sequentially vertically and horizontally flipped and rotated 90°, 180°, and 270°, respectively, as shown in Figure 7. Through the above data augmentation operations, we obtained 86400 training images. These images are input in batches during the network training process to reduce the calculation and avoid local extreme value problems.Figure 7
The operation of data augmentation.During the training, the weighting parameters in the loss functions are set asα=0.5, β=1, γ=1, and λ=10−5 according to the empirical values [35]. The batch size is set as 256 and there are 200 iterations for the whole network training. During the training, we used a variable learning rate, where the initial learning rate is set as 0.01 and the value is reduced by 1/10 every 40 iterations. The trained network is tested with the Kodak database and the McMaster database [38]. The Kodak database consists of 24 images with the size of 768×512 pixels. The McMaster database consists of 18 images with the size of 500×500 pixels.In order to quantitatively evaluate the performance of the proposed network, we used color peak signal to noise ratio (CPSNR) and structural similarity index (SSIM) as measurement standards for the demosaicing results. The CPSNR value is calculated as(11)CPSNR=10log25521/3∗H∗W∑t3∑h=0H−1∑w=0W−1yh,w,t−xh,w,t2,where xh,w,t and yh,w,t represent the h,w pixel value of the ground truth image x and the demosaiced image y for the t-th color channel, respectively. H and W represent the height size and width size of the image.The SSIM measures the similarity between two images, which is defined as(12)SSIMx,y=2μxμy+C1σxy+C2μx2+μy2+C1σxσy+C2,where μx and σx represent the mean intensity and the standard deviation of the ground truth image x. μy and σy represent the mean intensity and the standard deviation of the demosaiced image y; σxy is the covariance between x and y. C1 and C2 are two constants used to keep the equation balanced and stable, which are usually set as C1=K1∗K^2 and C2=K2∗K^2, with K1=0.01, K2=0.03, and K=255.


### 5.2. Image Demosaicing Test

In this section, we prove the effectiveness of the proposed method by comparing different demosaicing methods. The methods used for comparison are the Bilinear [22], TL [25], HQL [23], Zhang’s [21], LDI-NAT [27], ARI [29], MLRI [28], RI-GBTF [30], and DnCNN [31] methods, as well as the proposed method.Table2 shows the CPSNR and SSIM of the test results for the Kodak database from different methods. Figure 8 shows the corresponding box plots of the CPSNR and SSIM for easier comparison. Table 3 shows the CPSNR and SSIM of the test results for the McMaster database from different methods. Similarly, Figure 9 shows the corresponding box plots. We can see that the proposed method shows higher CPSNR and lower SSIM, which means better performance compared with other methods.Table 2
CPSNR and SSIM of the test results for the Kodak database by different methods.
No.12345678CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear33.550.806038.660.898439.590.933338.560.911834.220.877734.940.844339.160.953833.190.8243TL32.980.808136.910.856438.640.920137.160.878833.760.850034.140.836437.060.928832.750.8323HQL36.020.950541.390.960243.130.976142.320.970937.300.970137.400.956642.470.982735.170.9502Zhang’s38.790.988140.460.975441.450.986140.110.979337.050.985940.230.988640.250.987735.990.9865LDI-NAT33.690.963339.010.968340.400.980239.020.971834.770.979734.900.969340.930.986932.010.9676ARI38.840.986339.690.970442.750.987440.660.090438.290.988740.550.988542.740.988834.990.9801MLRI36.860.981841.490.980742.950.988341.990.984237.680.988538.990.986542.790.990435.170.9825RI-GBTF39.030.988040.080.975341.510.987040.490.980337.810.987840.890.989940.930.986932.010.9676DnCNN41.960.987443.730.973242.180.986943.260.986737.770.984940.260.987342.080.987336.490.9738Proposed42.070.988243.830.975742.220.987643.930.988039.490.989741.770.990143.530.989436.880.9869No.910111213141516CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear38.580.919838.820.919136.020.868038.890.906332.510.769035.540.870938.760.917436.880.8748TL38.180.915638.070.908335.340.854537.800.897532.320.786734.580.846737.970.905835.800.8601HQL41.590.970242.410.972238.750.961142.410.969735.030.942038.540.961641.740.965939.870.9629Zhang’s42.270.984842.040.985839.850.986842.830.986835.010.982135.520.980639.120.975843.730.9900LDI-NAT40.110.974439.810.977836.130.972840.720.977729.640.946135.410.971837.580.967138.560.9746ARI41.960.971441.670.981939.670.984343.250.986935.220.980437.570.983638.640.971543.330.9892MLRI42.360.981742.440.984739.360.986343.370.986833.150.973237.750.983439.850.977742.840.9885RI-GBTF42.540.984842.610.986640.240.988043.390.987635.320.982436.310.981738.900.973844.380.9901DnCNN42.760.985740.040.983642.320.987542.150.987036.940.983739.500.980340.950.972544.650.9913Proposed43.430.988343.600.987142.660.990143.990.989437.030.986140.020.984542.360.977944.960.9904No.1718192021222324CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear38.580.924435.550.873036.010.875038.530.919835.810.887836.650.881641.320.955135.530.8774TL37.820.913634.840.859535.460.879838.040.913235.240.872036.010.877940.030.939935.240.8790HQL42.170.976038.500.961238.200.961941.270.965138.420.964037.780.960644.460.979338.540.9663Zhang’s41.670.988237.130.979440.680.985640.940.976939.120.982437.960.974141.990.982035.380.9853LDI-NAT38.310.978933.890.965337.220.969238.310.967235.220.968637.110.971941.800.981131.770.9731ARI41.320.986836.910.975340.650.982140.890.976039.280.974838.080.972843.240.981435.360.9849MLRI40.690.986536.550.978240.030.982340.710.977038.170.980238.550.977243.670.985734.640.9842RI-GBTF41.750.989037.380.980541.120.986041.210.978039.630.983338.360.976341.960.981635.010.9858DnCNN42.180.984938.960.981542.740.984740.550.987340.080.982539.790.980544.670.984339.130.9867Proposed42.250.989139.470.998342.960.992642.860.987740.870.989440.370.986244.980.991640.070.9954Figure 8
Box plots of (a) CPSNR and (b) SSIM for the Kodak database by different methods. The middle red line in each box represents the median value. The red plus symbol represents the outlier.
(a)(b)Table 3
CPSNR and SSIM of the test results for the McMaster database by different methods.
No.123456789CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear29.580.893933.390.925228.990.916931.550.956434.650.937437.830.959931.780.888233.100.948634.760.9498TL33.670.826135.720.874033.960.875335.960.942235.910.871536.290.888734.810.866237.390.925437.030.9047HQL34.100.873337.330.917235.720.950137.930.978437.660.923938.230.938938.220.953739.730.960738.760.9373Zhang’s25.970.848432.890.913531.990.950333.760.981129.890.899931.850.900038.710.973436.730.961033.140.9237LDI-NAT27.470.876033.980.927131.660.954234.860.984232.580.928535.610.942035.140.954636.650.965635.510.9449ARI29.630.924335.230.945534.650.972837.920.989435.440.962538.790.962639.750.979239.550.978437.900.9630MLRI28.980.913535.060.941733.850.969737.660.989333.940.953338.290.966037.500.970537.050.971036.490.9588RI-GBTF27.440.886933.640.923532.710.960334.850.984331.430.932234.470.943139.110.975436.360.964034.100.9429DnCNN29.390.923636.660.935835.860.967337.740.992136.280.957937.950.964239.300.973538.450.969138.510.9582Proposed34.290.936737.670.948635.880.973838.060.998037.700.963538.930.969339.730.979839.870.980439.740.9679No.101112131415161718CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear36.860.954336.950.949534.620.952039.040.960637.550.948737.660.944731.450.927834.750.937531.460.9098TL37.100.896038.440.920637.400.902139.640.915338.790.914739.100.917234.790.879934.320.872735.560.8756HQL40.170.953940.510.945140.060.951740.040.943440.790.942640.770.936036.330.905136.120.896337.320.9361Zhang’s35.010.940235.910.927335.580.944637.730.938136.490.936836.360.925729.030.867527.650.832433.240.9379LDI-NAT37.240.956037.570.942737.330.954739.720.948837.760.947137.830.941431.010.891330.810.903133.710.9397ARI39.280.972340.230.973040.070.964740.640.950739.050.957539.410.959335.640.967334.710.961636.430.9666MLRI38.650.968839.980.971439.650.963640.630.953138.810.955238.920.952835.160.963232.580.941536.120.9649RI-GBTF36.270.955237.850.961038.130.957938.760.944837.270.943136.650.930932.190.944229.060.939235.280.9590DnCNN39.040.967839.470.974640.030.961439.870.952539.360.957940.180.948735.940.960333.890.958937.210.9631Proposed40.350.975540.620.978740.110.967540.710.961640.950.967040.930.960336.820.967934.640.967737.650.9690Figure 9
Box plots of (a) CPSNR and (b) SSIM for the McMaster database by different methods. The middle red line in each box represents the median value. The red plus symbol represents the outlier.
(a)(b)For additional comparison, Figure10 shows the reconstructed images using different methods for the 19th image in the Kodak database. The marked portion of the image within the black box (the fence) is enlarged for clearer comparison. It can be seen that this part has obvious vertical textures and it is prone to artifacts. Residual images (i.e., difference between the ground truth image and the demosaiced images) for the enlarged portion are also shown for easier comparison. From the reconstructed images and the residual images, we can see that, compared with other methods, the proposed method can more effectively suppress the artifact phenomenon, especially for some tiny edges and angle areas.Figure 10
Comparison of reconstructed images from different demosaicing methods for the 19th image in the Kodak database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figure11 shows the reconstructed images using different methods for the 22th image in the Kodak database. The marked portion in the black box (the window) is enlarged. It can be seen that this part is prone to appearing color stripes and zipperings. The residual images are also shown for easier comparison. From the reconstructed images and the residual images, we can see that most methods can obtain satisfactory results for the smooth areas, while there may appear some wrong colors at the edges. From the comparison, we can see that the results of the proposed method show relatively fewer artifacts and color stripes.Figure 11
Comparison of reconstructed images from different demosaicing methods for the 22th image in the Kodak database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figures12 and 13 show the reconstructed images using different methods for the 1st and 12th images in the McMaster database, respectively. Similarly, the marked portions in the images are enlarged and the residual images for the enlarged portion are shown for clear comparisons. It can be seen that, compared with other methods, the proposed method can better recover the images with fewer artifacts, especially at some tiny edges, which proved the validity and performance of the proposed method.Figure 12
Comparison of reconstructed images from different demosaicing methods for the 1st image in the McMaster database. (a) Ground truth image; (b) enlarged portion in the black box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figure 13
Comparison of reconstructed images from different demosaicing methods for the 12th image in the McMaster database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).


## 5.1. Training Details

The training database used in this paper is from the Waterloo Exploration database (WED) [37], which contains 4744 pristine natural images. We first randomly selected 400 images to create the training set. For the training set, we used data augmentation operations such as cropping and rotations to increase the number of images. To be more specific, we first scaled down each selected image by 1, 0.9, 0.8, and 0.7 times and then used a sliding window to crop the scaled images into patches with a size of 40×40 pixels. The sliding step-lengths in the horizontal and vertical directions are both 20 pixels. Subsequently, the obtained patches are sequentially vertically and horizontally flipped and rotated 90°, 180°, and 270°, respectively, as shown in Figure 7. Through the above data augmentation operations, we obtained 86400 training images. These images are input in batches during the network training process to reduce the calculation and avoid local extreme value problems.Figure 7
The operation of data augmentation.During the training, the weighting parameters in the loss functions are set asα=0.5, β=1, γ=1, and λ=10−5 according to the empirical values [35]. The batch size is set as 256 and there are 200 iterations for the whole network training. During the training, we used a variable learning rate, where the initial learning rate is set as 0.01 and the value is reduced by 1/10 every 40 iterations. The trained network is tested with the Kodak database and the McMaster database [38]. The Kodak database consists of 24 images with the size of 768×512 pixels. The McMaster database consists of 18 images with the size of 500×500 pixels.In order to quantitatively evaluate the performance of the proposed network, we used color peak signal to noise ratio (CPSNR) and structural similarity index (SSIM) as measurement standards for the demosaicing results. The CPSNR value is calculated as(11)CPSNR=10log25521/3∗H∗W∑t3∑h=0H−1∑w=0W−1yh,w,t−xh,w,t2,where xh,w,t and yh,w,t represent the h,w pixel value of the ground truth image x and the demosaiced image y for the t-th color channel, respectively. H and W represent the height size and width size of the image.The SSIM measures the similarity between two images, which is defined as(12)SSIMx,y=2μxμy+C1σxy+C2μx2+μy2+C1σxσy+C2,where μx and σx represent the mean intensity and the standard deviation of the ground truth image x. μy and σy represent the mean intensity and the standard deviation of the demosaiced image y; σxy is the covariance between x and y. C1 and C2 are two constants used to keep the equation balanced and stable, which are usually set as C1=K1∗K^2 and C2=K2∗K^2, with K1=0.01, K2=0.03, and K=255.


## 5.2. Image Demosaicing Test

In this section, we prove the effectiveness of the proposed method by comparing different demosaicing methods. The methods used for comparison are the Bilinear [22], TL [25], HQL [23], Zhang’s [21], LDI-NAT [27], ARI [29], MLRI [28], RI-GBTF [30], and DnCNN [31] methods, as well as the proposed method.Table2 shows the CPSNR and SSIM of the test results for the Kodak database from different methods. Figure 8 shows the corresponding box plots of the CPSNR and SSIM for easier comparison. Table 3 shows the CPSNR and SSIM of the test results for the McMaster database from different methods. Similarly, Figure 9 shows the corresponding box plots. We can see that the proposed method shows higher CPSNR and lower SSIM, which means better performance compared with other methods.Table 2
CPSNR and SSIM of the test results for the Kodak database by different methods.
No.12345678CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear33.550.806038.660.898439.590.933338.560.911834.220.877734.940.844339.160.953833.190.8243TL32.980.808136.910.856438.640.920137.160.878833.760.850034.140.836437.060.928832.750.8323HQL36.020.950541.390.960243.130.976142.320.970937.300.970137.400.956642.470.982735.170.9502Zhang’s38.790.988140.460.975441.450.986140.110.979337.050.985940.230.988640.250.987735.990.9865LDI-NAT33.690.963339.010.968340.400.980239.020.971834.770.979734.900.969340.930.986932.010.9676ARI38.840.986339.690.970442.750.987440.660.090438.290.988740.550.988542.740.988834.990.9801MLRI36.860.981841.490.980742.950.988341.990.984237.680.988538.990.986542.790.990435.170.9825RI-GBTF39.030.988040.080.975341.510.987040.490.980337.810.987840.890.989940.930.986932.010.9676DnCNN41.960.987443.730.973242.180.986943.260.986737.770.984940.260.987342.080.987336.490.9738Proposed42.070.988243.830.975742.220.987643.930.988039.490.989741.770.990143.530.989436.880.9869No.910111213141516CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear38.580.919838.820.919136.020.868038.890.906332.510.769035.540.870938.760.917436.880.8748TL38.180.915638.070.908335.340.854537.800.897532.320.786734.580.846737.970.905835.800.8601HQL41.590.970242.410.972238.750.961142.410.969735.030.942038.540.961641.740.965939.870.9629Zhang’s42.270.984842.040.985839.850.986842.830.986835.010.982135.520.980639.120.975843.730.9900LDI-NAT40.110.974439.810.977836.130.972840.720.977729.640.946135.410.971837.580.967138.560.9746ARI41.960.971441.670.981939.670.984343.250.986935.220.980437.570.983638.640.971543.330.9892MLRI42.360.981742.440.984739.360.986343.370.986833.150.973237.750.983439.850.977742.840.9885RI-GBTF42.540.984842.610.986640.240.988043.390.987635.320.982436.310.981738.900.973844.380.9901DnCNN42.760.985740.040.983642.320.987542.150.987036.940.983739.500.980340.950.972544.650.9913Proposed43.430.988343.600.987142.660.990143.990.989437.030.986140.020.984542.360.977944.960.9904No.1718192021222324CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear38.580.924435.550.873036.010.875038.530.919835.810.887836.650.881641.320.955135.530.8774TL37.820.913634.840.859535.460.879838.040.913235.240.872036.010.877940.030.939935.240.8790HQL42.170.976038.500.961238.200.961941.270.965138.420.964037.780.960644.460.979338.540.9663Zhang’s41.670.988237.130.979440.680.985640.940.976939.120.982437.960.974141.990.982035.380.9853LDI-NAT38.310.978933.890.965337.220.969238.310.967235.220.968637.110.971941.800.981131.770.9731ARI41.320.986836.910.975340.650.982140.890.976039.280.974838.080.972843.240.981435.360.9849MLRI40.690.986536.550.978240.030.982340.710.977038.170.980238.550.977243.670.985734.640.9842RI-GBTF41.750.989037.380.980541.120.986041.210.978039.630.983338.360.976341.960.981635.010.9858DnCNN42.180.984938.960.981542.740.984740.550.987340.080.982539.790.980544.670.984339.130.9867Proposed42.250.989139.470.998342.960.992642.860.987740.870.989440.370.986244.980.991640.070.9954Figure 8
Box plots of (a) CPSNR and (b) SSIM for the Kodak database by different methods. The middle red line in each box represents the median value. The red plus symbol represents the outlier.
(a)(b)Table 3
CPSNR and SSIM of the test results for the McMaster database by different methods.
No.123456789CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear29.580.893933.390.925228.990.916931.550.956434.650.937437.830.959931.780.888233.100.948634.760.9498TL33.670.826135.720.874033.960.875335.960.942235.910.871536.290.888734.810.866237.390.925437.030.9047HQL34.100.873337.330.917235.720.950137.930.978437.660.923938.230.938938.220.953739.730.960738.760.9373Zhang’s25.970.848432.890.913531.990.950333.760.981129.890.899931.850.900038.710.973436.730.961033.140.9237LDI-NAT27.470.876033.980.927131.660.954234.860.984232.580.928535.610.942035.140.954636.650.965635.510.9449ARI29.630.924335.230.945534.650.972837.920.989435.440.962538.790.962639.750.979239.550.978437.900.9630MLRI28.980.913535.060.941733.850.969737.660.989333.940.953338.290.966037.500.970537.050.971036.490.9588RI-GBTF27.440.886933.640.923532.710.960334.850.984331.430.932234.470.943139.110.975436.360.964034.100.9429DnCNN29.390.923636.660.935835.860.967337.740.992136.280.957937.950.964239.300.973538.450.969138.510.9582Proposed34.290.936737.670.948635.880.973838.060.998037.700.963538.930.969339.730.979839.870.980439.740.9679No.101112131415161718CPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMCPSNRSSIMBilinear36.860.954336.950.949534.620.952039.040.960637.550.948737.660.944731.450.927834.750.937531.460.9098TL37.100.896038.440.920637.400.902139.640.915338.790.914739.100.917234.790.879934.320.872735.560.8756HQL40.170.953940.510.945140.060.951740.040.943440.790.942640.770.936036.330.905136.120.896337.320.9361Zhang’s35.010.940235.910.927335.580.944637.730.938136.490.936836.360.925729.030.867527.650.832433.240.9379LDI-NAT37.240.956037.570.942737.330.954739.720.948837.760.947137.830.941431.010.891330.810.903133.710.9397ARI39.280.972340.230.973040.070.964740.640.950739.050.957539.410.959335.640.967334.710.961636.430.9666MLRI38.650.968839.980.971439.650.963640.630.953138.810.955238.920.952835.160.963232.580.941536.120.9649RI-GBTF36.270.955237.850.961038.130.957938.760.944837.270.943136.650.930932.190.944229.060.939235.280.9590DnCNN39.040.967839.470.974640.030.961439.870.952539.360.957940.180.948735.940.960333.890.958937.210.9631Proposed40.350.975540.620.978740.110.967540.710.961640.950.967040.930.960336.820.967934.640.967737.650.9690Figure 9
Box plots of (a) CPSNR and (b) SSIM for the McMaster database by different methods. The middle red line in each box represents the median value. The red plus symbol represents the outlier.
(a)(b)For additional comparison, Figure10 shows the reconstructed images using different methods for the 19th image in the Kodak database. The marked portion of the image within the black box (the fence) is enlarged for clearer comparison. It can be seen that this part has obvious vertical textures and it is prone to artifacts. Residual images (i.e., difference between the ground truth image and the demosaiced images) for the enlarged portion are also shown for easier comparison. From the reconstructed images and the residual images, we can see that, compared with other methods, the proposed method can more effectively suppress the artifact phenomenon, especially for some tiny edges and angle areas.Figure 10
Comparison of reconstructed images from different demosaicing methods for the 19th image in the Kodak database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figure11 shows the reconstructed images using different methods for the 22th image in the Kodak database. The marked portion in the black box (the window) is enlarged. It can be seen that this part is prone to appearing color stripes and zipperings. The residual images are also shown for easier comparison. From the reconstructed images and the residual images, we can see that most methods can obtain satisfactory results for the smooth areas, while there may appear some wrong colors at the edges. From the comparison, we can see that the results of the proposed method show relatively fewer artifacts and color stripes.Figure 11
Comparison of reconstructed images from different demosaicing methods for the 22th image in the Kodak database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figures12 and 13 show the reconstructed images using different methods for the 1st and 12th images in the McMaster database, respectively. Similarly, the marked portions in the images are enlarged and the residual images for the enlarged portion are shown for clear comparisons. It can be seen that, compared with other methods, the proposed method can better recover the images with fewer artifacts, especially at some tiny edges, which proved the validity and performance of the proposed method.Figure 12
Comparison of reconstructed images from different demosaicing methods for the 1st image in the McMaster database. (a) Ground truth image; (b) enlarged portion in the black box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).Figure 13
Comparison of reconstructed images from different demosaicing methods for the 12th image in the McMaster database. (a) Ground truth image; (b) enlarged portion in the green box; (c–l) the results by using Bilinear, TL, HQL, Zhang’s, LDI-NAT, ARI, MLRI, RI-GBTF, and DnCNN methods, as well as the proposed method, respectively; (m–v) the residual images corresponding to (c–l).


## 6. Discussion

In this work, we proposed a new method for image demosaicing based on GAN, which aims to more effectively reconstruct the full-color image. One of the challenges of this task is the recovery of the high-frequency information in the image, such as edges and angles. Many related algorithms have strong ability to process the smooth part of the image; however artifacts, zippering, and strip colors still exist in the high-frequency part. In the current work, we redesigned the generator and discriminator of GAN and combined the adversarial loss, the feature loss, and the pixel loss to further improve the network performance. Numerical experiments showed that the proposed algorithm can effectively reduce the artifacts at the edges and produce near-real reconstructed images, which can be the basis for subsequent image processing, such as image recognition and image transmission. The proposed method can produce better recovered color images; however, the learning-based strategy is relatively time-consuming in the data training phase. Therefore, how to improve the efficiency of the network training is an important aspect to further enhance the performance of the learning-based technology. In practice, there are many kinds of CFA patterns. We used the Bayer pattern in this paper. Different patterns of the CFA image may have different impact on the reconstructed image, so we will use CFA images with different designs to test the network in the near future. In the current work, we assumed the CFA images are noiseless. However, the images from cameras in practice may have been affected by noises. Therefore, we will try combining image demosaicing and denoising in the future. The current work focuses on directly generating the demosaiced images using neural network. We will test on combining traditional demosaicing algorithms and the neural network in the future.


## 7. Conclusions

In this paper, we proposed an image demosaicing method based on GAN. The generator is designed by using the improved U-net architecture to directly generate the demosaicing images. For the discriminator, we used the dense residual network including dense residual blocks with long jump connections and dense connections to overcome the problem of gradient disappearance and gradient dispersion during the network training, which can improve the discriminant ability of the network. In addition, we combined the adversarial loss, the pixel loss, and the feature loss together to improve the loss function. The network was trained using images from the Waterloo Exploration database and the trained network was tested with the Kodak database and the McMaster database. Comparisons among different image demosaicing methods showed that the proposed method can better eliminate artifacts in the reconstructed image and can especially better restore high-frequency features, such as edges and angles of the image.



---

*Source: 7367608-2020-06-16.xml*