# Demosaicing by Differentiable Deep Restoration
## 3 The Proposed Method
### Differentiable Deep Restoration
#### 3.3.1 Bases Generation Network with Sparse Data  
One challenge in designing the bases generation network is to deal with the raw image \(M\) with missing entries. For example, as shown in Figure 3, in the raw image generated by the Bayer pattern, only one of the four neighboring pixels have red intensity measured. The red intensity is missing at all the other three pixels. This sparsity makes the raw image \(M\) inappropriate for conventional CNNs with weight sharing for different pixels.  
Various preprocessing methods have been proposed to deal with this sparsity. The most commonly used preprocessing operations are rearranging and interpolation. The  
Figure 3: Example of CFA patterns and the corresponding raw images. The top row shows the Bayer pattern, the second row is our learned CFA pattern. The values of a CFA are marked in the CFA cells.  
rearranging operation is adopted in [9] to convert the raw image sampled from a \(2\times 2\) Bayer pattern as the following,  
\[D_{c}(i)=M(k\times i_{x}+c\text{ mod }k,k\times i_{y}+\Big{\lfloor}\frac{c}{k} \Big{\rfloor}). \tag{10}\]  
Basically, it reduces the image resolution and packs all pixels with red, green, or blue intensities to form the R, G, B channels, respectively. This operation is computational efficient and also reduces the computation of the subsequent networks, since it reduces the image resolution. However, when the size of the CFA pattern enlarges, which tends to improve the image quality as reported in [22], this rearranging operation losses spatial information due to the shrinkage of image resolution. In comparison, the interpolation operation is used in paper [35] to preprocess the raw image with hand-crafted interpolation weights. The missing R, G, B values are linearly interpolated from the corresponding channel in a neighborhood, which can be written as  
\[D_{c}(i)=\sum_{o}W_{c}(o)M(i+o)Mask_{c}(i+o), \tag{11}\]  
where \(W\) is the hand-crafted interpolation weights, \(o\) is a coordinate offset within the local window, and \(Mask_{c}\) is a binary mask indicating whether the pixel has values on channel \(c\). However, hand-crafted interpolation weights could be sub-optimal and degrades the demosaicing results. Interpolating across edges may introduce large errors and make the subsequent processing difficult.  
In comparison, we propose a novel pattern sharing convolutional (PSC) layer to allow the network to deal with the sparse image \(M\). Mathematically, our processed data is  
\[D_{c}(i)=\sum_{o}W_{c}^{l}(i\text{ mod }k,o)M(i+o), \tag{12}\]  
where the \(W^{l}\) are parameters learned with the whole network. As the color channels of raw image are periodic, the parameters are shared with the same period \(k\). Basically, this PSC layer learns interpolation parameters instead of the hand-crafted ones in Equation (11). Furthermore, it takes intensity values from all color channels in the neighborhood to interpolate the missing entries.  
Compared with the rearranging operation, our PSC layer can take the advantage of the neighborhood relationship information in local and maintain the resolution of image which makes it suitable for CFA pattern with large size. Compared with the hand-crafted interpolation operation, our PSC layer is a more general way to process the sparse data whose parameters can be learned with the whole network in an end-to-end manner. More comparisons and discussions can be found in detail in Section 4.2.