# Demosaicing by Differentiable Deep Restoration
## 3 The Proposed Method
### Model the Imaging Process  
We can model the imaging process as a linear process depending on the color image and the corresponding CFA along the color channels. Most of the existing CFAs are periodic tiling of square pixels with the same size of the image resolution. For a CFA pattern \(P\) with size of \(k\times k\), the CFA is:  
\[H_{c}(i)=P_{c}(i\bmod k),\text{ for }c\in C, \tag{2}\]  
Figure 1: The overall architecture of our deep restoration network. The whole architecture consists of imaging process and demosaicing process. For imaging process, the input image is filtered by the color filter arrays (CFA) to produce the raw image. For demosaicing process, the prediction is reconstructed with basis maps generated from the U-net structure and the corresponding coefficient maps optimized by a differentiable closed-form solver. The whole architecture is trained end-to-end.  
Figure 2: Example of a color image reconstruction from the estimated basis maps and the corresponding optimized coefficient maps. The value of basis maps and the value of coefficient maps have been normalized for visualization.  
where \(i\) is the 2D pixel coordinates with \(x\) and \(y\) coordinates, \(C\) is the set of color channels \(\{R,G,B\}\) and mod indicates the modulus operator which works element-wisely. To guarantee the designed CFA is physically realizable, \(P_{c}\) takes values between \([0,1]\) and satisfies  
\[P_{R}+P_{G}+P_{B}=\mathbf{1}. \tag{3}\]  
Here, the \(\mathbf{1}\) is an all-one matrix, where all matrix elements equals 1. When the CFA is given, the mosaiced image \(M\) obtained from this imaging process can be formulated as  
\[M(i)=\sum_{c}H_{c}(i)\cdot I_{c}(i)+N(i). \tag{4}\]  
Here, \(N\) is the additive noise. For noise-free data in ideal conditions, \(N\) can be ignored in this imaging process. Otherwise, we need to model the additive noise \(N\) to simulate the raw image from color image. The potential influence of float precision of \(H\) and \(N\) is not considered in the simulation process.  
In this imaging process, the choice of the CFA pattern directly affects the quality of the raw image \(M\) we obtained from the sensor. Plenty of CFAs have been proposed in the long time study. The Bayer pattern [1]\(Ba\) is the most popular CFA pattern with \(2\times 2\) local window, where the color is a selection of primary colors of R, G, B for each pixel. According to our formulation, this CFA pattern can be written as,  
\[P_{R}^{Ba}=\begin{bmatrix}1&0\\ 0&0\end{bmatrix},P_{G}^{Ba}=\begin{bmatrix}0&1\\ 1&0\end{bmatrix},P_{B}^{Ba}=\begin{bmatrix}0&0\\ 0&1\end{bmatrix}. \tag{5}\]  
In addition to the primary colors, the color for CFA pattern can also be chosen from other spectral bands. For example, the Bean pattern [41]\(Be\) is designed as,  
\[P_{R}^{Be}=\begin{bmatrix}0&\frac{1}{2}\\ 0&\frac{1}{2}\end{bmatrix},P_{G}^{Be}=\begin{bmatrix}\frac{1}{2}&0\\ 0&\frac{1}{2}\end{bmatrix},P_{B}^{Be}=\begin{bmatrix}\frac{1}{2}&\frac{1}{2}\\ 1&0\end{bmatrix}. \tag{6}\]  
According to Equation (3), the sum of the CFA \(H\) across all channels should be an all-one matrix. Furthermore, all elements in a CFA must be non-negative. These two constraints can be written as  
\[\left\{\begin{array}{cc}\sum_{c}H_{c}=\mathbf{1},\\ H\geq\mathbf{0}.\end{array}\right. \tag{7}\]  
In the proposed method, in which the CFA pattern is learnable and can be optimized jointly with the demosaicing network, a Softmax layer with positive and normalized outputs is employed on the learned parameters to guarantee the generated CFA satisfies the above requirements. As pointed by [3], this kind of CFA which is a linear combination of primary colors can be manufactured by technology described in [42]. One of the CFA pattern learned by the proposed method is shown in Figure 1.