# Demosaicing by Differentiable Deep Restoration
## 1 Introduction  
In digital imaging pipeline, a mosaic of color filter arrays (CFAs) is applied in front of the camera sensor to capture the raw image where each pixel only measures the intensity of one color band. This raw image measurement is referred to as imaging process and typically processed by a demosaicing algorithm to reconstruct a full-color image. The demosaicing process is a challenging ill-posed inverse problem with at least two-thirds of information missing in the raw image. The demosaicing problem can be formulated as an image restoration problem, which aims to recover a full-color image \(I\) from the degraded raw image \(M=HI+N\), where \(H\) is the degradation matrix and \(N\) is the additive noise. In contrast to other image restoration tasks, e.g., deblurring and denoising, where the matrices \(H\) are uncontrollable, the matrix \(H\) for demosaicing depends on the adopted CFA that is designable. Thus, both the design of CFA and demosaicing algorithm are critical for the quality of the final color image and have been studied for decades. A variety of CFA patterns have been proposed and most demosaicing methods are developed accordingly to deal with raw images generated by the most popular CFA, e.g., Bayer pattern CFA [1]. As a limitation, these methods are difficult to generalize to images filtered by other CFAs. Recently, with a great success in many computer vision tasks, convolutional neural networks (CNNs) have been employed to solve the demosaicing problem by directly regressing the color image. This kind of learning-based methods presents impressive performance improvements and flexibility to fit any given CFA. However, since the tight coupling of imaging and demosaicing processes, learning a demosaicing network with a predefined CFA tends to be suboptimal. End-to-end learning of deep neural networks makes it possible to jointly learn the CFA and demosaicing algorithm. The pioneering works [2, 3] proposed joint optimization by characterizing the CFA pattern as learnable parameters and simultaneously learn the CFA and demosaicing network. However, in the inference stage, the demosaicing network only takes the raw image as input to regress the final color image, which does not make full use of the available CFA information. Instead of applying CNNs to directly regress the color image, we design a deep neural network to solve the image restoration problem to find a color image \(I\) satisfying \(M=HI+N\) according to the degradation model. The network learns to constrain the original ill-posed problem for robust solution, while the optimization module enforces the physical degradation model, i.e., the recovered color image \(I\) can produce the raw image \(M\) under the given CFA. In this way, the demosaicing network not only takes the raw image as input, but takes the degradation model with the adopted CFA into account as well. As proved by the studies in other vision tasks [4; 5; 6], this approach of combining machine learning and physically-based optimization often leads to superior results and better data generalization. At the same time, our whole network is differentiable, which also enables joint learning of the CFA and demosaicing network. To this end, we introduce a differentiable deep restoration network for demosaicing. Specifically, we employ a network to predict the subspace where the color image \(I\) should lie in, i.e., generating multiple basis vectors and solving the color image \(I\) as a linear combination of these bases. In this way, the original ill-posed restoration problem \(M=HI+N\) becomes well-posed and can be solved easily in closed-form. This closed-form solver minimizes the re-imaging error, i.e., \(|M-HI|^{2}\). Unlike LSM [4] and BA-Net [7] where the network produces basis vectors for the whole image and solves the output at once, we operate at image patch level to make the network training and generalization easier. Ideally, we can generate the bases for an \(n\times n\) image patch, and solve the result image \(I\) within that patch.