Unfolding Taylor's Approximations for Image Restoration

1 Introduction . Image restoration has long been an important task in computer vision , which demands a delicate balance between spatial fine-grained details and high-level contextualized information while recovering a latent clear image from a given degraded observation . It is a highly ill-posed issue as there exists infinite feasible results for single degraded image . Representative image restoration tasks include image deraining , image deblurring . Much research efforts have been devoted to solve the single image restoration problem , which can be categorized into two groups : traditional optimization methods [ 16 , 36 ] and deep learning based methods . In detail , various natural images priors have been developed in traditional image restoration methods to regularize the solution space of the latent clear image , e.g. , low-rank prior [ 36 , 42 ] , dark channel prior [ 31 , 32 , 54 ] , graph-based prior [ 28 , 2 ] , total variation regularization [ 5 , 8 , 1 ] and sparse image priors [ 29 , 56 ] . However , these priors are difficult to design and the methods are difficult to optimize , limiting their practical usage . Convolutional neural networks ( CNNs ) have been applied in numerous high-level computer vision problems and obtained promising improvement in image restoration tasks over traditional methods [ 52 , 63 , 15 , 26 , 45 , 25 , 7 , 66 , 67 , 10 , 37 ] . However , most existing CNN-based image restoration methods empirically construct encapsulated end-to-end mapping networks without deepening into the rationality , and neglect the intrinsic properties of the degradation process . Specifically , it makes ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods unable to balance effectively the two competing goals of spatial fine-grained details and high-level contextualized information while recovering images from the degraded inputs , limiting the model performance . Moreover , existing forward-pass mapping methods lack of sufficient interpretability , making it hard to analyze the effect of each module and preventing further improvement . To solve the aforementioned problems , in this paper , we first revisit the connection between Taylor ’ s Approximation and image restoration task , and propose to unfold Taylor ’ s Formula as blueprints to construct a novel framework that enforces each process part to coincide with the intrinsic properties of image restoration task . We figure out that the main part and derivative part of Taylor ’ s Approximation take the same effect as the above two competing goals of image restoration respectively . In this way , our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process . Specifically , we break down the overall recovery process into two manageable steps , corresponding to two specific operation steps that are responsible for the mapping and derivative functions respectively . The former first learns the highlevel contextualized information and the later combines it with the degraded input to progressively recover local high-order spatial details . Moreover , our proposed framework is orthogonal to existing methods and thus can be easily integrated with them for further performance gain . The conducted extensive experiments demonstrate the effectiveness and scalability of our proposed framework over two popular image restoration tasks , image deraining and image deblurring . The contributions of this paper can be summarized as follows : 1 ) We introduce a new perspective for designing image restoration framework by unfolding the Taylor ’ s Formula . To the best of our knowledge , this is the first effort to solve image restoration task inspired by Taylor ’ s Approximations . 2 ) In contrast to existing methods that optimally balance the competing goals of high-level contextualized information and local high-order spatial details of image restoration respectively in the overall recovery process , we break down it into two manageable steps , corresponding to two specific steps that are responsible for the mapping and derivative functions respectively . 3 ) Our proposed framework ( i.e. , Deep Taylor ’ s Approximations Framework ) is orthogonal to existing CNN-based methods and these methods can be easily integrated with our framework directly for further improvement . Extensive experiments conducted on standard benchmarks demonstrate the effectiveness of the framework . 2 Related Work . Image restoration aims to recover the latent clean images from its degraded observation , which is beneficial for several downstream high-level computer vision tasks , e.g. , object detection , image classification and image segmentation . It is a highly ill-posed problem as there exists infinite feasible results for a single degraded image . To this end , it attracts more attention over the whole community . Representative tasks of different degradation include image deraining , image deblurring . Image deraining aims to remove the rain streaks while maintaining its image textures , and has gained much advances owing to the breakthrough of deep learning [ 12 , 59 , 60 , 13 , 48 , 27 , 28 , 35 , 47 , 9 , 68 ] . Fu et al . [ 11 ] early attempt to design a CNN-based architecture for image deraining and obtain favorable performance with a larger margin than classic promising methods such as guided filter , nonlocal means filtering and more [ 50 , 3 , 22 , 34 ] . Yang et al . [ 55 ] introduce a multi-task network with a series of contextualized dilated convolution and recurrent framework to achieve joint detection and removal of rain streaks . Ren et al . [ 35 ] present the PreNet as a simple baseline for image deraining that combines Resblocks and recurrent layers in a progressive and recursive manner . Overall , the state-of-the-art deep learning methods rely on designing complex network structure . However , these complicated network architectures make them lack of evident interpretability and still have room for further improvement . Image deblurring is a typical ill-posed problem which aims at generating a sharp latent image from a blurry observation . Early Bayesian-based iterative deblurring methods include the Wiener filter [ 49 ] and the Richardson-Lucy algorithm [ 38 ] . Later works commonly rely on developing effective image priors [ 24 , 40 , 51 , 69 ] or sophisticated data terms [ 6 ] . Recently , several CNN-based methods have been proposed for image deblurring [ 17 , 25 , 30 , 63 , 26 , 45 , 41 , 15 , 64 , 33 , 61 , 43 , 37 ] . For example , Sun et al . [ 44 ] propose a CNN-based model to estimate a kernel and remove non-uniform motion blur . Chakrabarti [ 4 ] uses a network to compute estimations of sharp images that are blurred by an unknown motion kernel . Nah et al . [ 30 ] propose a multi-scale loss function to apply a coarse-to-fine strategy . Kupyn et al . propose DeblurGAN [ 25 ] and DeblurGAN-v2 [ 26 ] to remove blur based on adversarial learning . Despite of the encouraging performance achieved by CNN-based methods for image deblurring , they fail to reconstruct sharp results with good interpretability . The Taylor ’ s Approximations is one of the most popular methods in numerical approximation . In previous researches , Tukey [ 46 ] is the first to exploit the Taylor ’ s series expansion to study error propagation . Afterwards , a number of studies [ 18 , 20 , 21 , 23 , 19 ] have been made and based on the first-order Taylor ’ s expansion . It is , after all , a liner approximations that results in the inaccurate values . Xue et al . [ 53 ] develop above methods by high-order Taylor series expansion , which further improve the performance . However , it has never been explored for image restoration . In this paper , we revisit the connection between the Taylor ’ s Approximations and image restoration process for the first time , and propose to unfold the Taylor ’ s Formula to construct a novel framework , named Deep Taylor ’ s Approximations Framework . 3 Proposed Method . 3.1 Revisiting Taylor ’ s Approximations and Image Restoration . Problem Formulation Image degradation is the most common and inevitable phenomena in imaging systems . It is commonly formulated as y = Ax+N , ( 1 ) where y , x , A and N denote a degraded observation , a latent clear image , a degraded matrix and noise respectively . Specifically , whenA is the identity matrix andN is the rain streak , it transforms as image deraining task . Moreover , when A is the blurry matrix and N is the additional noise , it turns to image deblurring task . Observing Equation ( 1 ) , for the inverse process , it can be reformulated as y0 = Ax = ( y −N ) , ( 2 ) where we denote the termAx as y0 , representing the latent image without additional noise . In detail , N is rain streak R over image deraining task while acting as the additional noise in image deblurring . Existing Methods Most of the existing CNN-based image restoration methods empirically construct encapsulated end-to-end mapping networks which neglect the intrinsic prior knowledge of image restoration task . Specifically , it acts as x = F ( y ) , ( 3 ) where F is the existing-designed mapping network , which empirically learns the mapping between the degraded and clean pairs x and y . Ours : Revisiting Image Restoration Task Different from existing methods , we try to associate Taylor ’ s Approximations with image restoration , which has not been considered in previous methods . Referring to Equation ( 2 ) , we define the inverse process using the function F , and then we can obtain the relationship as x = F ( y0 ) = F ( y −N ) . ( 4 ) It can be figured out that the mapping function F in our framework corresponds the inverse operation of degradation matrixA . Deviating from most existing CNN-based methods , our framework considers the intrinsic prior knowledge of image restoration task and has better interpretability . The inverse operation of degradation matrixA is denoted asA− F = A− . ( 5 ) Let −N = = y0 − y , we expand Equation ( 4 ) with an infinite-order Taylor ’ s series expansion of a scalar-valued of more than one variable , written compactly as x = F ( y0 ) = F ( y + ) ( 6 ) = F ( y ) + 1 1 ! n∑ i = 1 ∂F ( y ) ∂yi i+ ... + 1 k ! ∑ i1 , ... , ik∈ { 1,2 , ... , n } ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik + ... ( 7 ) = ∞∑ k = 0 1 k ! ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) , ( 8 ) where we use the notation ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) ≡ n∑ i1 = 1 n∑ i2 = 1 ... n∑ ik = 1 ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik ( 9 ) = ∑ i1 , ... , ik∈ { 1,2 , ... , n } ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik . ( 10 ) When only regarding n order Taylor ’ s Approximations , it can be simplified as x = F ( y ) + n∑ k = 1 1 k ! ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) . ( 11 ) It can be separated into two parts for consideration . In detail , the first term , as main part , F ( y ) represents the high-level contextualized information , which gives a constant approximation of clear image ( ground truth ) while the rest is the local high-order spatial details . When we revisit the connection between the goals of image restoration and Equation ( 11 ) , we can clearly find that the main part and derivative part of Taylor ’ s Approximations take the same effect as the two competing goals of high-level contextualized information and spatial details of image restoration respectively . In this way , our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process . Specifically , we break down the overall recovery process into two manageable steps . The former first learns the high-level contextualized structures and the later combines them with the degraded input to progressively recover local high-order spatial details . The details of our proposed framework are illustrated as bellow .

Summary: The author proposes a novel deep neural network, which leverages the intrinsic knowledge of low-level vision and Taylor’s Approximations, for the image restoration tasks. The author found that the main part and derivative part of Taylor’s Approximations takes the internal similarity as the high-level structure and spatial details of image restoration, respectively. From this, the author breaks down the restoration process into two manageable steps, corresponding to the mapping function and derivative function, respectively. The proposed framework is orthogonal and can be easily integrated into existing deep learning-based methods. The idea of using Taylor’s Approximations to guide deep network design for image restoration is quiteThe limitation and potential negative societal impact of their work have been discussed in their paper. interesting.