Unfolding Taylor's Approximations for Image Restoration

1 Introduction . Image restoration has long been an important task in computer vision , which demands a delicate balance between spatial fine-grained details and high-level contextualized information while recovering a latent clear image from a given degraded observation . It is a highly ill-posed issue as there exists infinite feasible results for single degraded image . Representative image restoration tasks include image deraining , image deblurring . Much research efforts have been devoted to solve the single image restoration problem , which can be categorized into two groups : traditional optimization methods [ 16 , 36 ] and deep learning based methods . In detail , various natural images priors have been developed in traditional image restoration methods to regularize the solution space of the latent clear image , e.g. , low-rank prior [ 36 , 42 ] , dark channel prior [ 31 , 32 , 54 ] , graph-based prior [ 28 , 2 ] , total variation regularization [ 5 , 8 , 1 ] and sparse image priors [ 29 , 56 ] . However , these priors are difficult to design and the methods are difficult to optimize , limiting their practical usage . Convolutional neural networks ( CNNs ) have been applied in numerous high-level computer vision problems and obtained promising improvement in image restoration tasks over traditional methods [ 52 , 63 , 15 , 26 , 45 , 25 , 7 , 66 , 67 , 10 , 37 ] . However , most existing CNN-based image restoration methods empirically construct encapsulated end-to-end mapping networks without deepening into the rationality , and neglect the intrinsic properties of the degradation process . Specifically , it makes ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods unable to balance effectively the two competing goals of spatial fine-grained details and high-level contextualized information while recovering images from the degraded inputs , limiting the model performance . Moreover , existing forward-pass mapping methods lack of sufficient interpretability , making it hard to analyze the effect of each module and preventing further improvement . To solve the aforementioned problems , in this paper , we first revisit the connection between Taylor ’ s Approximation and image restoration task , and propose to unfold Taylor ’ s Formula as blueprints to construct a novel framework that enforces each process part to coincide with the intrinsic properties of image restoration task . We figure out that the main part and derivative part of Taylor ’ s Approximation take the same effect as the above two competing goals of image restoration respectively . In this way , our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process . Specifically , we break down the overall recovery process into two manageable steps , corresponding to two specific operation steps that are responsible for the mapping and derivative functions respectively . The former first learns the highlevel contextualized information and the later combines it with the degraded input to progressively recover local high-order spatial details . Moreover , our proposed framework is orthogonal to existing methods and thus can be easily integrated with them for further performance gain . The conducted extensive experiments demonstrate the effectiveness and scalability of our proposed framework over two popular image restoration tasks , image deraining and image deblurring . The contributions of this paper can be summarized as follows : 1 ) We introduce a new perspective for designing image restoration framework by unfolding the Taylor ’ s Formula . To the best of our knowledge , this is the first effort to solve image restoration task inspired by Taylor ’ s Approximations . 2 ) In contrast to existing methods that optimally balance the competing goals of high-level contextualized information and local high-order spatial details of image restoration respectively in the overall recovery process , we break down it into two manageable steps , corresponding to two specific steps that are responsible for the mapping and derivative functions respectively . 3 ) Our proposed framework ( i.e. , Deep Taylor ’ s Approximations Framework ) is orthogonal to existing CNN-based methods and these methods can be easily integrated with our framework directly for further improvement . Extensive experiments conducted on standard benchmarks demonstrate the effectiveness of the framework . 2 Related Work . Image restoration aims to recover the latent clean images from its degraded observation , which is beneficial for several downstream high-level computer vision tasks , e.g. , object detection , image classification and image segmentation . It is a highly ill-posed problem as there exists infinite feasible results for a single degraded image . To this end , it attracts more attention over the whole community . Representative tasks of different degradation include image deraining , image deblurring . Image deraining aims to remove the rain streaks while maintaining its image textures , and has gained much advances owing to the breakthrough of deep learning [ 12 , 59 , 60 , 13 , 48 , 27 , 28 , 35 , 47 , 9 , 68 ] . Fu et al . [ 11 ] early attempt to design a CNN-based architecture for image deraining and obtain favorable performance with a larger margin than classic promising methods such as guided filter , nonlocal means filtering and more [ 50 , 3 , 22 , 34 ] . Yang et al . [ 55 ] introduce a multi-task network with a series of contextualized dilated convolution and recurrent framework to achieve joint detection and removal of rain streaks . Ren et al . [ 35 ] present the PreNet as a simple baseline for image deraining that combines Resblocks and recurrent layers in a progressive and recursive manner . Overall , the state-of-the-art deep learning methods rely on designing complex network structure . However , these complicated network architectures make them lack of evident interpretability and still have room for further improvement . Image deblurring is a typical ill-posed problem which aims at generating a sharp latent image from a blurry observation . Early Bayesian-based iterative deblurring methods include the Wiener filter [ 49 ] and the Richardson-Lucy algorithm [ 38 ] . Later works commonly rely on developing effective image priors [ 24 , 40 , 51 , 69 ] or sophisticated data terms [ 6 ] . Recently , several CNN-based methods have been proposed for image deblurring [ 17 , 25 , 30 , 63 , 26 , 45 , 41 , 15 , 64 , 33 , 61 , 43 , 37 ] . For example , Sun et al . [ 44 ] propose a CNN-based model to estimate a kernel and remove non-uniform motion blur . Chakrabarti [ 4 ] uses a network to compute estimations of sharp images that are blurred by an unknown motion kernel . Nah et al . [ 30 ] propose a multi-scale loss function to apply a coarse-to-fine strategy . Kupyn et al . propose DeblurGAN [ 25 ] and DeblurGAN-v2 [ 26 ] to remove blur based on adversarial learning . Despite of the encouraging performance achieved by CNN-based methods for image deblurring , they fail to reconstruct sharp results with good interpretability . The Taylor ’ s Approximations is one of the most popular methods in numerical approximation . In previous researches , Tukey [ 46 ] is the first to exploit the Taylor ’ s series expansion to study error propagation . Afterwards , a number of studies [ 18 , 20 , 21 , 23 , 19 ] have been made and based on the first-order Taylor ’ s expansion . It is , after all , a liner approximations that results in the inaccurate values . Xue et al . [ 53 ] develop above methods by high-order Taylor series expansion , which further improve the performance . However , it has never been explored for image restoration . In this paper , we revisit the connection between the Taylor ’ s Approximations and image restoration process for the first time , and propose to unfold the Taylor ’ s Formula to construct a novel framework , named Deep Taylor ’ s Approximations Framework . 3 Proposed Method . 3.1 Revisiting Taylor ’ s Approximations and Image Restoration . Problem Formulation Image degradation is the most common and inevitable phenomena in imaging systems . It is commonly formulated as y = Ax+N , ( 1 ) where y , x , A and N denote a degraded observation , a latent clear image , a degraded matrix and noise respectively . Specifically , whenA is the identity matrix andN is the rain streak , it transforms as image deraining task . Moreover , when A is the blurry matrix and N is the additional noise , it turns to image deblurring task . Observing Equation ( 1 ) , for the inverse process , it can be reformulated as y0 = Ax = ( y −N ) , ( 2 ) where we denote the termAx as y0 , representing the latent image without additional noise . In detail , N is rain streak R over image deraining task while acting as the additional noise in image deblurring . Existing Methods Most of the existing CNN-based image restoration methods empirically construct encapsulated end-to-end mapping networks which neglect the intrinsic prior knowledge of image restoration task . Specifically , it acts as x = F ( y ) , ( 3 ) where F is the existing-designed mapping network , which empirically learns the mapping between the degraded and clean pairs x and y . Ours : Revisiting Image Restoration Task Different from existing methods , we try to associate Taylor ’ s Approximations with image restoration , which has not been considered in previous methods . Referring to Equation ( 2 ) , we define the inverse process using the function F , and then we can obtain the relationship as x = F ( y0 ) = F ( y −N ) . ( 4 ) It can be figured out that the mapping function F in our framework corresponds the inverse operation of degradation matrixA . Deviating from most existing CNN-based methods , our framework considers the intrinsic prior knowledge of image restoration task and has better interpretability . The inverse operation of degradation matrixA is denoted asA− F = A− . ( 5 ) Let −N = = y0 − y , we expand Equation ( 4 ) with an infinite-order Taylor ’ s series expansion of a scalar-valued of more than one variable , written compactly as x = F ( y0 ) = F ( y + ) ( 6 ) = F ( y ) + 1 1 ! n∑ i = 1 ∂F ( y ) ∂yi i+ ... + 1 k ! ∑ i1 , ... , ik∈ { 1,2 , ... , n } ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik + ... ( 7 ) = ∞∑ k = 0 1 k ! ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) , ( 8 ) where we use the notation ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) ≡ n∑ i1 = 1 n∑ i2 = 1 ... n∑ ik = 1 ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik ( 9 ) = ∑ i1 , ... , ik∈ { 1,2 , ... , n } ∂kF ( y ) ∂yi1 ... ∂yik i1 ... ik . ( 10 ) When only regarding n order Taylor ’ s Approximations , it can be simplified as x = F ( y ) + n∑ k = 1 1 k ! ( n∑ i = 1 i ∂ ∂yi ) k F ( y ) . ( 11 ) It can be separated into two parts for consideration . In detail , the first term , as main part , F ( y ) represents the high-level contextualized information , which gives a constant approximation of clear image ( ground truth ) while the rest is the local high-order spatial details . When we revisit the connection between the goals of image restoration and Equation ( 11 ) , we can clearly find that the main part and derivative part of Taylor ’ s Approximations take the same effect as the two competing goals of high-level contextualized information and spatial details of image restoration respectively . In this way , our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process . Specifically , we break down the overall recovery process into two manageable steps . The former first learns the high-level contextualized structures and the later combines them with the degraded input to progressively recover local high-order spatial details . The details of our proposed framework are illustrated as bellow .

Summary: This paper aims to develop an image restoration framework through a combination of Taylor’s Approximation and deep learning to break the entire recovery process into two manageable steps. Specifically, the former learns high-level contextualized information, and the latter combines it with the degraded input to progressively recover local high-order spatial details. Several promising methods are integrated with the proposed framework and achieve some improvement. Ablation studies about Taylor’s order are also conducted to provide a comprehensive analysis.