Unfolding Taylor's Approximations for Image Restoration
Authors: man zhou
Xueyang Fu
Zeyu Xiao
Gang Yang
Aiping Liu
Zhiwei Xiong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Taylor’s Approximations
Image Restoration

Abstract: Deep learning provides a new avenue for image restoration, which demands a delicate balance between fine-grained details and high-level contextualized information during recovering the latent clear image. In practice, however, existing methods empirically construct encapsulated end-to-end mapping networks without deepening into the rationality, and neglect the intrinsic prior knowledge of restoration task. To solve the above problems, inspired by Taylor’s Approximations, we unfold Taylor’s Formula to construct a novel framework for image restoration. We find the main part and the derivative part of Taylor’s Approximations take the same effect as the two competing goals of high-level contextualized information and spatial details of image restoration respectively. Specifically, our framework consists of two steps, which are correspondingly responsible for the mapping and derivative functions. The former first learns the high-level contextualized information and the later combines it with the degraded input to progressively recover local high-order spatial details. Our proposed framework is orthogonal to existing methods and thus can be easily integrated with them for further improvement, and extensive experiments demonstrate the effectiveness and scalability of our proposed framework.

Unfolding Taylor's Approximations for Image Restoration
Man Zhou'
Xueyang Fu 1+ Zeyu Xiao Aiping Liu
Gang Yang' Uniieersity of Science and Technology of China
Zhiwei Xiong
[manman) @mail. ustc. edu. en xyfu@ustc edu, cn
Abstract
Deep learning provides a new avenue for image restoration, which demands a deli- cate balance between fine-grained details and high-level contextualized information during recovering the latent clear image. In practice, however, existing methods empirically construct encapsulated end-to-end mapping networks without deep- ening into the rationality, and neglect the intrinsic prior knowledge of restoration task. To solve the above problems, inspired by Taylor's 's Approximations, we unfold Taylor's Formula to construct novel framework for image restoration. We find the main part and the derivative part of Taylor's Approximations take the same effect as the two competing goals of high-level contextualized information and spatial details of image restoration respectively. Specifically,, our framework consists of two steps, correspondingly responsible for the mapping and derivative functions. The former first learns the high-level contextualized information and the later com- bines it with the degraded input to progressively recover local high-order spatial details. Our proposed framework orthogonal to existing methods and thus can be easily integrated with them for further improvement. and extensive experiments demonstrate the effectiveness and scalability of our proposed framework. Code will be publicly available upon acceptance.
Introduction
Image restoration has long been an important task in computer vision, which demands a delicate bal- ance between spatial fine-grained details and high-level contextualized information while recovering latent clear image from given degraded observation. is a highly ill-posed issue there exists infinite feasible results for single degraded image. Representative image restoration tasks include image deraining, image deblurring.
Much research efforts have been devoted to solve the single image restoration problem, which can be categorized into groups: traditional optimization methods [16] 36) and deep learning based methods. In detail, various natural images priors have been developed in traditional image restoration
methods regularize the solution space of the latent clear image, e.g., low-rank prior 36 42 dark
channel prior [31 32 [4], graph-based prior 28 2]. total variation regularization (5) g and sparse
image priors [29 56]. However, these priors are difficult to design and the methods are difficult to
optimize, limiting their practical usage.
Convolutional neural networks (CNNs) have been applied in numerous high-level computer vision problems and obtained promising improvement in image restoration tasks over traditional meth- 67 10 37). However, most existing CNN-based image restoration methods empirically construct encapsulated end-to-end mapping networks without deepening into the rationality, and neglect the intrinsic properties of the degradation process. Specifically, makes
ods 552 63 15
26
45 25 66,
Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

these methods unable to balance effectively the two competing goals of spatial fine- grained de- tails and high-level contextualized information while recovering images from the degraded inputs. limiting the model performance. Moreover, existing forward- -awarddpasssa mapping methods lack of suffi- cient interpretability, making it hard to analyze the effect of each module and preventing further improvement.
To solve the aforementioned problems, in this paper, we first revisit the connection between Taylor's Approximation and image restoration task, and propose to unfold Taylor's Formula as blueprints to construct novel framework that enforces each process part to coincide with the intrinsic properties of image restoration task. We figure out that the main part and derivative part of Taylor's s Approximation take the same effect as the above two competing goals of image restoration respectively In this way, our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process. Specifically, we break down the overall recovery process into two manageable steps, corresponding to two specific operation steps that are responsible for the mapping and derivative functions respectively. The former first learns the high- level contextualized information and the later combines it with the degraded input to progressively recover local high-order spatial details. Moreover, our proposed framework orthogonal to existing methods and thus can be easily integrated with them for further performance gain. The conducted extensive experiments demonstrate the effectiveness and scalability of our proposed framework over two popular image restoration tasks, image deraining and image deblurring.
The contributions of this paper can be summarized as follows:
1) We introduce new perspective for designing image restoration framework by unfolding the Taylor's 's Formula. To the best of our knowledge, this is the first effort to solve image restoration task inspired by Taylor's Approximations.
2) In contrast to existing methods that optimally balance the competing goals of high-level contextualized information and local high-order spatial details of image restoration re- spectively in the overall recovery process, we break down it into two manageable steps, corresponding to two specific steps that are responsible for the mapping and derivative functions respectively.
3) Our proposed framework (i.e., Deep Taylor's Approximations Framework) is orthogonal to existing CNN-based methods and these methods can be directly for further improvement. Extensive experiments conducted on standard benchmarks demonstrate the effectiveness of the framework.
easily integrated with our framework
2 Related Work
Image restoration aims to recover the latent clean images from its degraded observation, which is beneficial for several downstream high-level computer vision tasks, e.g., object detection, image classification and image segmentation. It is highly ill- ill-posed problem there exists infinite feasible results for a single degraded image. To this end, it attracts more attention over the whole community. Representative tasks of different degradation include image deraining, image deblurring.
Image deraining aims remove the rain streaks while maintaining its image textures, and has gained much advances owing to the breakthrough of deep learning [12 55 60 13]
27
28
35
47
9 68
Fu et al. [T]] early attempt to design a -ase- architecture for image deraining and obtain favorable performance with larger margin than classic promising methods such as guided filter, nonlocal means filtering and more [50 22 22 34 Yang el al. (55) introduce a multi-task network with a series of contextualized dilated convolution and recurrent framework to achieve joint detection and removal of rain streaks. Ren et al. (35) present the PreNet as a simple baseline for image deraining that combines Resblocks and recurrent layers in progressive and recursive manner. Overall, the state-of-ther ofrtte-artttttte- deep learning methods rely on designing complex network structure. However, these complicated network architectures make them lack of evident interpretability and still have room for further improvement.
Image deblurring is a typical ill- possd problem which aims at generating a sharp latent image from a blurry observation. Early Bayesian- based iterative deblurring methods include the Wiener filter [49 and the Richardson-Luce algorithm (88 Later works commonly rely developing effective image
priors 24 40 51 699 or sophisticated data terms 6 Recently, several CNN-based methods have
been proposed for image deblurring [17 25
30
63
26
45
15
64
33
61
43
37]
For example,

---

Sun et al. 44 propose a CNN- based model to estimate a kernel and remove non uniform motion blur. Chakrabarti uses network compute estimations of sharp images that are blurred by an unknown motion kernel. Nah el al. 30 propose multi-scale loss function to apply coarse-to- to-fiie strategy. Kupyn et al. propose DeblurGAN 25 and DeblurGAN-v2 (26] to remove blur based on adversarial learning. Despite of the encouraging performance achieved by CNN-based methods for image deblurring, they fail to reconstruct sharp results with good interpretability.
The Taylor's Approximations is one of the most popular methods in numerical approximation. In previous researches, Tukey [46] is the first to exploit the Taylor's 's series expansion to study error propagation. Afterwards, number of studies [18] 20 23 19] have been made and based on the first-order Taylor's 's expansion. is, after all, a liner approximations that results in the inaccurate values. Xue et al. [53] develop above methods by high-order Taylor series expansion, which further improve the performance However, has never been explored for image restoration. In this paper, we revisit the connection between the Taylor's Approximations and image restoration process for the first time, and propose to unfold the Taylor's Formula to construct a novel framework, named Deep Taylor's s Approximations Framework.
3 Proposed Method
3.1 Revisiting Taylor's Approximations and Image Restoration
Problem Formulation Image degradation is the most common and inevitable phenomena in imaging systems. It commonly formulated as
y= Ac+
(1)
where y. æ. A and N denote degraded observation, latent clear image, a degraded matrix and noise respectively. Specifically, when A is the identity matrix and N is the rain streak, it transforms as image deraining task. Moreover, when is the blurry matrix and N is the additional noise, turns to image deblurring task. Observing Equation ① for the inverse process, it can be reformulated as
Yo = Ax = (y-
(2)
where we denote the term A. as Yo- representing the latent image without additional noise. In detail, N is rain streak R over image deraining task while acting as the additional noise in image deblurring.
Existing Methods Most of the existing CNN-based image restoration methods empirically con- struct encapsulated end-to-end mapping networks which neglect the intrinsic prior knowledge of image restoration task. Specifically, it acts as
x = F(y),
where F is the clisting-designnn mapping network, which empirically learns the mapping between the degraded and clean pairs æ and y.
Ours: Revisiting Image Restoration Task Different from existing methods, we try to associate Taylor's Approximations with image restoration, which has not been considered in previous methods. Referring to Equation ( we define the inverse process using the function F, and then we can obtain the relationship as
x = F(#n) = Fly N).
(4)
can be figured out that the mapping function our framework corresponds the inverse operation of degradation matrix A. Deviating from most existing CNN-based methods, our framework considers the intrinsic prior knowledge of image restoration task and has better interpretability. The inverse
operation of degradation matrix A is denoted as A
F = AT.
(5)
3

---

Let - = c Yo y, we expand Equation 4 with an infinite-order Taylor's series expansion of cualar-valued of more than one variable, written compactly as
x = F(yo) = Fly + E)
(6)
1 Fyy
(F(y)
Σ
8 F
+
1
dy:
k!
Dy.....
...
(7)
80
① h
Fly),
(8)
Oy:
where we use the notation
0
a
n
n
2r F
C
Fly)
Σ
Σ
Ty .y. Ci
(9)
Ty:
11
lig
I F Eis Cn
Σ
(10)
Byhn ...
& m
When only regarding n order Taylor's Approximations, can be simplified as
a By.
F(y) Σ
Fly).
(11)
can be separated into two parts for consideration. detail, the first term, as main part, Fly represents the high-level contextualized information, which gives constant approximation of clear image (ground truth) while the rest is the local high-order spatial details. When we revisit the connection between the goals of image restoration and Equation (1], we clearly find that the main part and derivative part of Taylor's Approximations take the same effect as the two competing goals of high-level contextualized information and spatial details of image restoration respectively. In this way, our approach deviates from existing methods that optimally balance the above goals in the overall recovery process and independent from the degradation process. Specifically. we break down the overall recovery process into two manageable steps. The former first learns the high-level contextualized structures and the later combines them with the degraded input to progressively recover local high-order spatial details. The details of our proposed framework are illustrated as bellow.
3.2
Deep Taylor's Approximations Framework
Based on above analysis, we reasonably associate Taylor's Approximations with image restoration task, which has not been considered in previous restoration methods. In detail, we propose to unfold Taylor's Formula as blueprints to construct a novel framework, named Deep Taylor's Approximations Framework. In this way, almost every process part one-to-one corresponds each operation involved in Taylor's 's Formula, thereby breaking down the overall recovery process into two manageable steps. Specifically, our framework consists of two operation parts, correspondingly responsible for the mapping and derivative functions.
Mapping Function Part As aforementioned function F, it takes responsibility for mapping the degraded input the approximation of expected latent clear image. As shown in Figure [ it can be implemented with existing CNN-Based or traditional methods. In the following, we choose several representative image restoration methods F to validate the effectiveness and scalability.
Derivative Function Part Recalling Equation as
11 for the k order derivative part, it can be written
8 F'F(
())(+t)
Σ
Ei ...Ei
(12)
Jy.,...yy Byh.
11 44 (1..........m))

---

Mapping function part
y
F(y)
fout
fernvioivefunction part
0
[P
Fy yo)
11
fout + Jout
y y]
four + Jons + Jout
F" yyo)²
1l yl
(y)) yo)
foue +
kt gout
Mapping function F
Derivative function G (sharing weights)
C Concat
! Sum
The overview structure of Deep Taylor's Approximations Framework. It consists of two parts, Mapping function part and Derivative function part G. The former learns map the degraded imput image y the main energy of ground truth, while the later combines them with the degraded input to progressively recover the local high-order details. The paramcters of Derivative function part G are shared across the progressive stages. The whole framework is trained in end-to-end manner.
Figure 1:
differentiating above k order part (r())))) for as
a (
(c) p(++)()) - F() x x((())
(13)
By
multiplying above Equation (
(3) by
(F(((())(())
x (E==(((H))))(( (x) KF(() (z) x
(F(++))
(14)
dy
((+1)) (c) 1F(()) (c)
(15)
To this end, we exploit Derivative function sub-network, named G to take effect above process. We denote the l order output of network G (*)) (e)( simply recorded g Referring Equation 15 we can find out the connection between order output and k + order one
Yout
G(gone) + (k) (z)
(16)
Replacing (()) (x) with gout as Fout G(ggin)) + k o
(17)
Implementation Referring above analysis, we can find that the Mapping function part only requires the degraded image as input
fous F(y),
(18)
referring Equation 17b. the Derivative function part G needs the your from the Mapping function sub-network F and blurry image It is because that the unfolding iteration process of G running involves y. In this regard, concatenating oout and y into G as input for inference
k+1
G(Concat([g) yl))
(19)
Taking together above operation steps, the final output of order Deep Taylor's Approximations framework can be obtained as
12
fout +
Σ
kl
Sout
(20)
5

---

Loss Function Following the setting in 14,, we apply the L1 loss function to optimize the proposed framework. Suppose æ denotes the corresponding ground truth of given degraded image y, the final loss can be recorded as
== -i - -0 + (x - Sout )
(21)
where À is a weighted factor.
4 Experiments
To demonstrate the effectiveness and scalability of our proposed framework, we conduct experiments over two image restoration tasks, i.e., image deraining and image deblurring. In the following part, we first conduct the ablation experiments over several representative methods with/without integrating with our framework and report results standard benchmarks. And then, we show the quantitative and qualitative results to explore our Deep Taylor's Approximations Framework with different series orders. Finally, we demonstrate the visual results to further validate the interpretability and the underlying mechanism.
4.1 Experimental Settings
For image deraining task, we choose four representative deraining methods, including DDN [2]. PReNet [3], DCM [14] and MPRnet [57] to integrate with our proposed Deep Taylor's Approx- imations Framework for comparison between the integrated and the original. Specifically, all the aforementioned methods are exploited as mapping function part of our framework and then operate together with derivative function part for image deraining. Regarding experiment rainy datasets, the first three methods including DDN, DCM and PReNet are trained and evaluated over three widely- used standard benchmark datasets, including Rain 100H, Rain Rain800. In detail, Roin heavy rainy dataset are composed with 1,800 rainy images for training and 100 rainy samples for testing. However, referring the PReNet work, we find that 546 rainy images from the 1,800 training samples have the same background contents with testing images. Therefore, for fair comparison, we screen out these 546 rainy images from training set, and train all the selected baseline models on the remaining 1,254 training images. And, Rain 100L dataset contains 200 training samples and 100 testing images with light level degraded rain streaks. addition, the Rain800 is proposed in (58), which includes 700 training and 100 testing images. As for the recent method MPRnet, referring original paper, 13,712 clean-rain image pairs gathered from Rain 100H, Rain 1001., Test100, Test2800 and Test 200 are used for training and two above Rain Main1001 10H, 100L for testing. For fair comparison, we follow the datasets setting as original paper. For the implementation, owing to unrelcased open codes of DCM, we insert it into the training framework of PreNet for experiment comparison while the remaining follows the same pipeline.
For image deblurring task, typical methods like DCM [14. MSCNN 30 and RDN 66 65 are
adopted in our experiments. As in 622 45 26], we use the GoPro 30 dataset that contains 2,103 image pairs for training and 1,111 11 pairs for evaluation. Furthermore, to demonstrate generalizability, we take the model trained on the GoPro dataset and directly apply it on the test images of the HIDE [1 and RealBlur (39] datasets. The HIDE dataset is specifically collected for human-aware motion deblurring and its test set contains 2,025 images. While the GoPro and HIDE datasets are synthetically generated, the image pairs of RealBlur dataset are captured in real-world conditions. The RualBlur dataset has two subsets: (1) RealBlur-J is formed with the camera JPEG outputs, and (2) RealBlur-R R is generated offline by applying white balance, demosaicking, and denoising operations to the RAW images.
For quantitative comparisons, two popular measurement metrics are employed in the following quantitative comparison, the peak signal-to-non -oise ratio (PSNR) and structural similarity index (SSIM).
4.2 Integrating Existing Methods into Our Framework
We evaluate the integrated models against their baselines (i.e., the models trained without integration) in terms of PSNR/SSIM with 3-order (see Section 4.3). As shown Tanle Table we can clearly find that, by integrating with our proposed framework, all the baselines obtain performance gain over all the datasets in image deraining and image deblurring task, which validates the effec-
tiveness of our framework. For example, in Table 2 DCM [14 obtains 0.36dB and 0. 0.I psnr

---

gain on GoPro and RealBlur dataset. Taking two representative examples in Figure 3 the result of "Integrated" is cleaner than that of "Original" with fewer blurry effect. Meanwhile, for image deraining, the results of "Original" maintain the better spatial details than that of "Integrated". The quantitative comparison from Table | also testifies above analysis.
Table I: Comparison of quantitative results in terms of PSNR (dB) and SSIM over image deraining. "Original" represents to employ the same architecture as original paper, and "Integrated" indicates to integrate them with our proposed framework. The corresponding experiment setting can be referred as section 4.2
Roin
Rain
Rain800
Model
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Original
22.26
0.690
34.85
0.951
24.04
0.867
DDN
12
Integrated
22.34
0.701
35.16
0.953
24.21
0.867
Original
26.77
0.858
32,44
0.950
22.03
0.720
PReNet 35
Integrated
26.78
0.858
34.44
0.950
22.04
0.723
Original
28.66
01,889
37.15
0.980
26.78
0.859
DCM 14
Integrated
28.77
0.901
37.31
0.981
26.93
0.861
Original
30.44
0.872
36,24
0.959
MPRnet
Integrated
30,52
0.873
36,30
0.960
Table 2: Comparison of quantitative results in terms of PSNR (dB) and SSIM over image deblurring. All the models follow the same settings original works.
GoPro
HIDE
RealBlur-J
RealBlur-R
Model
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Original
30.25
0.929
28.00
0.901
35.55
0.927
28.60
0.871
DCM
Integrated
30.39
0.933
28.10
0.926
35.91
0.930
28.69
0.873
Original
29.08
0.914
25.72
0.874
32.51
0.841
27.87
0.827
MSCNN
30
Integrated
29.19
0.916
25.80
0.877
32.79
0.852
27.99
0.830
Original Integrated
29.20
0.929
26.44
0.859
28.38
0.899
26.50
0.871
RDN 66
29.31
0.934
26.50
0.866
28.44
0.901
26.56
0.880
4.3
Deep Taylor's Approximations Framework with Different Orders
In this section, we perform the ablation studies about different orders of our proposed Deep Taylor's Approximations Framework. For simplicity, we take the representative method, i.e., DCM, to validate the underlying mechanism over image deraining and image deblurring. detail, DCM is used for mapping function part and taken together with different orders derivative function part from to 6.
Table 3: Comparison of quantitative results terms PSNR (dB) and SSIM about the different order Taylor's
approximation results from 0 to over image deraining.
Rain
Rain 100L
Rain800
Model
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
order-O
28.66
0.889
37.15
0.980
26.78
0.859
order-1
28.67
0.889
37.17
0.980
26.78
0,859
order-2
28.71
0.890
37.20
0.980
26.79
0.859
DCM 14
order-3
28.77
0.901
37.31
0.981
26.93
0.861
order-4
28.69
0.889
37.22
0.980
26.76
0.859
order-5
28.73
0.889
37.24
0.981
26.83
0.860
order-6
28.77
0.891
37.18
0.980
26.72
0.859
Table 4: Comparison of quantitative results in terms of PSNR (dB) and SSIM about the different order Taylor's approximation results from to over image deblurring.
GoPro
HIDE
RealBlur-J
RealBlur-R PSNR
Model
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SSIM
order-0
30.32
0.921
28.03
0.913
35.83
0.918
28.62
0.862
order-1 1
30.33
0.922
28.05
0.914
35.85
0.918
28.64
0.862
order-2
30.36
0.928
28.07
0.920
35.88
0.925
28.66
0.867
DCM 14
order-3
30.39
0.933
28.10
0,926
35.91
0.930
28.69
0.873
order-4
30.35
0.932
28.06
0.925
35.87
0.929
28.65
0.872
order-S
30.36
0.925
28.07
0.918
35.88
0,922
28.66
0.865
order-6
30.36
0.932
28.08
0.924
35.89
0.929
28.66
0.871
7

---

Ge) Rainy impoi
(b) O-arder
(c) I-erder
(d) 2-order
(a)
(c)
Co Rainy input
(b) G-arder
(c) I-order
(d) 2-order
(a) 3-order
4-order
(c) S-order
6-order
Figure 2: Visual results from the Rain800 rainy dataset for DCM 14 with different-order framework.
The O-order represents the original DCM 14 method. And the parameters of derivative function part are shared cross all the derivative stages
Training Contiguration. One NVIDIA GTX 2080Ti GPU is used for training. In the experiments, all the variant DCM networks share the same training setting. The patch size is 100 x 100, and the batch size is 4. The ADAM algorithm is adopted to train the models with initial learning rate 1 x 10-3, and ends after 100 epochs. When reaching 30, 50 and 80 epochs, the learning rate is decayed by multiplying 0.2 and À is set 1.0 in loss function.
Experiment Analysis. shown in Table 3 and Table 4 we can clearly find that, by integrating with our proposed framwork, all the baselines obtain performance gain in image deraining and image deblurring task over standard benchmarks, which validates the effectiveness of our framework. In addition, we also report the visual results in Figure 2 and Figure for image deraining and deblurring. Owing to the 3-order performs the best, is chose as the baseline framework for implementation. Taking the trade-off 3-order DCM for example, we evaluate it over image deraining task and analyze the different high-order output of derivative function part to testify the underlying mechanism.
4.4 Limitations and Discussions
First, evaluate the effectiveness of proposed framework over two typical image restoration tasks (i.e., image deraining and image deblurring) and we will conduct more comprehensive experiments restoration tasks (e.g., image super-resolution image denoising and image dehazing). Second, the focus of this work is not designing a new image restoration network, so both the mapping function and derivative function steps can be readily replaced by other advanced embodiments for better performance. since we the mapping function part is implemented only by several convolution layers in work. It should be constructed by various complex neural architectures for comparison in the future. In addition, the parameters of derivative step are shared across different orders. We will explore the situation when the parameters are not sharing weights.
We use two convolutional layers in the experiments, and this two-layer structure can be readily replaced by other advanced embodiments for better performance.

---

9901 E
2 B7H
8T2-3300
872-3300
(a) Blurry impot
(b) 0-arder
(c) I-erdet
(d) 2-order
2, 17
872-3300
872-3300
872-2390
872-2970
(c) S-erder
(a) Blurry Inpest
(b) (-order
(c) I-onder
(d) 2-order
A38
(a) 3-order
(b) 4-arder
(c) 5-order
Figure 3: Visual results from the GoPro blurry dataset for DCM [4] with different-order framework.
5
Conclusion
In this paper, we propose novel framework, Deep Taylor's Approximations Framework for image restoration by unfolding Taylor's Formula. Our proposed framework is orthogonal to existing deep learning-based image restoration methods and thus can be easily integrated with these methods for further improvement. Extensive evaluations demonstrate the effectiveness and interpretability of our framework in image restoration task, i..., image deraining and image deblurring. We believe the Deep Taylor's ' Approximations Framework also has potential to advance other image/video restoration tasks, e.g., image/video super-resolution and image/video denoising.
Broader Impact
Since image degradation is the most common and inevitable phenomena in imaging systems, e.g., from the point spread function of the optical system the shaking during shooting, the image restoration technology has broad impacts and practical values in various applications. Related fields include remote sensing, medicine, astronomy, military, and civilian imaging equipment. Image restoration technology aims to recover high-quality images from given low-quality counterparts. In daily life, it can help people who cannot afford professional cameras to take photos with low- end devices. Therefore, our image restoration method based on the proposed unfolding Taylor's approximations can provide high-quality clear images to facilitate intelligent data analysis tasks in these fields.
The negative consequences may accompany image restoration technology despite the many benefits brings. This is mainly associated with certain risks of privacy and consumer experience. For example, media or criminal cases, the identity of certain persons will be blurred to protect privacy. this case, image deblurring technology may reveal the personal identity, thereby compromising their privacy. addition, some users may beautify the image before sharing Therefore, the use of image restoration technologies to restore the posted images may arouse users' antipathy. Furthermore, it important to be cautious of the results of any image restoration algorithms as failures, leading to misjudgments and affecting subsequent use.

---

References
[1] S. D. Babacan, R. Molina, and A. K. Katsaggelos. Variational bayesian blind deconvolution using a total variation prior. IEEE Transactions on Image Processing, 18(1): 26, 2009.
[2] Yuanchao Bai, Gene Cheung, Xianming Liu, and Wen Gao. Graph-bascr blind image deblurring from a single photograph. IEEE Transactions on Image Processing, 28(3):1404-) 404-1418, 2019.
[3] Jérémie Bossu, Nicolas Hautiere, and Lean-Philipps Tarel. Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International journal of computer vision, 93(3):348-
367, 2011.
[4] Ayan Chakrabarti. A neural approach blind motion deblurring. In European Conference on Computer Vision, 2016.
[5] T.F. Chan and Chiu-Kwong Wong. Total variation blind deconvolution. IEEE Transactions on Image Processing, 7(3):370-375, 1998.
[6] Jiangxin Dong, Jinshan Pan, Deqing Sun, Zhixun Su, and Ming- Ming-Hsuan Yang. Learning data terms for non-blind deblurring. In European Conference on Computer Vision, 2018.
[7] Stefan Roth Dong. Jiangxin and Bernt Schiele, Deep wiener deconvolution: Wiener meets deep learning for image deblurring. In Newral Information Processing Systems, 2020.
[8] Shuangli Du Yiguang Liu, Mao Ye, Zhenyu Xu, Jie Li, and Jianguo Liu. Single image deraining via decorrelating the rain streaks and background scene in gradient domain. Pattern Recognition, 79:303-311 2018
19] Yingjun Du, Jun Xu, Qiang Qiu, Xiantong Zhen, and Lei Zhang. Variational image deraining. In WACV, 2020.
[10] Yuchen Fan, Jiahui Yo, Yiqun Mei, Yulun Zhang, Yun Fu, Ding Liu, and Thomas S Huang. Neural sparse representation for image restoration. arXiv preprint arXiv:2006. 2003520000043377 2020.
[11] Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6):2944- 2956, 2017.
[12] Xueyang Fu, Jiabin Huang. Delu Zeng. Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
[13] Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. Lightweight pyramid networks for image deraining. IEEE transactions on neural networks and learning systems, 3 ((14)))774-880 2019.
[14] Xueyang Fu, Qi Qi, Yurui Zhu, Xinghao Ding, and Zheng-Jun Zha. Rain streak removal via dual graph convolutional network. In AAAI, 2021.
[15] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In IEEE Conference on Computer Vision and Puttern Recognition. 2019.
[16] Dong Gong, Mingkui Tan, Yanning Zhang, Anton van den Hengel, and Qinfeng Shi. Blind image deconvolution by automatic gradient activation. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[17] Dong Gong, Jie Yang. Linggiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. From motion blur to motion flow: a deep learning solution for removing heterogencous motion blur. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
[18] M. F. Goodchild, A general framework for error analysis in measurement-basee gis. Journal of Geographi- cal Systems, 6(4):323-324, 2004.
[19] J. C. Helton and F. J. Davis. Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliability Engineering System Safety, 81(1):23 69, 2003.
[20] Gbm Heuvelink. Error propagation in environmental modelling with gis. University of Liverpool. 1998.
[21] Gbm Heuvelink and PabA Stein. Propagation of errors in spatial modelling with gis. International journal of geographical information systems, 1989.
[22] Jin-Hwan Kim, Chul Lee, Young Sim, and Chang-Su Kim. Single-image deraining using an adaptive nonlocal means filter. In ICIP, 2013.
[23] T. Kobayashi, H. Miller, and W. Othman. Analytical methods for error propagation in planar space time prisms. Journal of Geographical Systems, 13(4):377-354, 2011.
[24] Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper-laplacian priors. In Neural Information Processing Systems, 2009.
[25] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In IEEE Conference on Computer Vision
10

---

and Paliern Recognition, 2018.
[26] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2. Deblurring (orders- of magnituue) faster and better. IEEE International Conference on Computer Vision, 2019.
[27] Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In IEEE Conference on Computer Vision and Puttern Recognition, 2019.
[28] Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang. Eric K Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. Single image deraining: comprchensive benchmark analysis. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[29] Siyuan Li, Wenqi Ren, Feng Wang. Iago Breno Araujo, Eric K Tokuda, Roberto Hirata Junior, Roberto M Cesar-Jr, Zhangyang Wang, and Xiaochun Cao. A comprehensive benchmark analysis of single im- age deraining: Current challenges and future perspectives. International Journal of Computer Vision, 129(4):1301-1322 2021.
[30] Scungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[31] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[32] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsaan Hsuan Yang. Deblurring images via dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 40 10):000):52222 2211812228,22 2017.
[33] Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In European Conference on Computer Vision, 2020.
[34] Wan-Joo Park and Kwae-Hi Hi Lee. Rain removal using kalman tilter in video. In ICSMA, 2008
[35] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[36] Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojje Guo, Wangmeng Zmo, and Ming- Huung Yang. Image deblurring via enhanced low-rank prior. IEEE Transactions on Image Processing, 25(7):3426-344 2016.
[37] Wenqi Ren, Jiawei Zhang, Jinshan Pan, Sifci Liu, Jimmy Ren, Junping Du, Xiaochun Cao, and Ming- Hisng- Yang. Deblurring dynamic scenes via spatially varying recurrent neural networks. IEEE transactions on pattern analysis and machine intelligence, 2021.
[38]
William Hadley Richardson. Bayesian- iterative method of image restoration. JoSA, 62(1):55-59,
1972.
[39] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithmm. In European Conference Computer Vision, 2020.
[40] Uwe Schmidt and Stefan Roth. Shrinkage fields for effective image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[41] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and I Ling Shao. Human- aware motion deblurring. In IEEE International Conference on Computer Vision, 2019.
[42] Feng Shi, Jian Cheng, Li Wang, Pew-Thian Yap, and Dinggang Shen. Lrtv; Mr image super-resolution with low-rank and total variation regularizations. IEEE transactions on medical imaging, 34(12):2459 -466. 2015.
[43] Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive aatch-hicrarchical network for adaptive motion deblurring. In IEEE Conference on Computer Vision and Puttern Recognition, 2020.
[44] Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. Learning a convolutional neural network for non-unite non-uniform motion blur removal. IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[45] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scalc-recurrent network for deep image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[46] J. W. Tukey. The propagation of errors, fluctuations and tolerances, supplementary formulas. 1957.
[47] Cong Wang, Xiaoying Xing, Yutong Wu, Zhixun Su, and Junyang Chen. Desfn: Deep cross-scale fusion network for single image rain removal. In ACM MM, 2020.
[48] Hong Wang, Yichen Wu, Minghan Li, Qian Zhao, and Deyu Meng. A survey on rain removal from video and single image. arXiv preprint arXiv: /909.08326 2019.
[49] N Weiner and Interpolation Extrapolation. Smoothing of stationary time series; With engincering applica- tions, 1949.
11

---

[50] Jing Xu, Wei Zhao, Peng Liu, and Xianglong Tang. Removing rain and snow in single image using guided filter. In CSAE, 2012.
[51] Li Xu and Jiaya Jia. Two-phase kernel estimation for robust motion deblurring. In European Conference on Computer Vision, 2010.
[52] Li Xu, Jimmy S. J. Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvolution. In Neural Information P Processing Systems, 2014.
[53] J. Xue, Y. Leung. and J. H. Ma. High-order taylor series expansion methods for error propagation in geographic information systems. Journal of Geographical Systems, 17(2):187-206. 2015.
[54] Yanyang Yan, Wenqi Ren, Yuanfang Guo, Rui Wang. and Xiaochun Cao. Image deblurring via extreme channels prior. In IEEE Conference on Computer Vision and Puttern Recognition.
[55] Wenhan Yang. Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
[56] Wenhan Yang. Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu. Single image deraining: From model-hased to data-driven and beyond. IEFE Transactions on pattern analysis and machine intelligence, 2020.
[57] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuun Yang, and Ling Shao. Multi-stage progressive image restoration. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[58] H. Zhang, V. Sindagi, and V. M. Patel. Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, 2017.
[59] He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi-stream dense network.
In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[60] He Zhang, Vishwanath Sindagi. and Vishal M Patel. Image de-raining using a conditional generative adversarial network. TCSVT, 2019.
[61] Hongguang Zhang. Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image dehlurring. In IEFF Conference on Computer Vision and Puttern Recognition, 2019.
[62] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[63] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[64] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger. Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In IEEE Conference on Computer Vision and Puttern Recognition, 2020.
[65] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong. and Yun Fu. Residual non-local attention networks for image restoration. IEEE transactions pattern analysis and machine intelligence, 2019.
[66] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, June 2018
[67] Yulun Zhang, Yapeng Tian, Yu Kong. Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 2020.
[68] Honghe Zhu, Cong Wang, Yajie Zhang, Zhixun Su, and Guohui Zhao. Physical model guided deep image deraining. In ICME, 2020.
[69] Daniel Zoran and Yair Weiss. From learning models natural image patches to whole image restoration.
In IELE International Conference on Computer Vision, 2011.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. see Section 4.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes].
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. See Section 1. [ and Section 443
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments. Reproducibility can be guaranteed.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes]. See Section [4.1 and Section 4.3
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you't 're using/curating? [Yes], the data creators claim that they allow and encourage the data for scientific research.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We train and test the framework using the publicly accessible datasets.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "lck2cukO8gY", "reviewer": "Reviewer_Rxva", "summary": "This paper aims to develop an image restoration framework through a combination of Taylor’s Approximation and deep learning to break the entire recovery process into two manageable steps. Specifically, the former learns high-level contextualized information, and the latter combines it with the degraded input to progressively recover local high-order spatial details. Several promising methods are integrated with the proposed framework and achieve some improvement. Ablation studies about Taylor’s order are also conducted to provide a comprehensive analysis. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ The authors innovatively propose a new image restoration framework, in which others researchers or users are likely to utilize it for further restoration performance improvement. The exploration of combining classic methods with deep learning for image restoration is worth trying and encouraging.\n\n+ The claim of this submission is well supported by theoretical analysis. The authors revisit the connection between image restoration process and Taylor’s Formula, and break the restoration process into two manageable steps for the deep network design.\n\n+ Extensive analysis on two image restoration tasks (image de-blurring and de-raining) clearly verify the effectiveness and scalability of the proposed method. \n\nCons: \n-Limited reported image restoration tasks. I am curious that whether the proposed method can deal with other ill-posed image restoration tasks, like the classical super-resolution and image de-noising. It would be better if the authors could provide more discussion about it. \n\n-Some detailed comparison between the original baselines and that integrating with Taylor’s framework is missing, e.g., the computational time and Flops.\n\n-The premise of Taylor’s Formula is that \\epsilon in Equation (6) should be small enough. Are there any other image restoration tasks that do not meet this premise?\n\n-What is the difference between the proposed Taylor’s approximation framework and other deep unfolding frameworks, such as HQS [*1, *2] or proximal gradient operator [*3]?\n\n[*1] Learning Deep Priors for Image Dehazing, ICCV, 2019.\n\n[*2] Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels, CVPR, 2019.\n\n[*3] Deep Gradient Projection Networks for Pan-sharpening, CVPR, 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has clearly presented and solved the limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "WGV08MGRSGB", "reviewer": "Reviewer_FEEp", "summary": "The authors proposed to use additional network structures to approximate the derivative parts of Taylor expansion for image restoration. The idea is similar to residual learning by recursively adding details to the images for a better visual quality. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros,\n\n- The novel idea of combining Taylor formula with image restoration tasks, and good explanation of the motivations.\n- Extensive experiments on different image restoration tasks including deraining and deblurring. \n\nCons,\n\n- The proposed methods may not be as effective as expected, since the implementation of the derivative parts are the simple additional RNN structures with feature aggregation from different levels. The derivative functions are also approximated by simple convolutional layers. \n- The quantitative comparisons with other baselines after integrating the new frameworks may not be fair enough, since the new model contains additional parameters and it will bring performance improvement.\n- The authors claim a better detail restoration, but it cannot be reflected using PSNR or SSIM metrics. PSNR can be still higher even though the results are blur. \n- The visual results are not clearly revealing the better quality, and the 3-order network is not clearly better than 0-order in visual.\n- The authors empirically chose 3-order as the best-performance baseline, but what should be the theory support that 3-order network works the best? Table 3 and 4 also show some randomness of different orders, without showing clear evidence of increasing performance of using higher orders. Intuitively, it should be the case that using higher-order will bring back more details. But the results make sense given the current implementation.\n- Some typos in line 250-251.\n\nI like the idea of approximate a Taylor formula for image restoration, but the current implementation is not the best way to achieve this idea. The experiment results lack fairness, and more visual comparisons should be provided. Currently, I sit on borderline. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the above comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5"}, "extra_text": {}}, {"review_id": "MZSpGCR3DeO", "reviewer": "Reviewer_79vC", "summary": "This paper introduces a new perspective for designing image restoration framework by unfolding the Taylor’s Formula. The proposed framework consists of two steps, and can be integrated to existing deep networks.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The authors analyze image restoration tasks in a framework of Taylor's Formula. The idea is interesting.\n2. The proposed method can be integrated into exisiting methods and boost performance.\n\nWeakness.\n1. My main concern is the technical correctness part of the method.\n    First, most image restoration tasks are ill-posed problems, thus a analytical and accurate inverse process is impossible by principle, as in Eq (4).\n    Second, Taylor's formula is only suitble for functions that are high-order smooth. As for CNN, operators such as dropout or max-pooling are not smooth. Then what kind of CNNs are suible for the proposed method\n\n2. How to supervised G, so its result can approximate k order output ?\n    If we cannot ensure the meaning of G's output, is it just some kind of residual learning?\n    I would like to see more comparisons between residual learning method and this method.\n\n3. G is a network with additional parameters. When it is integreted to other methods, the total parameters increase.\n    Is this the reason for performance gain ?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mathematically, using Taylor's formula requires several assumpations about the function, which may not be suitable for all restoration tasks.\nI think it would be better to be more strict and analyze what kind of restoration tasks are suitble for this method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "8XdqwHWVKK3", "reviewer": "Reviewer_qVNs", "summary": "The author proposes a novel deep neural network, which leverages the intrinsic knowledge of low-level vision and Taylor’s Approximations, for the image restoration tasks. The author found that the main part and derivative part of Taylor’s Approximations takes the internal similarity as the high-level structure and spatial details of image restoration, respectively. From this, the author breaks down the restoration process into two manageable steps, corresponding to the mapping function and derivative function, respectively. The proposed framework is orthogonal and can be easily integrated into existing deep learning-based methods. The idea of using Taylor’s Approximations to guide deep network design for image restoration is quiteThe limitation and potential negative societal impact of their work have been discussed in their paper. interesting. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1)\tThe author proposes a novel image restoration framework by leveraging the intrinsic knowledge of image restoration task and Taylor’s Approximations. This idea is interesting and clearly differs from previous related methods. \n\n2)\tThe framework is designed by unfolding Taylor’s formula, which is sufficiently supported by theoretical analysis. This design is more reasonable than current roughly layer-stacking deep networks for image restoration.\n\n3)\tThe proposed framework can be easily integrated with existing methods, which improves their performance by higher PSNR values and sharper details. Moreover, the submission is clearly written, well organized and easy-to-follow.\n\n4)\tExtensive experiments over the representative image de-blurring and image de-raining tasks are reported. Quantitative and qualitative results show its effectiveness and scalability. Besides, ablation studies about Taylor’s order are well designed and reported to analyze the order selection.\n\nWeaknesses: \n\n1)\tThe author employs Taylor’s Approximation to unfold the universal image restoration model in Line (130-132). However, it is inadequate that only image de-raining and image de-blurring tasks are reported to demonstrate the effect. More comparisons and results of other image restoration tasks, e.g., super-resolution, should be presented.\n\n2)\tThe author presents the ablation experiments about Taylor’s orders. The results show that that of Order = 3 performs the best. However, the authors have not clearly illustrated the underlying reasons. For example, feature visualization or other technologies should be adopted to do so. \n\n3)\tIf the loss is supervised at each layer of the derivative function part, will the effect be better?\n\n4)\tHow is the generalization ability of the model in real scenes?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation and potential negative societal impact of their work have been discussed in their paper.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for image restoration using deep neural networks based on decomposing the problem using a Taylor approximation. 3 out of 4 reviewers appreciate the contribution as useful and novel. The most negative reviewer (79vC) has the reasonable concern that the proposed method seems mostly empirical and is not well supported by theory. Unfortunately this reviewer only provided a short review and did not engage in committee discussions or acknowledge the author response: for this reason we cannot give it too much weight in making a decision. The other reviewers acknowledged the author response and are satisfied by it. No serious concerns were identified that would invalidate the contribution made by this paper. The author response looks convincing to me as well: Authors, please integrate this discussion and all promised new results into the camera ready version of the paper.
Decision: Accept (Poster)