Clean Images are Hard to Reblur: Exploiting the Ill-Posed Inverse Task for Dynamic Scene Deblurring
Authors: Seungjun Nah
Sanghyun Son
Jaerin Lee
Kyoung Mu Lee
Venue: ICLR 2022 Poster (2022)
Keywords: Deblur
Reblur
Loss
Test-time adaptation
Self-supervised

Abstract: The goal of dynamic scene deblurring is to remove the motion blur in a given image. Typical learning-based approaches implement their solutions by minimizing the L1 or L2 distance between the output and the reference sharp image. Recent attempts adopt visual recognition features in training to improve the perceptual quality. However, those features are primarily designed to capture high-level contexts rather than low-level structures such as blurriness. Instead, we propose a more direct way to make images sharper by exploiting the inverse task of deblurring, namely, reblurring. Reblurring amplifies the remaining blur to rebuild the original blur, however, a well-deblurred clean image with zero-magnitude blur is hard to reblur. Thus, we design two types of reblurring loss functions for better deblurring. The supervised reblurring loss at training stage compares the amplified blur between the deblurred and the sharp images. The self-supervised reblurring loss at inference stage inspects if noticeable blur remains in the deblurred. Our experimental results on large-scale benchmarks and real images demonstrate the effectiveness of the reblurring losses in improving the perceptual quality of the deblurred images in terms of NIQE and LPIPS scores as well as visual sharpness.

Published as a conference paper at ICLR 2022
CLEAN IMAGES ARE HARD TO REBLUR: EXPLOITING THE ILL-POSED INVERSE TASK FOR DYNAMIC SCENE DEBLURRING
Seungjun 1:2 Sanghyun Son', Jaerin Lee' & Kyoung Mu Leet
AARRI, Department of ECE, Seoul National University, Seoul, Korea 2NVIDIA seungjun n aaahh/mmaa...mmmmmmm com, {thstkdgus35, ironjr, kyoungmuJesnu.ac..
ABSTRACT
The goal of dynamic scene deblurring is to remove the motion blur in a given im- age. Typical learning-based approaches implement their solutions by minimizing the L1 or L2 distance between the output and the reference sharp image. Recent attempts adopt visual recognition features in training to improve the perceptual quality. However, those features are primarily designed to capture high-level con- texts rather than low-level structures such blurriness. Instead, we propose more direct way to make images sharper by exploiting the inverse task of deblur- ring, namely, reblurring. Reblurring amplifies the remaining blur rebuild the original blur, however, a well-deblureed clean image with zero-magnitude blur is hard to reblur. Thus, we design two types of reblurring loss functions for better de- blurring. The supervised reblurring loss at training stage compares the amplified blur between the deblurred and the sharp images. The self-supervised reblurring loss at inference stage inspects if there noticeable blur remains in the deblurred. Our experimental results on large-scale benchmarks and real images demonstrate the effectiveness of the reblurring losses in improving the perceptual quality of the deblurred images in terms of NIQE and L.PIPS scores as well as visual sharpness.
INTRODUCTION
Motion blur commonly arises when the cameras move or scene changes during the exposure in dynamic environments. Dynamic scene deblurring is challenging ill-posed task finding both the locally-varying blur and the latent sharp image from a large solution space. Traditional ap- proaches (Hirsch et al., 2011; Whyte et al., 2012; Kim al., 2013; Kim & Lee, 2014) tried to alleviate the ill-posedness by using statistical prior on sharp images such as gradient sparsity.
Instead of using such handcrafted knowledge, recent methods take advantage of large-scale datasets as well as deep neural networks (Nah et al., 2017: Su et al., 2017; Noroozi et al., 2017; Nah et al., 2019; Shen et al., 2019). Usually, the learning is driven by minimizing the pixel-wise distance the ground truth, e.g., L1 or L2, so that the PSNR between the deblurred and the sharp reference can be maximized. By utilizing modern ConvNet architectures and training techniques, state-of-the-ee approaches (Nah et al., 2017: Tao et al., 2017; Gao et al., 2019; Yuan et al., 2020; Park et al., 2020; Chi et al., 2021) have been developed toward higher capacity and deblurring accuracy. Still, most behavior often witnessed in ill-posed problems with large solution space (Ledig et al., 2017; Menon et al., 2020).
methods tend to suffer from the blurry predictions due to the regression-to-mmes
To overcome limitations of the conventional objectives, concepts of perceptual (Johnson et al., 2016) and adversarial (Ledig et al., 2017: Nah et al., 2017: Kupyn et al., 2018) loss terms from high- level semantic tasks have been introduced to improve the visual quality of the deblurred results. Nevertheless, such high-level losses may not serve as optimal goals for blur removal as low-level structural properties, e.g., blurriness, are not the primary features considered in their formulations. As illustrated in Figure 1, results from the previous deblurring methods are still blurry to a degree and the VGG and the adversarial losses images across different architectures (Tao et al., 2018; Gao et al., 2019; Kupyn et al., 2019).
sufficient obtain perceptually pleasing and sharp
Most work was done SNU

---

Published as a conference paper at ICLR 2022
(a) True Sharp
(h) SRN
(c) SF-Sharing
(d) DeblurGANv2
(c) Ours (Deblurred)
(I) True Blur
(g) Reblurred
(h) Reblurred
(i) Reblurred
(j) Ours (Reblurred)
Figure Comparison of the deblurred images and their reblurred counterparts. For each image, we visualize the remaining blur kernel (Cho & Lee, 2009) at the center pixel visualized on the right bottom side. Upper: The kernels from the previous deblurring methods implicate the direction of the original blur. Lower: When the proposed reblurring module is applied, our result does not lose sharpness as we reconstruct the output that is hard to be reblurred.
While the deblurred images look less blurry compared with the original input, it is still possible to find nontrivial blur kernels with directional motion information. From the observation, we introduce the concept of reblurring which amplifies the unremoved blur in the given image and reconstructs the original blur. We note that our reblurring operation aims to recover the original motion trajectory in the blurry input, rather than synthesize arbitrary, e.g-, Gaussian, blurs. Therefore, deblurred clean image is hard to reblur as no noticeable blur can be found to be amplified, making predict the original shape of blur
ideally
reblurring an ill-posed task. In contrast, is straightforward from insufficiently deblurred images as shown in Figure 1. We propose to use the difference between non-ideally deblurred image and the ideal sharp image in terms of reblurring feasibility as the new optimization objective, reblurring loss for the image deblurring problem.
The reblurring loss is realized by jointly training pair of deblurring and reblurring modules. The reblurring module performs the inverse operation of deblurring, trying to reconstruct the original blurry image from a deblurred output. Using the property that the blurriness of a reblurred image depends on the sharpness quality of the deblurred result, we construct two types of loss functions. During the joint training. supervised reblurring loss compares the amplified blurs between the de- blurred and the sharp image. Complementing L1 intensity loss, the supervised reblurring loss guides the deblurring module to focus on and eliminate the remaining blur. While our training strategy is similar the adversarial training of GANs (Goodfellow et al., 2014) in sense that our deblurring and reblurring modules play the opposite roles, the purposes and effects of the adversary are differ- ent. The reblurring loss concentrates on image blurriness regardless of image realism. Furthermore, in contrast to the GAN discriminators that are not often used at test time, our reblurring module can be used to facilitate self self-supervised reblurring loss. By making the deblurred image harder to reblur, the deblurring module can adaptively optimize itself without referring to the ground truth.
Our reblurring loss functions provide additional optimization directives to the deblurring module and can be generally applied to any learning beaniin-bbssed image deblurring methods. With the proposed approach, we can derive sharper predictions from existing deblurring methods without modifying their architectures. We summarize our contributions follows:
Based on the observation that clean images are hard to reblur, we propose novel loss functions for image deblurring. Our reblurring loss reflects the preference for sharper images and contributes to visually pleasing deblurring results.
At test-time, the reblurring loss can be implemented without a ground-truth image. We perform test-time adaptive inference via self-super optimization with each input.
Our method is generally applicable to any learning- basedd based methods and jointly with other loss terms. Experiments show that the concept of reblurring loss consistently contributes to achieving state-of-the-are visual sharpness well L.IPS and NIQE across different model architectures.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORKS
Image Deblurring, Classical energy optimization framework is formulated by likelihood and prior terms. Due to the ill-posedness of dynamic scene deblurring problem, prior terms have been essential in alleviating the optimization ambiguity, encoding the preference on the solutions. Sophisticated prior terms were carefully designed with human knowledge on natural image statistics (Levin, 2006; Cho & Lee, 2009; Hirsch et al., 2011; Whyte et al., 2012; Sun et al., 2013; Xu et al., 2013; Kim et al., 2013; Kim & Lee, 2014; Pan et al., 2016). Recently in Li et al. (2018), learned prior from a classifier discriminating blurry and clean images was also shown to be effective. Deep priors were also used for image deconvolution problems (Ren et al., 2020; Nan & Ji, 2020).
On the other hand, deep learning methods have benefited from learning large-scale datasets. The datasets consisting of realistic blur (Nah et al., 2017; Su et al., 2017; Noroozi et al., 2017; Nah et al., 2019; Gao et al., 2019; Jin et al., 2019; Shen al., 2019) align the temporal center of the blurry and the sharp image pairs with high-speed cameras. Learning from such temporally aligned data relieve the ill-posedness of deblurring compared with difficult energy optimization framework. Thus, more attention has been paid to designing CNN architectures and datasets than designing loss terms.
In the early work of Schuler et al. (2015), the alternating estimation of blur kernel and restored image (Cho & Lee, 2009) was adopted in CNN architecture. In Sun et al. (2015); Gong et al. (2017), the spatially varying blur kernels are estimated by assuming locally linear blur followed by non-blind deconvolution. Later, end-to-end learning without explicit kernel estimation became popular. Motivated from the coarse-to-fine approach, multi-scale CNN was proposed (Nah et al., 2017) to expand the receptive field efficiently, followed by scale-recurrent architectures (Tao et al., 2018; Gao et al., 2019). On the other hand, Zhang et al. (2019); Suin et al. (2020) sequentially stacked network modules. Recently, Park et al. (2020) proposed an image recursively. To handle spatially varying blur kernels efficiently, spatially non-uniform operations were embedded in neural networks (Zhang et al., 2018a; Yuan al., 2020).
nelti-te model that deblurs
Perceptual Image Restoration. Often, L1 or L.2 losses are used at training to achieve higher PSNR. However, such approaches suffer from blurry and over-smoothed outputs (Johnson al., 2016; Zhang et al., 2018b; Menon et al., 2020) as the learned models predict an average of all possible solutions under the ill-posedness (Ledig et al., 2017). To deal with the issue, several studies utilize deep features of the pretrained VGG (Simonyan & Zisserman, 2014) and other networks that are more related human perception (Johnson et al., 2016; Zhang et al., 2018b) and with analysis on frequency space (Tariq et al., 2020; Czolbe et al., 2020). Recent methods introduce adversarial training (Goodfellow et al., 2014) so that outputs of the restoration models be indistinguishable from real samples (Nah et al., 2017: Nimisha et al., 2017: Ledig et al., 2017; Kupyn et al., 2018; 2019). Also, there were attempts to exploit statistical properties of images and features with contextual loss (Mechrez et al., 2018) and projected distribution loss (Delbracio et al., 2021).
Nevertheless, an inherent limitation of existing perceptual objectives that they are not task- specialized for image restoration. For example, the VGG features are learned for high-level visual recognition while the adversarial loss only contributes to reconstructing realistic images without considering the existence of motion blur. Therefore, blindly optimizing those terms may not yield an optimal solution in terms of image deblurring. In practice, we observed that those objectives still tend to leave blur footprints unremoved, making possible to estimate the original blur. Our reblurring loss is explicitly designed to improve the perceptual sharpness of deblurred images by reducing remaining blurriness and thus more suitable for deblurring, acting as a learned prior.
Image Blurring. As an image could be blurred in various directions and strength, image blurring another ill ill-posed problem without additional information. Thus, intrinsic or extrinsic information is often incorporated. With a non-idcally sharp image. Bac & Durand (2007) detected the small local blur kernel in the image to magnify the defocus blur for bokeh effect. On the other hand, Chen et al. (2018) estimated the kernel by computing the optical flow from the neighboring video frames. Similarly, Brooks & Barron (2019) used multiple video frames to synthesize blur. Without such external information, Zhang et al. (2020) used generative model to synthesize many blurry images. In contrast, Bahat et al. (2017) deliberately blurred an already blurry image in many ways to find the local blur kemne.. Our image reblurring concept is similar to Bae & Durand (2007) in the sense that intrinsic cue in an image is used to amplify blur. Nonetheless, our main goal is to use reblurring provide guide to deblurring model so that such blur cues would be better removed.
3

---

Published as a conference paper at ICLR 2022
Trained
Trained
Fixed
MR
Blur
Mp
MR
Ma(E)
B
Mu(L)
RRchlur
MR
Sharrp
MR
(5(5)
5
My
(a) Reblurring module training process.
(b) Image deblurring with reblurring loss
Figure 2: Overviews of the proposed reblurring and deblurring framework. Reblurring module MR tries to reconstruct blurry image B from deblurred image L while preserving the sharpness of a pseudo-sharp image S = (D)) Mcanwhile, the deblurring module MD tries to make L sharper by comparing the amplified blur from L and the sharp image S.
#ResBlocks in deblurring module MD
4
Deblur PSNR wrt sharp GT Reblur PSNR wrt blur GT
28.17
29.67
30.78 31.90
31.48
34.29
32.66
31.48
Table I: Deblurring and reblurring PSNR (dB) by deblurring model capacity. Both tasks are
trained independently with LI loss the GOPRO dataset. The number of ResBlocks in MR is 2.
PROPOSED METHOD
In this section, we describe detailed concept of image reblurring and how the operation can be learned. Then, we demonstrate that the operation can support the deblurring module to reconstruct perceptually favorable and sharp images. We also propose a self-supervised test-time optimization strategy by using the learned reblurring module. For simplicity, we refer to the blurry, the deblurred, and the sharp image B, L. and S, respectively.
3.1 CLEAN IMAGES ARE HARD TO REBLUR
As shown in Figure 1, outputs from the existing deblurring methods still contain undesired motion trajectories that not completely removed from the input. Ideally, well-deblurred image should not contain any motion cues, making reblurring ill ill-posed. We first validate our motivation by bu- liding reblurring module MR which amplifies the remaining blur from L. MM is trained with the following blur reconstruction loss LBtur so that it would learn the inverse operation of deblurring.
Later [M (L) - -BII-
(1)
We apply Mr to the deblurred images from deblurring modules of varying capacities. Table shows that the higher the deblurring PSNR, the lower the reblurring PSNR becomes when both modules are trained with conventional L1 loss, independently from each other. It demonstrates that the better deblurred images are harder reblur, consistent our motivation.
In contrast to the non-ideally deblurred images, MR is not able to generate motion blur from sharp image S no motion information found. For a high-quality clean image, MR should preserve the sharpness. However, optimizing the blur reconstruction loss L with S may fall into learning the pixel average all blur trajectories the training dataset, i.e. Gaussian blur. In such a case, MM will apply the single uniform blur on every image without considering the scene information. To let the blur domain of MR confined to the motion-incurred blur, we use sharp images to penalize such undesired operations. Specifically, we introduce a network-generate sharp image S obtained by feeding real sharp image S to the deblurring module MD as Š MD(S). We define sharpness preservation loss SSuur as follows:
Cshap = M $1I-
We use the pseudo-sharp image Š instead of real image to make reblurring module focus image sharpness and blurriness rather than the realism. While Š differ from only by the sharpness, also differ by the realism which can be easily detected by neural networks (Wang et al., 2020).
Combining both terms together, we train the reblurring module MR by optimizing the joint loss LR:
LR
Lulur
L
(3)
4

---

Published as a conference paper at ICLR 2022
Real
Deblur Reblur
Real
Ladv LReblar
B
S
B
Blurry
Sharp
Blurry
Sharp
(L
Ma(L)
Fake
Fake
Deblurring and Rehlurring operations
(b) Rehlurring and Adversarial losses
Image deblurring and reblurring illustrated from the perspective of sharpness and realism. Training our modules with CReblur improves image sharpness without considering the image realism. The image realism can be optionally handled by adversarial loss Adv-
Figure 3:
As zero- zero- magnitude blur should remain unaltered from MMR the sharpness preservation loss can be considered a special case of the blur reconstruction loss. Figure 2a illustrates the way our reblurring module is trained from Lg.
3.2 SUPERVISION FROM REBLURRING LOSS
The blurriness of images can be more easily witnessed by amplifying the blur. Thus, we propose new optimization objective by processing the deblurred and the sharp image with the jointly trained reblurring model Mgg. To suppress the remaining blur in the output image L (M(BB) from the deblurring module Mp. the supervised reblurring loss LReblur for image deblurring is defined as
L [MR - MR((S)
(4)
Unlike the sharpness preservation term in equation 2, do not use the pseudo-sharp image $ in reblurring loss, LReblur- As the quality of the pseudo-sharp image Š depends the state of deblurring module MD, using S may make training unstable and difficult optimize, especially at the early stage. Thus we use a real sharp image S to stabilize the training. Nevertheless, as MR is trained focus on the sharpness from equation so does the reblurring loss, Leeblur.
Using our reblurring loss in equation 4, the deblurring module MD is trained to minimize the fol- lowing objective Lp:
CD = L1 + XCRoblur:
where L1 is conventional L1 loss, and the hyperparameter À is empirically set to 1. Figure 2b illustrates how the deblurring model is trained with the guide from Mgg-
At each training iterations, alternately optimize two modules MD and MMg by Lu and CR. respec- tively. While such a strategy may look similar to the adversarial training scheme, the optimization objectives are different. As the neural networks well known easily discriminate real and fake images (Wang et al., 2020), the realism could be as more obvious feature than image blurriness. Thus, adversarial loss may overlook image blurriness L and S can already be discriminated by the difference in realism. On the other hand, our reblurring loss is explicitly designed to prefer sharp images regardless of realism as we use instead of S in Lsharp to train Mg. Figure 3 conceptually compares the actual role of the reblurring loss LReblur and the adversarial loss LAdv-
3.3 TEST-TIME ADAPTATION BY SELF-SUPERVISION
After the training is over, the models learned from supervised loss terms have fixed weights at test time. When a new example that deviates from the distribution of training data is given, the supervised methods may lack ability to generalize. Our reblurring module, however, can further provide self-supervised guide that the model could be further optimized for each image at test time. While the supervised reblurring loss CReblur finds the blurriness L by comparison with the ground truth, MR also inspect the blurriness of an image without reference.
As Mg is trained to magnify the blur in L, imperfectly deblurred image would be blurred. Thus, the difference between Mg (L) and L can serve as feedback without having reference S. Fur- thermore, due to the sharpness preservation loss S a sufficiently sharp image would have little difference when reblurred. Based the property, construct the self-supervised reblurring loss
5

---

Published as a conference paper at ICLR 2022
On GOPRO dataset
On REDS
dataset
Model
Optimization
LPIPS.
NIQE,
PSNR
SSIM
LPIPS;
NIQE,
PSNR
SSIM
L
0.1635
5,996
29.66
0.8874
0.1486
3.649
30.80
0.8772
L CResius, al
0.1365
5,629
29.58
0,8869
0.1435
3,487
30.76
0,8776
U-Net
- TTA step 5
0.1327
5,599
29.52
0.8878
0.1403
3,476
30.64
0,8781
L1 CRebier a2
0.1238
5.124
29.44
0.8824
0.1252
2.918
30.46
0.8717
TTA step 5
0.1187
5,000
29.42
0.8831
0.1226
2.849
30.25
0.8701
C1
0.1246
5.252
30.62
0,9078
0.1148
3.392
31.89
0.8999
L1 CReblur ml
0.1140
5.136
30.74
0,9104
0.1071
3.305
32.01
0.9044
SRN
+ TTA step 5
0.1101
5,079
30.60
0,9100
0.1029
3,278
31,83
0,9040
C, CReNiEr, a2
0.1037
4.887
30.57
0,9074
0.0947
2.875
31.82
0,9026
TTA step 5
0.0983
4.730
30.44
0,9067
0.0909
2.798
31.50
0.9008
C
0.1179
5,490
31.53
0,9207
0.0942
3,288
32.65
0,9152
L1 CResiue al
0.0975
5,472
31.53
0,9217
0.0931
3.248
32.57
0.9143
DHN
= TTA step 5
0.0940
5,343
31.32
0,9208
0.0887
3.220
32.38
0,9139
L1 - CRebler al
0.0837
5.076
31.34
0.9177
0.0805
2.830
32,44
0.9122
TTA step 5
0.0805
4.948
31,28
0.9174
0.0763
2.761
32.17
0.9110
Table 2: Quantitative analysis of the reblurring losses and test-time adaptation applied to var- ious deblurring networks on GOPRO and REDS datasets.
that serves as a prior term embedding the preference on sharp images as
Cself = [l-M(()) L.
(6)
where L,, denotes the image with the same value as L but the gradient does not backpropagate in the optimization process. We minimize Lself for each test data obtain the sharper image. Allowing gradient to flow through L, can let L to fall into undesired local minima where both the and MR (L) are blurry. We iteratively optimize the weights of MD with fixed .RR. As Laelf only considers the sharpness of an image, we keep the color consistency by matching the color histogram between the test-time adapted image and the initially deblurred image. For the detailed optimization process of test-time adaptation strategy, please refer to the Appendix Algorithm A. The effect of test-time adaptation is conceptually visualized in Figure More iterations make the deblurred image sharper. We note that our loss functions and the test-time adaptation are applicable to general learning-based approaches.
EXPERIMENTS
We verify the effectiivneess of our reblurring loss and the generalization by applying it to multiple deblurring model architectures. We show the experimental results with a baseline residual U-Net and state-of-the- thrtt-- image deblurring models, the sRGB version SRN (Tao et al., 2018) and DHN, modified version of DMPHN (Zhang et al., 2019). For the reblurring module, we use simple residual networks with 2 ResBlock(s) with 5 convolution kernels. The training and evaluation were done with the widely used GOPRO (Nah et al., 2017) and REDS (Nah et al., 2019) datasets. The GOPRO dataset consists of 2103 training and 1111 test images with various dynamic motion blur. Similarly, the REDS dataset has 24000 training and 3000 validation data publicly available. For each dataset, every experiment was done in the same training environment. We mainly compare LPIPS (Zhang et al., 2018b) and NIQE (Mittal et al., 2012) perceptual metrics as our goal is to make images sharper. For more implementation details, please refer to the Appendix.
4.1 EFFECT OF SUPERVISED REBLURRING Loss
We implement the reblurring loss in varying degrees of emphasis on sharpness by controlling the reblurring module capacity. For a more balanced quality between PSNR and perceptual sharpness, ResBlock for MR- To put more weight the perceptual quality, we allocate a larger capacity on Mg by using ResBlooks. For notation simplicity, we denote the reblurring loss with l ResBlock(s)) in the reblurring module LReblur, nk-
Table 2 shows how the deblurring performance varies by the use of reblurring loss functions. With CRebiu nl, LPIPS and NIQE improves moderate degree while PSNR and SSIM metrics remain

---

Published as a conference paper at ICLR 2022
(a) B
(b) L1
(c) LVGG
(d) Lade
(e) LReMur
(1) Lachlur
Figure 4: Visual comparison of deblurred results by training loss function on GOPRO dataset. Upper: SRN, Lower: U-Net.
Method
LPIPS, NIQE,
PSNR
SSIM
Method
LPIPS, NIQE,
PSNR
SSIM
U-Net (baseline)
0.1635 5.996
29.66
0.8874
SRN (C1)
0.1246 5.252
30.62
0.9078
U-Net (Ceuu)
0.1301 5.132
29.47
0.8839
+0.001LAdv
0.1141
4.960
30.53
0.9068
+ CSturp with
0.1410
5.307
29.15
0.8694
+0.3Lvcu
0.1037
4.945
30.60
0.9074
+ Lshopp with
0.1238 5.124
29.44
0.8824
LR
0.1037 4.887
30.57
0.9074
Table 3: The effect of the sharpness preservation in
Table 4: Comparison of reblurring loss and other perceptual losses on GOPRO dataset applied to SRN.
training our reblurring module measured on GO- PRO dataset.
similar level. Mcanwhile, CReblur, n2 more aggressively optimizes the perceptual metrics. This is analogous to the perception-distorton trade-off witnessed in the image restoration literature (Blau & Michaeli, 2018; Blau et al., 2018). The perceptual metric improvements are consistently witnessed with various architectures on both the GOPRO and the REDS dataset.
4.2 EFFECT OF SHARPNESS PRESERVATION LOss
In training MR, we used both the blur reconstruction loss Lauu and the sharpness preservation loss CSharp- The latter term LSharp plays an essential role in letting MR concentrate only on the motion- driven blur in the given image and keep sharp image remain sharp. Table 3 presents the performance gains from using Cshar jointly with L in terms of the perceptual quality.
Also, the use the pseudo-sharp image is justified by comparing it with the use case of real sharp image S. We found the using S for Cshar with Cytur makes the training less stable than using S. Using the pseudo-sharp image confines the input data distribution of MM to the domain of MD outputs. For neural networks, it is very easy to discriminate real and fake images (Wang et al., 2020). By using fake image Š instead of real image S. let Mg focus the sharpness of an image and avoid being distracted by a more obvious difference between real and fake images. Furthermore, it leads the two loss terms L and Lshup reside under the same objective, amplifying any noticeable blur and keeping sharpness when motion blur is in zero-magnitude.
4.3 COMPARISON WITH OTHER PERCEPTUAL LOSSES
The reblurring loss provides conceptually different learning objectives from the adversarial and the perceptual losses and is designed to focus on the motion blur. Table 4 compares the effect of CReblur with adversarial loss LAdv. and the VGG perceptual loss (Johnson et al., 2016) by applying them to SRN (Tao et al., 2018) GOPRO dataset. While our method provides quantitatively better perceptual scores, the different perceptual losses are oriented to different goals. They do not necessarily compete conflict with each other and can be jointly applied at training to catch the perceptual quality in varying aspects. Figure 4, the effect of the reblurring loss is visually compared with the previous perceptual loss functions.

---

Published as a conference paper at ICLR 2022
0.300
at-EMIN -X aI-NQQE
Sharp
LPIPS
4.5
Test-time
0.105
Adaptation
TTA step
Iniria]
Deblur
5.0
0.115
L
5.1
B
0.120
TTA step
Initial
5.2
0.125
Baseline
30.45
30.55
30,65
30.75
Blurry
Figure 5: The proposed self-supervised test-time adaptation. The iterative optimization improves the image sharpness by finding an image that re- blurs to the current deblurred image.
Test-time adaption (SRN) on GOPRO dataset. Reblurring loss improves the trade-off between the perception (LPIPS, NIQE) and PSNR compared with the baseline.
Figure 6:
240 0212
240 0212
240 0212
(a) Blurry input B
(b) SE-Sharing
(c) DeblurGAN-v2
(d) Ours (TTA step 5)
Figure 7: Qualitative comparison between sfate-of-the-art deblurring methods on the GOPRO dataset. We used the SRN model baseline architecture.
4.4 EFFECT OF TEST-TIME ADAPTATION
We conduct test-time adaptation with the proposed self-supervised reblurring loss, Rebiur to make the deblurred image even sharper. Figure shows the -as-time--dapped result with SRN. Compared with the baseline trained with L1 loss, PSNR and the perceptual metrics, LPIPS and NIQE. Table 2 provides detailed quantitative test- time adaptation results on GOPRO and REDS dataset, respectively with various deblurring module architectures. The effect of test-time adaptation is visually shown in Figure 8.
results exhibit improved trade-off relations between
4.5 COMPARISON WITH STATE-OF-TIE--A METHOSS
We have improved the perceptual quality of the deblurred images by training several different model architectures. We compare the perceptual quality with the other state-of-the-a methods in Figure 7. Especially, DeblurGAN-v2 was trained with the VGG loss and the adversarial loss. Our results achieve visually sharper texture from the reblurring loss and test-time adaptation.
4.6 REAL WORLD IMAGE DEBLURRING
While our method uses synthetic datasets (Nah et al., 2017: 2019) for training, the trained models generalize real blurry images. In Figure 9, show deblurred results from Lai et al. (2016) dataset with DHN model. Compared with the baseline L, loss, our reblurring loss LReblur, a2 provides an improved deblurring quality. As the real test image could deviate from the training data distribution, a single forward inference may not produce optimal results. With the self-supervised test-tini time adaptation, deblurred images reveal sharper and detailed textures.
8

---

Published as a conference paper at ICLR 2022
(a) B
L1
(c) Rebbar
(d) TTA step 20
Figure 8: Qualitative comparison between different training objectives and the test-time adap- tation. Patches are sampled from the REDS dataset validation set.
178
110
219
770
AI
213
(a) Blurry input B
(b) Ci
(c) LRehbur a2 (Ours)
(d) + TTA step (Ours)
Figure 9: Qualitative comparison of deblurring results on the real-world images (Lai et al., 2016) by different loss functions and test-time adaptation. The proposed test-time adaptation greatly improves visual quality and sharpness of the deblurred images.
CONCLUSION
In this paper, we validate new observation that clean sharp images are hard to reblur and develop novel low-level perceptual objective terms, namely reblurring loss. The term is constructed to care the image blurriness by jointly training a pair of deblurring and reblurring modules. The supervised reblurring loss provides amplified comparison spects the blurriness a single image with the learned reblurring module. The self-supervision lets the deblurring module adapt to the image test time without ground truth. By applying the loss terms to state-of-the-a deblurring methods, we demonstrate our method consistently improves the the perceptual sharpness of the deblurred images visually as well as quantitatively.
motion blur while the self-supervised loss in-

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported in part by IITP grant funded by the Korea government [No. 2021-0- 01343, Artificial Intelligence Graduate School Program (Seoul National University)), and in part by AIRS Company in Hyundai Motor Company Kia Motors Corporation through HMC/KIA-SNU AI Consortium.
REFERENCES
Soonmin Bae and Frédo Durand. Defocus magnification. Computer Graphics Forum, 26(3):571-
579, 2007.
Yuval Bahat, Netalee Efrat, and Michal Irani. Non-uniform blind deblurring by reblurring. In ICCV, 2017.
Yochai Blau and Tomer Michaeli. The perception -iittrttoonnooooo tradeoff. In CVPR, 2018
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. The 2018 pirm challenge on perceptual image super-resolution. Proceedings of the ECCV Workshops, September 2018.
Tim Brooks and Jonathan T. Barron. Learning to synthesize motion blur. In CVPR, 2019.
Huaijin Chen, Jinwei Gu, Orazio Gallo, Ming-Y, Yu Liu, Ashok Veeraraghavan, and Jan Kautz, Re- blur2deblur: Deblurring videos via self-supervised learning. ICCP, 2018.
Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning, In CVPR, pp. 9137-9146, June 2021.
Sunghyun Cho and Seungyong Lee. Fast motion deblurring. In ACM SIGGRAPH Asia, 2009.
Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Igel. loss function for generative neural networks based on watson's perceptual model. In NeurlPS, 2020.
Mauricio Delbracio, Hossein Talebi, and Peyman Milanfar. Projected distribution loss for image enhancement. In ICCP, 2021.
Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR, 2019.
Dong Gong, Jie Yang. Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton van den Hen- gel, and Qinfeng Shi. From motion blur motion flow: deep learning solution for removing heterogeneous motion blur. In CVPR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Michael Hirsch, Christian Schuler, Stefan Harmeling, and Bernhard Schölkopf. Fast removal of non-uniform camera sshake. In ICCV, 2011.
Meiguang Jin, Zhe Hu, and Paolo Favaro. Learning to extract flawless slow motion from blurry videos. CVPR, 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Tae Hyun Kim and Kyoung Mu Lee. Segmentation-free dynamic scene deblurring, In CVPR, 2014.
Tae Hyun Kim, Byeongjoo Ahn, and Kyoung Mu Lee. Dynamic scene deblurring. In ICCV, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 141.6994/,.6980, 2014.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblur- GAN: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2022
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-vv: Deblurring (orders-of magnnitudd) faster and better. In JCCV, 2019.
Wei-Sheng Lai, Jia-Bin Huang, Zhe Hu, Narendra Ahuja, and Ming-Hsuan Yang. study for single image blind deblurring. In CVPR, 2016.
comparative
Christian Ledig, Lucas Theis, Ferenc Huszar. Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.
Anat Levin. Blind motion deblurring using image statistics. NIPS, 2006.
Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang. Learning a discriminative prior for blind image deblurring. In CVPR, 2018.
Roey Mechrez, Itamar Talmi, Firas Shama, and Lihi Zelnik-Manor. Maintaining natural image statistics with the contextual loss. In ACCV, pp. 427-443 Springer, 2018.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self- supervised photo upsampling via latent space exploration of generative models. In CVPR, 2020.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Olcksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint
2017.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE SPL, 20(3):209-212, 2012.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
Seungjun Nah, Sungyong Baik, Scokil Hong. Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. NTIRE 2019 challenges video deblurring and super-resolution: Dataset and study. In CVPR Workshops, 2019.
Yuesong Nan and Hui Ji. Deep learning for handling kernel/model uncertainty in image deconvolu- tion. In CVPR, 2020.
T. M. Nimisha, Akash Kumar Singh, and A. N. Rajagopalan. Blur-invariant deep learning for blind- deblurring. In ICCV, 2017.
Mehdi Noroozi, Paramanand Chandramouli, and Paolo Favaro. Motion deblurring in the wild. In GCPR, 2017.
Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In CVPR, 2016.
Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal train- ing. In ECCV, 2020.
Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo. Neural blind deconvo- lution using deep priors. In CVPR, 2020.
Christian Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. Learning to deblur. IEEE TPAMI, (837))1439-111 1451, 2015.
Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In ICCV, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X9v 2014.
Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held cameras. In CVPR, 2017.

---

Published as a conference paper at ICLR 2022
Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentivee patch patch-hierarchical net- work for adaptive motion deblurring. In CVPR, 2020.
Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. Learning a convolutional neural network for non-uniform motion blur removal. In CVPR, 2015.
Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. Edge-based blur kernel estimation using patch priors. In ICCP, 2013.
Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revealing deep video super- resolution. In ICCV, 2017.
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018.
Taimoor Tariq, Okan Tarhan Tursun, Munchurl Kim, and Piotr Didyk. Why are deep representations good perceptual quality features? In ECCV, pp. 461-461. Springer, 2020.
Sheng-Yo Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. CNN- Generated images are surprisingly easy to spot... for now. In CVPR, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility structural similarity. IEEE TIP, 13(4):600-612, 2004.
Oliver Whyte, Josef Sivie, Andrew Zisserman, and Jean Ponce. Non-uniform deblurring for shaken images. IJCV, 1682))668-188) 2012.
Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural L0 sparse representation for natural image deblur- ring. In CVPR, 2013.
Yuan Yuan, Wei Su, and Dandan Ma. Efficient dynamic scene deblurring using spatially variant deconvolution network with optical flow guided training. In CVPR, 2020.
Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi- patch network for image deblurring. In CVPR, 2019.
Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson W.H. Lau, and Ming- Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In CVPR, 2018a.
Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In CVPR, 2020.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018b.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
In this appendix, we provide the implementation details and additional experimental analysis. In Section B, explain the implementation details with the model architecture specifics, training de- tails, and the evaluation metrics. Section describes how the reblurring module design and the size determined. Then in Section D, describe the different characteristics of the proposed reblur- ring loss and the other perceptual losses. We combine our reblurring loss with the other perceptual losses take advantage in multiple perspectives. In Section B, the test-time adaptation algorithm is described. In Section F, we show the quantitative effect of test-time adaptation and show the trade- off relation between the conventional distortion quality metric (PSNR, SSIM) and the perceptual metrics (LPIPS, NIQE) compared with the baselines.
B IMPLEMENTATION DETAILS
Model Architecture. the main manuscript, we mainly performed the experiments with 3 different model architectures. First, we set our baseline model as light-weight residual U-Net architecture that runs fast speed. The baseline model is used to design our reblurring loss with pseudo-sharp images through ablation study in Table 3.
For reblurring operation, we use simple residual network MR without strides to avoid deconvolu- tion artifacts. The baseline U-Net and the reblurring module architectures are shown in Figure A. The detailed parameters for U-Net and Mg are each specified in Table and C.
8 ResBlocks
MD
Blur
Deblurred
(U-Net)
ResBlocks
MR
Deblurred
Reblurred
Conv
ResBlock
ResBlock
Conv-Transpose
ReLU
Figure A: The baseline U-Net architecture and the reblurring module architecture We use the same reblurring module for all experiments except the number of ResBlocks. In addition the U-Net, experiments
conducted with state-of-the-an deblurring models based on SRN (Tao et al., 2017) and DMPHN (Zhang et al., 2019). SRN (Tao et al., 2018) was originally designed operate grayscale images with LSTM module. Later, the authors released the
13

---

Published as a conference paper at ICLR 2022
#
Layer description
Output shape
Input
3 H x W
5 x 5 conv
64 x H x W
3 conv
128 H/2 W/2
3
3 3 conv
192 H/4 W/4
4-19
8 ResBlocks (3 x 3) conv
192 x H/4 X W/4
20
128 H/2 W/2
21
3 3 conv
64 H W
22
5 x 5 conv
3 x H x W
Table A: U-Net module specifics
Method
LPIPS,
NIQE,
PSNR
SSIM
DMPHN (Ci only)
0.1184
5,542
0.9191 31.42 31.53
DHN (Ei only)
0.1179
5.490
0.9207
Table B: DMPHN modification results on GOPRO dataset. DHN without patch-wise wise convolution brings improved accuracy.
sRGB version code without LSTM, exhibiting improved accuracy. We adopted the revised SRN structure in our experiments.
The other model chose is based DMPHN (1-2-4-8) (Zhang al., 2019). DMPHN performs hierarchical residual refinement to produce the final output. The model consists convolutional layers with ReLU activations that are spatially shift-equivariant. In Zhang et al. (2019), each level splits the given image and performs the convolutional operation on the divided patches. As the convolutional weights do not differ by the input patches, the operations do not necessarily have to be done patch-wise. Thus, we remove the multi-pate strategy and perform the convolution on the whole given input without dividing the image into patches. We refer the modified model DHN. As shown in Table B, convolution on the whole image compared with patch-wise convolution brings higher accuracy.
Metrics. To quantitatively compare the deblurred images in the following sections, we use PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018b), and NIQE (Mittal et al., 2012). In the image deblurring literature, SSIM has been measured by MATLAB ssim function on sRGB images with x W X C. SSIM was originally developed for grayscale images and MAT- LAB ssim function for 3-dimensional tensor considers image to be 3D grayscale vol- ume image. Thus, most the previous SSIM measures were not accurate, leading to higher values. Instead, measured all the SSIM for each channel separately and averaged them. We used sk mage. met r ics. st ructural a _as imá larity function in the scikit-image package for python to measure SSIM for multi-channel images.
Training. For all the experiments, we performed the same training process for fair comparison. On the GOPRO dataset (Nah al., 2017), trained each model for 4000 epochs. On the REDS dataset (Nah et al., 2019), the models are trained for 200 epochs. Adam (Kingma & Ba, 2014) optimizer used in all cases. When calculating distance between images with Lp norm, we always set 1, using L1 distance. Starting from the initial learning rate x 10 44 the learning rate halves when training reaches 50%, 75%, and 90% of the total epochs. We used PyToreh 1.8.1 with CUDA 11.1 to implement the deblurring methods. Mixed-precision training (Micikevicius et al., 2017) is employed to accelerate operations on RTX 2080 Ti GPUs.
C DETERMINING THE REBLURRING MODULE SIZE
As reblurring loss La is realized by MR. the reblurring module design plays essential role. As shown in Figure A, the Mg architecture is a simple ResNet. Table D shows the relation between the deblurring performance and MR size by changing the number of ResBlocks.
For all deblurring module MD architectures, LPIPS was the best when the number ResBlocks,
= 2. NIQE showed good performance when 2 n < 3. PSNR and SSIM had tendency to
14

---

Published as a conference paper at ICLR 2022
#
Layer description
Output shape
Input
3 x H x W
5 x 5 conv
64 x X W
2-5
ResBlocks (5 x 5)
64 W
6
5 5 conv
3 H X W
Table C: Reblurring module specifics
Method
LPIPS,
NIQE,
PSNR
SSIM
U-Net (Ci only)
0.1635
5,996
29.66
0.8874
CRasur
0.1365
5.629
29.58
0.8869
CRasus
0.1238
5.124
29.44
0.8824
CRasaus
0.1386
5,448
29.38
0.8819
CRebbur.
0.1415
5.513
29.25
0.8789
SRN (C1 only)
0.1246
5.252
30.62
0,9078
CRebbur.
0.1140
5,136
30.74
0.9104
Rabbur
0.1037
4.887
30.57
0,9074
Rebbur
0.1091
4.875
30.50
0,9060
CRbblur. m4
0.1155
5.041
30.53
0,9056
DHN (Li only)
0.1179
5.490
31.53
0.9207
CRaaus
0.0975
5,472
31.53
0.9217
CRavur
0.0837
5,076
31.34
0.9177
+ CResaur
0.0845
4,963
31.26
0.9159
CResbur. 24
0.0861
5.041
31.19
0.9149
Table D: The effect of reblurring loss on GOPRO dataset by the reblurring module size, Re- blurring module size varies by the number of ResBlocks.
decrease when 2 1. For larger number of ResBlocks, we witnessed sharper edges could be obtained but sometimes, cartoon artifacts with over-strong edges were witnessed.
Considering the trade-off between the PSNR and the perceptual metrics, we chose {1,2} in the following experiments. n finds balance between the PSNR and LPIPS and n = puts more weight the perceptual quality.
D COMBINING REBLURRING LOSS WITH OTHER PERCEPTUAL LOSSES
Our reblurring loss is new perceptual loss that sensitive blurriness of an image, a type of image structure-level information while other perceptual losses such as VGG loss (Johnson et al., 2016) and adversarial loss (Ledig al., 2017) are more related the high-level contexts. As VGG model (Simonyan & Zisserman, 2014) is trained to recognize image classes, optimizing with VGG loss could make an image better recognizable. In the GAN frameworks (Goodfellow et al., 2014), is well known that discriminators can easily tell fake images from real images (Wang et al., 2020), being robust against JPEG compression and blurring. In the adversarial loss from the discriminator, the realism difference could be more salient than other features such blurriness.
With the perceptual loss functions designed with different objectives, combining them could bring visual quality improvements in various aspects. Tables E and F show the effect of applying our reblurring loss jointly with other perceptual losses on GOPRO and REDS datasets. We omit the loss coefficients for simplicity. We used weight 0.3 for the VGG loss LVGg and 0.001 for the adversarial loss. Aav- witness LPIPS and NIQE further improves when our reblurring loss combined with LVaG LAde-
15

---

Published as a conference paper at ICLR 2022
Method
LPIPS,
NIQE,
PSNR
SSIM
SRN (Ci only)
0.1246
5.252
30.62
0,9078
+ Lvon
0.1037
4.945
30.60
0,9074
+ Evaa + CRaaI
0.0928
4.671
30.64
0,9079
+ Ladr
0.1141
4.960
30.53
0,9068
+Ladr + CResbus, a2
0.1014
4.811
30.56
0,9075
DHN (C1 only)
0.1179
5,490
31.53
0,9207
+LvGs
0.0994
5.022
31,48
0,9195
+Lvas + LRassur, + +Las
0.0773
4.897
31.28
0,9161
0.0969
5,026
31.46
0,9188
+ C + CRebbur.
0.0835
4.799
31.28
0.9162
Table E: Results on GOPRO dataset by adding reblurring loss to the other preceptual losses.
Method
LPIPS,
NIQE,
PSNR
SSIM
SRN (Ci only)
0.1148
3,392
31.89
0.8999
+voG
0.1000
3.256
31.86
0,9001
+vvo + CRablur.
0.0868
2.835
31,83
0,9015
0.1158
3.395
31.84
0.8993
-Las + CRavuur
0.0934
2,836
32.00
0.9061
DHN (L1 only)
0.0942
3,288
32.65
0.9152
+LvGG
0.0812
3.171
32.61
0.9146
+LvaG CRaau
0.0723
2.821
32.48
0,9133
+ L
0.0956
3.218
32.58
0,9128
Lade + LRebbur.
0.0820
2.809
32.45
0.9121
Table F: Results REDS dataset by adding reblurring loss to the other preceptual losses.
E TEST-TIME ADAPTATION DETAILS
We describe the detailed self-supervised test-time adaptation process. test time, the learning rate is set to iteratively minimized by updating the weights of Mp. As the self-supervised loss in equation only cares about image sharpness, the image may have color drifting issues. Thus, finally, we match the histogram of the updated image I to the histogram of L The overall process shown in Algorithm A.
10 From the initial deblurring result L the self-supervised loss is
Algorithm A Optimization process in test-time adaptation
1: procedure TEST-TIME ADAPTATION(B, Mp, MR)
2: Test-time learning rate μ + 3 x 10 10-6
3: Op f Weights Mp.
4: I (B)
5: for = O...N do
6:
L MD (B).
7:
Fechlur = M|MM(MMBB) - Lill
8:
Update Op by Vo, Cselr and J.
LN = MD (B).
9:
IN wwwsstogram.attchhnngggeeooog
Adapted L.N
L2)
10: return
Adapped
F PERCEPTION VS. DISTORTION TRADE-OFF
known image restoration literature that the distortion error and the perceptual quality error in trade-off relation (Blau & Michaeli, 2018: Blau et al., 2018). The relation is often witnessed by training a single model with different loss functions. most cases, to obtain a better perceptual quality from single model architecture, retraining with another loss from scratch necessary. Our
16

---

Published as a conference paper at ICLR 2022
test-time adaptation from self-supervised reblurring loss, in contrast, can provide the steps toward perceptual quality without full retraining.
In Figure B and C, present the perception -isttortinn trade-off from test-time adaption. LPIPS and NIQE scores consistently improve from each adaptation step in both SRN and DHN models. While PSNR is moderately sacrificed from the adaptation, SSIM improves in the early steps it more reflects the structural information. Our results show improved trade-off between the distortion and perception metrics over the baseline models trained with LI loss.
0.100
LPIPS
LPPS
NIQ
LPPS
nd: NIQE
4.8
0.305
TTA step 1
L21PS
0.110
Intiaal
5.0
TtAstep
Initial
0.115
TTA step 5
5.1
x
0.120
Initial
TTA step
5.2
Initial
0.125
Baseline
Baseline
30.45
30,50
30.55
30.60
30.65
30.70
30.75
30.15
30.50
30.55
30.60
30.65
30.70
30.75
(a) PSNR vs LPIPS
(b) PSNR NIQE
0.300
NIQNIQ 22: NIQE
LPPS
4.8
LPPSS
02) NIQE
0.305
TTA step
4.9
LPIPS
0.110
Initial
N108
5.0
Initial
step
0.115
TTA s
0.120
TTA stop
0.125
Baseline
Initial
5.2
Baseline
Initill
0.907
0.908
0.909
0,910
0.911
0.907
0.905
0.909
0.10
0.911
SSIM
SSIM
(c) SSIM vs LPIPS
(d) SSIM NIQE
Figure B: perception-distoriin trade-off' from test-time adaptation applied to SRN model on GOPRO dataset.
LPESS
nl: NIQU NIQE
3.0
TTA step
TTA step3
Initial
62 LIES
LPIPS
Initial
n/: NIQE
5.2
0.10
5.3
stop
5.4
Baselino
5.5
step
Baseline
31.35
31,40
31.45
31.
31.30
31.35
45
31.50
PSNR
PSNR
(a) PSNR vs LPIPS
(b) PSNR vs NIQE
0.08
TTA step 3
- 21 NIQE
- xt NIQE
LPIS
NIQE
LOISS
NI0E
Intiall
TTA step
5.1
Initial
1PPS
5.2
0.10
TTA step
Inital
3.3
TTA STAs
0.11
5.4
Baseline
5.5
Buseline line
0.918
0.919
0,920
0.921
0.922
0.918
0.919
0.920
0.921
n922
SSIM
SSIM
(c) SSIM vs L.IPS
(d) SSIM vs NIQE
Figure C: Perception-distortion trade-off from test-time adaptation applied DHN model on GOPRO dataset.
17

---

Published as a conference paper at ICLR 2022
On GOPRO dataset
On REDS dataset
Model
Optimization
LPIPS|
NIQE,
PSNR
SSIM
LPIPS,
NIQE,
PSNR
SSIM
L1
0.1246
5.252
30.62
0.9078
0.1148
3.392
31.89
0.8999
+ Lxeblur, al
0.1140
5.136
30.74
0.9104
0.1071
3.305
32.01
0.9044
TTA step 5
0.1101
5.079
30.60
0.9100
0.1029
3.278
31.83
0.9040
+ TTA step 10
0.1103
5.036
30.11
0.9048
0.1025
3.261
31,29
0.8996
TTA step
0.1223
4.968
28.44
0.8806
0.1116
3.265
29.59
0.8807
SRN
TTA step 30
0.1470
4.924
26.42
0.8411
0.1306
3.301
27.73
0.8523
L1 + Cuablur
0.1037
4.887
30.57
0.9074
0.0947
2.875
31,82
0.9026
TTA step 5
0.0983
4.730
30.44
0.9067
0.0909
2.798
31,50
0.9008
TTA step 10
0.0962
4.569
30.07
0.9024
0.0913
2.741
30.87
0.8945
TTA step 20
0.1021
4.274
28.83
0.8836
0.1033
2.699
29.09
0.8697
TTA step 30
0.1199
4.045
27.26
0.8529
0.1259
2.729
27.20
0.8326
Table G: Quantitative analysis of the reblurring losses and test-time adaptation applied to SRN on GOPRO and REDS datasets.
G TEST-TIME ADAPTATION EFFECTS
In Table G, quantitatively compare the deblurred results from test-time adaptation in terms of no-reference metrie, NIQE, and reference- based metrics, LPIPS, PSNR, and SSIMs. By performing TTA up 30 steps as described Algorithm A, show that LPIPS and NIQE could be improved to a degree. On both GOPRO and REDS datasets, NIQE has a tendency to improve further after LPIPS has stopped its improvement. This due to the self-supervised nature of test-time adaptation that considers image sharpness without reference.
In Figures and E, we visually show the effect of time adaptation applied SRN with jointly trained reblurring module. By test-time adaptation, our model further improves the sharp edges of the images. In Figure D, the building structures and the horizontal lines are better witnessed. Also in Figure E, the vehicle's pole are better recovered and the text are clearer. While the PSNR and SSIM have decreased by test-time adaptation Table G, perceptually, the results from test-time adaptation tend to be sharper.
18

---

Published as a conference paper at ICLR 2022
(a) Blur
(b) Our deblurred image (TTA step 30)
Blur B
(d) L1
Ci + CRahier al
(f) Ours (TTA step 30)
Figure D: Visual comparison of deblurred results by reblurring loss and test-time adaptation on GOPRO dataset.
(a) Blur
(b) Our deblurred image (TTA step 30)
ZABITA 153
ZABITA 153
ZABITA 153
(e) Ca + CResier at Figure E: Visual comparison of deblurred results by reblurring loss and test-time adaptation on REDS dataset.
Blur B
(d) La
(f Ours (TTA step 30)
19


Reviews:
[{"review_id": "cFyM46PyPL", "reviewer": "Reviewer_oSPE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper is based on an interesting observation. The proposed formulation/implementation seems to take advantage of the observation.\n\nStrengths:\n- The main idea of the paper is interesting and the proposed formulation seems to take advantage of the observation.\n- Empirical results show that the method produces better results than the compared methods\nThe idea of using the re-blurring network to mitigate the out-of-distribution problems seems interesting.\n\nWeaknesses:\n- To implement the idea, the method introduces several loss terms that seem to be needed to avoid degenerate cases and other issues. In this sense, it feels that the formulation could be a little more elegant. The overall idea is that a deblurred image should be close to the target and also shouldn't have information about the original blur. In the end, the whole system seems to be doing this but I wonder if all the components are needed.\n\n- The paper idea seems to be limited to deblurring. But in many parts the paper claims that the ideas are \"applicable to general learning-based approaches\" If this is one of the claims, there needs to be evidence supporting this.\n\n- Analysis. There are many experiments but there's not too much analysis. From the exposition it is hard to claim that the method is actually improving over more classical losses like content loss (vgg loss, as shown in Table 4). So the question is more like if this method is better than simpler approaches? Also, if the issue is remaining blur, can't we just re-iterate the deblurring model till we reach a fixed-point?\n\n\nOther comments:\n\n- \"inherent limitation of PSNR-oriented solutions\" Do you mean the \"regression-to-mean\" problem that happens when we train with reference-based losses? This needs to be better discussed. \n\nThere are other reference based perceptual losses and relevant work that could be added to the overall discussion (not asking for more experiments but this could help improving the related work section):\n\nMechrez, R., Talmi, I., Shama, F. and Zelnik-Manor, L., 2018, December. Maintaining natural image statistics with the contextual loss. In Asian Conference on Computer Vision (pp. 427-443). Springer, Cham.\n\nM. Delbracio, H. Talebei and P. Milanfar,  \"Projected Distribution Loss for Image Enhancement,\" in 2021 IEEE International Conference on Computational Photography (ICCP), Haifa, Israel, 2021\n\nTariq, T., Tursun, O.T., Kim, M. and Didyk, P., 2020, August. Why Are Deep Representations Good Perceptual Quality Features?. In European Conference on Computer Vision (pp. 445-461). Springer, Cham.\n\nCzolbe, S., Krause, O., Cox, I. and Igel, C., 2020. A Loss Function for Generative Neural Networks Based on Watson’s Perceptual Model. Advances in Neural Information Processing Systems, 33.\n\n- Figure 1. This figure is unclear, please include an explanation for what is given on the top and bottom rows for each of the methods.\n\n- \"From a deblurred output, the reblurring module tries to make the reblurred image close to the original\" So in practice this reblurring network is the inverse of the deblurring module. Why not present this module in this way?\n\n- Writing is a little disorganized. For example in the introduction when referring to the \"reblurring loss\" it is unclear what is \"the difference\" that is mentioned. One of the listed contribution depends on this \"reblurring loss\" so this needs to be better explained in the introduction.\n\n- The need for the pseudo-sharp image \\hat{S}. The argument for using \\hat{S} and not S seems to be rather empirical so this should be supported by evidence. In the current presentation there's a lot of emphasis on measuring sharpness but not realism, but then later for evaluating the quality perceptual metrics such as LPIPS and NIQE are used. So, there seems to be a mismatch between what is claimed and what is actually happening. \n\n- The test-time adaptation inverts the re-blurring network. This needs more analysis in terms of: \nWhat happens if we minimize this loss (gradient descent till convergence)? How many gradient descent steps are needed. This is not discussed much in the paper and I found it to be an interesting contribution. It will be also helpful to show cross-datasets performances (trained with GoPro and evaluated with a different dataset, e.g. HIDE, Real-Blur or REDS).\n\n- Perceptual Distortion tradeoff. The perceptual distortion trade-off balances a reference based metric against a non-reference metric (e.g., distance to the natural image manifold). PSNR and LPIPS are both reference based metrics so it doesn't make sense (formally) to refer to this type of plot as a distortion-perception trade-off. Maybe the plot should be made with NIQE instead of LPIPS.\n", "summary_of_the_paper": "This work addresses the problem of image deblurring by training a deep model in an end-to-end fashion. Similar to recent work, the method makes use of large paired (blurry,sharp) training datasets to train the deblurring network.\n\nThe main idea of the work is that a correctly deblurred image should not contain any information about the original blur. The method exploits this observation by using an auxiliary network that tries to re-blur the deblurred image so it has the same blur as the original one. If the deblurred image was perfectly deblurred this shouldn't be possible.\n\nThese two networks end up having a \"similar\" role as a generator-discriminator on a GAN set-up. But even if there's this superficial connection with GANs the formalism is completely different. Here the reblurring network outputs an image that is compared to another image (i.e., reference-based).\n\nThe work introduces three different loss terms: L_blur, L_sharp and L_reblur as long as the typical regression loss L1 between deblurred and sharp images.\n\n- L_blur: is mainly used to train the re-blurring module;\n- L_sharp: is used to avoid the re-blurring module to just apply the average blurring;\n- L_reblur: is used as a regression loss on the space of re-blurred images.\n\nAdditionally, the paper proposes a test time adaptation where the reblurring network is used to boost the deblurring results on a given image. This is done by approximately inverting the re-blurring network using gradient descent. \n\nThe work presents several experiments including ablation studies with two of the popular (synthetic) datasets used in image/video deblurring (GoPro and REDS). Several comparisons with SOTA methods and some results on a real image dataset (Lai et al. 2016). Different quantitative metrics are given (PSNR,SSIM, LPIPS, NIQE) as well as several figures showing visual comparisons.", "summary_of_the_review": "This paper introduces an interesting idea for improving image deblurring models: the deblurred image should not contain information/traces about the original blur.  This is a bold statement. The paper crystalizes this idea using two networks (deblurring and re-blurring network) that are co-trained with different loss terms. The implementation seems (a little) too complex with many terms that try to avoid different types of artifacts/degenerative cases. But in the end this seems like a valid implementation of the initial idea. The current analysis didn't convince me that the method is doing much better or better than other methods that just compare the restored and sharp images on a pre-computed feature space (is this a matter of tuning the contributions of each loss term?). I think the paper needs to better discuss and analyze this. Additionally, the idea seems to be specific for dealing with blur. Can the same idea apply to other (restoration) tasks?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rezub_FSUIe", "reviewer": "Reviewer_Gxxe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting idea. For the task of deblurring, it represents a logical and novel formulation of the “adversary”: rather than just saying whether the input is truly sharp or the output of a deblurring network, the reblurring network has to reproduce the original blurry image in the latter case.\n\nWhile the results are encouraging, there are a number of issues with the paper:\n\nThe paper needs to do a much better job in clearly and systematically establishing the benefit of the proposed method against a standard adversarial loss and other perceptual losses (which are much simpler and more straightforward than the proposed reblurring loss). In particular, the efficacy of the network WITHOUT test-time adaptation should be established. This is because TTA is expensive, and it is not clear whether other methods couldn’t also be improved with TTA (for example, those based on standard adversarial losses) or by just having slower but deeper networks.\n\nTable 4 seems to suggest minimal additional benefit of the reblur loss (without TTA). There is some benefit in NIQE, none in LPIPS, and PSNR and SSIM get worse. This comparison is also to a specific weight on the adversarial loss. Did the authors sweep for different loss weights to find the optimal one for training only with an adversarial (+ VGG) loss? This is what I mean by a systematic evaluation: the paper should find the optimal hyperparameters for training with only a regular adversarial loss and compare to the optimal setting for the proposed method — without TTA and using a comparable architecture for the discriminator as the proposed MR.\n\n\nThe paper also needs more comprehensive results reported for comparisons to state-of-the-art. There are no quantitative results, and the qualitative results are only with TTA. Visually, the improvement of the proposed method with TTA over DeblurGAN is relatively small. It would be natural to wonder if that improvement disappears without TTA, and if infact, the proposed method does worse than DeblurGAN. As noted, TTA is expensive, and an orthogonal approach that could potentially be applied to other adversarially trained deblurring networks.\n\nThe paper, especially the introduction, could be a lot clearer in explaining the basic idea. Figure 1 is great in terms of layout, but the captions / legend is not informative making it very unintuitive.  The caption should at the very least explain what L, S, \\hat{S}, and B are without having readers to dig through the text in section 3. The “shared” is confusing — one would anyway assume that all M_R’s are the same, but the fact there’s shared lines only among pairs on the left and right makes us question if the two pairs are different from each other. Having (a) and (b) in Figure 1 with the corresponding headings is also confusing. They each respectively describe the training loss for M_R and M_D, and if I understand correctly, each iteration involves doing an update step of each. This does not come across from the figure. And finally, some more details of the test-time adaptation should be included in the main paper. Most of the text in the main paper in that section gives “intuition” — but is very confusing to read since the reader doesn’t know exactly what is being optimized and what is being updated (image or network weights).\n\n\nA lot of the explanations in the paper are very hand-wavy, and not at all clear to the reader. This is especially true about the use of a pseudo sharp image instead of a sharp image: for example, I don’t understand what the authors mean by “In contrast to S that differ from the deblurred image L by the realness, \\hat{S} can avoid being MR distracted by such an unintentional difference, focusing on image sharpness.” I’m actually not sure why using S-hat doesn’t just cause a degenerate solution where MD produces oversmoothed outputs all the time to prevent MR from figuring out if the original image was blurry or not.", "summary_of_the_paper": "The paper proposes a novel approach to using deep networks for image deblurring. Since training with simply reconstruction losses usually leads to oversmoothed results, recent works have looked at using perceptual and adversarial losses.\n\nThe paper proposes an approach similar to an adversarial loss. If an image is not de-blurred entirely, it leaves within itself some tell-tale signs of the original blur. The paper proposes learning a “reblurring” network that given a “deblurred” image should yield the original “blurry” image, while given a true sharp image should output the image itself. The deblurring network is then trained in a sense to fool the re-blurring network, by generating images that would be left untouched by the reblurring network. The paper also proposes a “test time adaptation” approach.", "summary_of_the_review": "Given the issues with the experiments and presentation above, I don't think the paper is ready for publication.\n\n### Post-rebuttal\n\nIncreasing score from reject to borderline/marginal  reject. I think the paper still needs a bit more work, but would not object to it being accepted. Please see detailed comment in response to rebuttal below.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4HmAhN7fbtS", "reviewer": "Reviewer_M9xc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Technical novelty\nThe observation that the \"clean images are hard to reblur\" is quite novel. \nFollowing the observation, the proposed reblurring loss is logical and experimental results are consistent. \nI'm concerned that the concept of deblur-reblur is similar to the GANs, but the authors well addressed the issue in the introduction. \n\n2. Writing\nOverall it is well-written and easy to read. However, more proofreading and re-arranging figures are needed before publication. \nIn Figure 1, the sharp image (supposed to be ground truth) and deblurred images are located in the top row. It may be a reasonable arrangement since they are seemingly similar. However since they have different meanings, there should be a better arrangement for easier understanding. \nReaders may be familiar with the blurry and deblurred image, but not the reblurred image since it is quite a new concept. So additional figures explaining and emphasizing such concepts would help the understanding. \nIn Figure 2, there are B and S in both of Figure 2 (a) and (b), where the arrangement is somewhat confusing. \nTable 1 is mentioned just below the equation (1) but there is no explanation about M_D before Table 1. \n\n3. Experimental results\nExperimental results improve SOTA in perceptual measures such as LPIPS and NIQE. However, it did not improve PSNR and SSIM, which gives the impression that the deblurred image may distort the original image and obtain better perceptual measures by sharpening. ", "summary_of_the_paper": "A novel deblurring method is proposed by introducing the concept of reblurring loss. Conventional techniques deblur blurry images to some extent however there are some unremoved blur contents, where it can be used to reconstruct (reblur) blurry images. On the other hand, cleanly deblurred images have only sharp contents and it is difficult to reblur the images since no blurry cues are left. By using such observation, a reblurring module and a deblurring module with reblurring loss are proposed. Experimental results show that the proposed framework outperforms SOTA quantitatively and qualitatively on the GOPRO and REDS dataset. ", "summary_of_the_review": "The paper introduces an interesting observation and proposes novel modules based on the observation where the development is very logical. Overall, the paper is easy to read, however more proofreading is required. Experimental results improve the SOTA but there should be justification for some results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an idea that was found interesting by all reviewers (including Gxxe who recommends a marginal reject). A majority of the reviewers also point out a few weaknesses of the paper, notably in terms of clarity of several statements that were found to be hand-wavy (see the reviews of Gxxe and oSPE for more precise details). The area chair agrees with those statements, but overall, the originality of the idea introduced in this paper outweighs these weaknesses, and the experimental study is conducted in a reasonably convincing manner.

Even though there is room for improvements, the area chair is happy to recommend an accept, but encourages the authors to follow the constructive feedback provided by the reviewers for the camera-ready version.
Decision: Accept (Poster)