Year: 2021

Abstract:
The goal of dynamic scene deblurring is to remove the motion blur in a given image. Most learning-based approaches implement their solutions by minimizing the L1 or L2 distance between the output and reference sharp image. Recent attempts improve the perceptual quality of deblurred images by using features learned from visual recognition tasks. However, those features are originally designed to capture high-level contexts rather than low-level structures of a given image, such as blurriness. Instead, we propose a more direct way to make images sharper by exploiting the inverse task of deblurring, namely, reblurring. Reblurring amplifies the remaining blur to reconstruct the original blur, however, a well-deblurred clean image with zero-magnitude blur is hard to reblur. Thus, we design two types of reblurring loss functions for better deblurring. The supervised reblurring loss at training stage compares the amplified blur between the deblurred and the sharp reference. The self-supervised reblurring loss at inference stage inspects if noticeable blur still remains in the deblurred image. Our experimental results on large-scale benchmarks and real images demonstrate the effectiveness of the reblurring losses in improving the perceptual quality of the deblurred images in terms of NIQE and LPIPS scores as well as visual sharpness.

heading: 1 INTRODUCTION
text: Motion blur is a common photographic artifact that occurs when the camera moves and the scene changes during the exposure in dynamic environments. Dynamic scene deblurring is a challenging ill-posed problem as both the locally-varying blur kernel and the latent sharp image have to be found from a large solution space. Traditional approaches (Hirsch et al., 2011; Whyte et al., 2012; Kim et al., 2013; Kim & Lee, 2014) tried to alleviate the ill-posedness by using prior knowledge on the statistical properties of sharp images, such as gradient sparsity.
Instead of using such handcrafted knowledge, recent methods take advantage of large-scale datasets as well as deep neural networks (Nah et al., 2017; Su et al., 2017; Noroozi et al., 2017; Nah et al., 2019; Shen et al., 2019). Usually, the learning is driven by minimizing the pixel-wise distance to the ground truth, e.g., L1 or L2, so that the PSNR between the deblurred and the sharp reference can be maximized. By utilizing modern ConvNet architectures and training techniques, state-of-the-art approaches (Nah et al., 2017; Tao et al., 2017; Gao et al., 2019; Yuan et al., 2020; Park et al., 2020; Chi et al., 2021) have been developed toward higher capacity and deblurring accuracy. Still, most methods tend to suffer from the blurry predictions due to the regression-to-mean behavior often witnessed in ill-posed problems with large solution space (Ledig et al., 2017; Menon et al., 2020).
To overcome limitations of the conventional objectives, concepts of perceptual (Johnson et al., 2016) and adversarial (Ledig et al., 2017; Nah et al., 2017; Kupyn et al., 2018) loss terms from highlevel semantic tasks have been introduced to improve the visual quality of the deblurred results. Nevertheless, such high-level losses may not serve as optimal goals for blur removal as low-level structural properties, e.g., blurriness, are not the primary features considered in their formulations. As illustrated in Figure 1, results from the previous deblurring methods are still blurry to a degree and the VGG and the adversarial losses are not sufficient to obtain perceptually pleasing and sharp images across different architectures (Tao et al., 2018; Gao et al., 2019; Kupyn et al., 2019).
While the deblurred images look less blurry compared with the original input, it is still possible to find nontrivial blur kernels with directional motion information. From the observation, we introduce the concept of reblurring which amplifies the unremoved blur in the given image and reconstructs the original blur. We note that our reblurring operation aims to recover the original motion trajectory in the blurry input, rather than to synthesize arbitrary, e.g., Gaussian, blurs. Therefore, an ideally deblurred clean image is hard to reblur as no noticeable blur can be found to be amplified, making reblurring an ill-posed task. In contrast, it is straightforward to predict the original shape of blur from insufficiently deblurred images as shown in Figure 1. We propose to use the difference between non-ideally deblurred image and the ideal sharp image in terms of reblurring feasibility as the new optimization objective, reblurring loss for the image deblurring problem.
The reblurring loss is realized by jointly training a pair of deblurring and reblurring modules. The reblurring module performs the inverse operation of deblurring, trying to reconstruct the original blurry image from a deblurred output. Using the property that the blurriness of a reblurred image depends on the sharpness quality of the deblurred result, we construct two types of loss functions. During the joint training, supervised reblurring loss compares the amplified blurs between the deblurred and the sharp image. Complementing L1 intensity loss, the supervised reblurring loss guides the deblurring module to focus on and eliminate the remaining blur. While our training strategy is similar to the adversarial training of GANs (Goodfellow et al., 2014) in a sense that our deblurring and reblurring modules play the opposite roles, the purposes and effects of the adversary are different. The reblurring loss concentrates on image blurriness regardless of image realism. Furthermore, in contrast to the GAN discriminators that are not often used at test time, our reblurring module can be used to facilitate self-supervised reblurring loss. By making the deblurred image harder to reblur, the deblurring module can adaptively optimize itself without referring to the ground truth.
Our reblurring loss functions provide additional optimization directives to the deblurring module and can be generally applied to any learning-based image deblurring methods. With the proposed approach, we can derive sharper predictions from existing deblurring methods without modifying their architectures. We summarize our contributions as follows:
• Based on the observation that clean images are hard to reblur, we propose novel loss functions for image deblurring. Our reblurring loss reflects the preference for sharper images and contributes to visually pleasing deblurring results. • At test-time, the reblurring loss can be implemented without a ground-truth image. We perform test-time adaptive inference via self-supervised optimization with each input. • Our method is generally applicable to any learning-based methods and jointly with other loss terms. Experiments show that the concept of reblurring loss consistently contributes to achieving state-of-the-art visual sharpness as well as LPIPS and NIQE across different model architectures.

heading: 2 RELATED WORKS
text: Image Deblurring. Classical energy optimization framework is formulated by likelihood and prior terms. Due to the ill-posedness of dynamic scene deblurring problem, prior terms have been essential in alleviating the optimization ambiguity, encoding the preference on the solutions. Sophisticated prior terms were carefully designed with human knowledge on natural image statistics (Levin, 2006; Cho & Lee, 2009; Hirsch et al., 2011; Whyte et al., 2012; Sun et al., 2013; Xu et al., 2013; Kim et al., 2013; Kim & Lee, 2014; Pan et al., 2016). Recently in Li et al. (2018), learned prior from a classifier discriminating blurry and clean images was also shown to be effective. Deep priors were also used for image deconvolution problems (Ren et al., 2020; Nan & Ji, 2020).
On the other hand, deep learning methods have benefited from learning on large-scale datasets. The datasets consisting of realistic blur (Nah et al., 2017; Su et al., 2017; Noroozi et al., 2017; Nah et al., 2019; Gao et al., 2019; Jin et al., 2019; Shen et al., 2019) align the temporal center of the blurry and the sharp image pairs with high-speed cameras. Learning from such temporally aligned data relieve the ill-posedness of deblurring compared with difficult energy optimization framework. Thus, more attention has been paid to designing CNN architectures and datasets than designing loss terms.
In the early work of Schuler et al. (2015), the alternating estimation of blur kernel and restored image (Cho & Lee, 2009) was adopted in CNN architecture. In Sun et al. (2015); Gong et al. (2017), the spatially varying blur kernels are estimated by assuming locally linear blur followed by non-blind deconvolution. Later, end-to-end learning without explicit kernel estimation became popular. Motivated from the coarse-to-fine approach, multi-scale CNN was proposed (Nah et al., 2017) to expand the receptive field efficiently, followed by scale-recurrent architectures (Tao et al., 2018; Gao et al., 2019). On the other hand, Zhang et al. (2019); Suin et al. (2020) sequentially stacked network modules. Recently, Park et al. (2020) proposed a multi-temporal model that deblurs an image recursively. To handle spatially varying blur kernels efficiently, spatially non-uniform operations were embedded in neural networks (Zhang et al., 2018a; Yuan et al., 2020).
Perceptual Image Restoration. Often, L1 or L2 losses are used at training to achieve higher PSNR. However, such approaches suffer from blurry and over-smoothed outputs (Johnson et al., 2016; Zhang et al., 2018b; Menon et al., 2020) as the learned models predict an average of all possible solutions under the ill-posedness (Ledig et al., 2017). To deal with the issue, several studies utilize deep features of the pretrained VGG (Simonyan & Zisserman, 2014) and other networks that are more related to human perception (Johnson et al., 2016; Zhang et al., 2018b) and with analysis on frequency space (Tariq et al., 2020; Czolbe et al., 2020). Recent methods introduce adversarial training (Goodfellow et al., 2014) so that outputs of the restoration models be indistinguishable from real samples (Nah et al., 2017; Nimisha et al., 2017; Ledig et al., 2017; Kupyn et al., 2018; 2019). Also, there were attempts to exploit statistical properties of images and features with contextual loss (Mechrez et al., 2018) and projected distribution loss (Delbracio et al., 2021).
Nevertheless, an inherent limitation of existing perceptual objectives is that they are not taskspecialized for image restoration. For example, the VGG features are learned for high-level visual recognition while the adversarial loss only contributes to reconstructing realistic images without considering the existence of motion blur. Therefore, blindly optimizing those terms may not yield an optimal solution in terms of image deblurring. In practice, we observed that those objectives still tend to leave blur footprints unremoved, making it possible to estimate the original blur. Our reblurring loss is explicitly designed to improve the perceptual sharpness of deblurred images by reducing remaining blurriness and thus more suitable for deblurring, acting as a learned prior.
Image Blurring. As an image could be blurred in various directions and strength, image blurring is another ill-posed problem without additional information. Thus, intrinsic or extrinsic information is often incorporated. With a non-ideally sharp image, Bae & Durand (2007) detected the small local blur kernel in the image to magnify the defocus blur for bokeh effect. On the other hand, Chen et al. (2018) estimated the kernel by computing the optical flow from the neighboring video frames. Similarly, Brooks & Barron (2019) used multiple video frames to synthesize blur. Without such external information, Zhang et al. (2020) used a generative model to synthesize many blurry images. In contrast, Bahat et al. (2017) deliberately blurred an already blurry image in many ways to find the local blur kernel. Our image reblurring concept is similar to Bae & Durand (2007) in the sense that intrinsic cue in an image is used to amplify blur. Nonetheless, our main goal is to use reblurring to provide a guide to deblurring model so that such blur cues would be better removed.

heading: 3 PROPOSED METHOD
text: In this section, we describe a detailed concept of image reblurring and how the operation can be learned. Then, we demonstrate that the operation can support the deblurring module to reconstruct perceptually favorable and sharp images. We also propose a self-supervised test-time optimization strategy by using the learned reblurring module. For simplicity, we refer to the blurry, the deblurred, and the sharp image as B, L, and S, respectively.

heading: 3.1 CLEAN IMAGES ARE HARD TO REBLUR
text: As shown in Figure 1, outputs from the existing deblurring methods still contain undesired motion trajectories that are not completely removed from the input. Ideally, a well-deblurred image should not contain any motion cues, making reblurring ill-posed. We first validate our motivation by buliding a reblurring moduleMR which amplifies the remaining blur from L. MR is trained with the following blur reconstruction loss LBlur so that it would learn the inverse operation of deblurring.
LBlur = ‖MR(L)−B‖. (1) We applyMR to the deblurred images from deblurring modules of varying capacities. Table 1 shows that the higher the deblurring PSNR, the lower the reblurring PSNR becomes when both modules are trained with conventional L1 loss, independently from each other. It demonstrates that the better deblurred images are harder to reblur, consistent to our motivation.
In contrast to the non-ideally deblurred images, MR is not able to generate a motion blur from a sharp image S as no motion information is found. For a high-quality clean image, MR should preserve the sharpness. However, optimizing the blur reconstruction loss LBlur with S may fall into learning the pixel average of all blur trajectories in the training dataset, i.e. Gaussian blur. In such a case,MR will apply the single uniform blur on every image without considering the scene information. To let the blur domain of MR confined to the motion-incurred blur, we use sharp images to penalize such undesired operations. Specifically, we introduce a network-generated sharp image Ŝ obtained by feeding a real sharp image S to the deblurring moduleMD as Ŝ =MD(S). We define sharpness preservation loss LSharp as follows:
LSharp = ‖MR(Ŝ)− Ŝ‖. (2)
We use the pseudo-sharp image Ŝ instead of a real image S to make our reblurring module focus on image sharpness and blurriness rather than the realism. While Ŝ differ from L only by the sharpness, S also differ by the realism which can be easily detected by neural networks (Wang et al., 2020).
Combining both terms together, we train the reblurring moduleMR by optimizing the joint loss LR: LR = LBlur + LSharp. (3)
As zero-magnitude blur should remain unaltered fromMR, the sharpness preservation loss can be considered a special case of the blur reconstruction loss. Figure 2a illustrates the way our reblurring module is trained from LR.

heading: 3.2 SUPERVISION FROM REBLURRING LOSS
text: The blurriness of images can be more easily witnessed by amplifying the blur. Thus, we propose a new optimization objective by processing the deblurred and the sharp image with the jointly trained reblurring modelMR. To suppress the remaining blur in the output image L =MD(B) from the deblurring moduleMD, the supervised reblurring loss LReblur for image deblurring is defined as
LReblur = ‖MR(L)−MR(S)‖. (4)
Unlike the sharpness preservation term in equation 2, we do not use the pseudo-sharp image Ŝ in our reblurring loss, LReblur. As the quality of the pseudo-sharp image Ŝ depends on the state of deblurring moduleMD, using Ŝ may make training unstable and difficult to optimize, especially at the early stage. Thus we use a real sharp image S to stabilize the training. Nevertheless, asMR is trained to focus on the sharpness from equation 3, so does the reblurring loss, LReblur. Using our reblurring loss in equation 4, the deblurring moduleMD is trained to minimize the following objective LD: LD = L1 + λLReblur, (5) where L1 is a conventional L1 loss, and the hyperparameter λ is empirically set to 1. Figure 2b illustrates how the deblurring model is trained with the guide fromMR. At each training iterations, we alternately optimize two modulesMD andMR byLD andLR, respectively. While such a strategy may look similar to the adversarial training scheme, the optimization objectives are different. As the neural networks are well known to easily discriminate real and fake images (Wang et al., 2020), the realism could be as a more obvious feature than image blurriness. Thus, adversarial loss may overlook image blurriness as L and S can already be discriminated by the difference in realism. On the other hand, our reblurring loss is explicitly designed to prefer sharp images regardless of realism as we use Ŝ instead of S in LSharp to trainMR. Figure 3 conceptually compares the actual role of the reblurring loss LReblur and the adversarial loss LAdv.

heading: 3.3 TEST-TIME ADAPTATION BY SELF-SUPERVISION
text: After the training is over, the models learned from supervised loss terms have fixed weights at test time. When a new example that deviates from the distribution of training data is given, the supervised methods may lack ability to generalize. Our reblurring module, however, can further provide self-supervised guide so that the model could be further optimized for each image at test time. While the supervised reblurring loss LReblur finds the blurriness of L by comparison with the ground truth,MR can also inspect the blurriness of an image without reference. AsMR is trained to magnify the blur in L, imperfectly deblurred image would be blurred. Thus, the difference betweenMR(L) and L can serve as a feedback without having to reference S. Furthermore, due to the sharpness preservation loss LSharp, a sufficiently sharp image would have little difference when reblurred. Based on the property, we construct the self-supervised reblurring loss
On GOPRO dataset On REDS dataset Model Optimization LPIPS↓ NIQE↓ PSNR↑ SSIM↑ LPIPS↓ NIQE↓ PSNR↑ SSIM↑
that serves as a prior term embedding the preference on sharp images as
LselfReblur = ‖MR(L)− L∗‖, (6) where L∗ denotes the image with the same value as L but the gradient does not backpropagate in the optimization process. We minimize LselfReblur for each test data to obtain the sharper image. Allowing gradient to flow through L∗ can let L to fall into undesired local minima where both the L and MR(L) are blurry. We iteratively optimize the weights of MD with fixed MR. As LselfReblur only considers the sharpness of an image, we keep the color consistency by matching the color histogram between the test-time adapted image and the initially deblurred image. For the detailed optimization process of test-time adaptation strategy, please refer to the Appendix Algorithm 1. The effect of test-time adaptation is conceptually visualized in Figure 5. More iterations make the deblurred image sharper. We note that our loss functions and the test-time adaptation are applicable to general learning-based approaches.

heading: 4 EXPERIMENTS
text: We verify the effectiveness of our reblurring loss and the generalization by applying it to multiple deblurring model architectures. We show the experimental results with a baseline residual U-Net and state-of-the-art image deblurring models, the sRGB version SRN (Tao et al., 2018) and DHN, our modified version of DMPHN (Zhang et al., 2019). For the reblurring module, we use simple residual networks with 1 or 2 ResBlock(s) with 5× 5 convolution kernels. The training and evaluation were done with the widely used GOPRO (Nah et al., 2017) and REDS (Nah et al., 2019) datasets. The GOPRO dataset consists of 2103 training and 1111 test images with various dynamic motion blur. Similarly, the REDS dataset has 24000 training and 3000 validation data publicly available. For each dataset, every experiment was done in the same training environment. We mainly compare LPIPS (Zhang et al., 2018b) and NIQE (Mittal et al., 2012) perceptual metrics as our goal is to make images sharper. For more implementation details, please refer to the Appendix.

heading: 4.1 EFFECT OF SUPERVISED REBLURRING LOSS
text: We implement the reblurring loss in varying degrees of emphasis on sharpness by controlling the reblurring module capacity. For a more balanced quality between PSNR and perceptual sharpness, we use 1 ResBlock for MR. To put more weight on the perceptual quality, we allocate a larger capacity onMR by using 2 ResBlocks. For notation simplicity, we denote the reblurring loss with k ResBlock(s) in the reblurring module as LReblur, nk. Table 2 shows how the deblurring performance varies by the use of reblurring loss functions. With LReblur, n1, LPIPS and NIQE improves to a moderate degree while PSNR and SSIM metrics remain
at a similar level. Meanwhile, LReblur, n2 more aggressively optimizes the perceptual metrics. This is analogous to the perception-distortion trade-off witnessed in the image restoration literature (Blau & Michaeli, 2018; Blau et al., 2018). The perceptual metric improvements are consistently witnessed with various architectures on both the GOPRO and the REDS dataset.

heading: 4.2 EFFECT OF SHARPNESS PRESERVATION LOSS
text: In trainingMR, we used both the blur reconstruction loss LBlur and the sharpness preservation loss LSharp. The latter term LSharp plays an essential role in lettingMR concentrate only on the motiondriven blur in the given image and keep sharp image remain sharp. Table 3 presents the performance gains from using LSharp jointly with LBlur in terms of the perceptual quality.
Also, the use of the pseudo-sharp image Ŝ is justified by comparing it with the use case of real sharp image S. We found the using S for LSharp with LBlur makes the training less stable than using Ŝ. Using the pseudo-sharp image confines the input data distribution ofMR to the domain ofMD outputs. For neural networks, it is very easy to discriminate real and fake images (Wang et al., 2020). By using a fake image Ŝ instead of a real image S, we letMR focus on the sharpness of an image and avoid being distracted by a more obvious difference between real and fake images. Furthermore, it leads the two loss terms LBlur and LSharp to reside under the same objective, amplifying any noticeable blur and keeping sharpness when motion blur is in zero-magnitude.

heading: 4.3 COMPARISON WITH OTHER PERCEPTUAL LOSSES
text: The reblurring loss provides a conceptually different learning objectives from the adversarial and the perceptual losses and is designed to focus on the motion blur. Table 4 compares the effect of LReblur with adversarial loss LAdv, and the VGG perceptual loss (Johnson et al., 2016) by applying them to SRN (Tao et al., 2018) on GOPRO dataset. While our method provides quantitatively better perceptual scores, the different perceptual losses are oriented to different goals. They do not necessarily compete or conflict with each other and can be jointly applied at training to catch the perceptual quality in varying aspects. In Figure 4, the effect of the reblurring loss is visually compared with the previous perceptual loss functions.

heading: 4.4 EFFECT OF TEST-TIME ADAPTATION
text: We conduct test-time adaptation with the proposed self-supervised reblurring loss, LselfReblur to make the deblurred image even sharper. Figure 6 shows the test-time-adapted result with SRN. Compared with the baseline trained with L1 loss, our results exhibit improved trade-off relations between PSNR and the perceptual metrics, LPIPS and NIQE. Table 2 provides detailed quantitative testtime adaptation results on GOPRO and REDS dataset, respectively with various deblurring module architectures. The effect of test-time adaptation is visually shown in Figure 8.

heading: 4.5 COMPARISON WITH STATE-OF-THE-ART METHODS
text: We have improved the perceptual quality of the deblurred images by training several different model architectures. We compare the perceptual quality with the other state-of-the-art methods in Figure 7. Especially, DeblurGAN-v2 was trained with the VGG loss and the adversarial loss. Our results achieve visually sharper texture from the reblurring loss and test-time adaptation.

heading: 4.6 REAL WORLD IMAGE DEBLURRING
text: While our method uses synthetic datasets (Nah et al., 2017; 2019) for training, the trained models generalize to real blurry images. In Figure 9, we show deblurred results from Lai et al. (2016) dataset with DHN model. Compared with the baseline L1 loss, our reblurring loss LReblur, n2 provides an improved deblurring quality. As the real test image could deviate from the training data distribution, a single forward inference may not produce optimal results. With the self-supervised test-time adaptation, our deblurred images reveal sharper and detailed textures.

heading: 5 CONCLUSION
text: In this paper, we validate a new observation that clean sharp images are hard to reblur and develop novel low-level perceptual objective terms, namely reblurring loss. The term is constructed to care the image blurriness by jointly training a pair of deblurring and reblurring modules. The supervised reblurring loss provides an amplified comparison on motion blur while the self-supervised loss inspects the blurriness in a single image with the learned reblurring module. The self-supervision lets the deblurring module adapt to the new image at test time without ground truth. By applying the loss terms to state-of-the-art deblurring methods, we demonstrate our method consistently improves the the perceptual sharpness of the deblurred images visually as well as quantitatively.

heading: A APPENDIX
text: In this appendix, we provide the implementation details and additional experimental analysis. In Section B, we explain the implementation details with the model architecture specifics, training details, and the evaluation metrics. Section C describes how the reblurring module design and the size are determined. Then in Section D, we describe the different characteristics of the proposed reblurring loss and the other perceptual losses. We combine our reblurring loss with the other perceptual losses to take advantage in multiple perspectives. In Section B, the test-time adaptation algorithm is described. In Section F, we show the quantitative effect of test-time adaptation and show the tradeoff relation between the conventional distortion quality metric (PSNR, SSIM) and the perceptual metrics (LPIPS, NIQE) compared with the baselines.
B IMPLEMENTATION DETAILS
Model Architecture. In the main manuscript, we mainly performed the experiments with 3 different model architectures. First, we set our baseline model as a light-weight residual U-Net architecture that runs in a fast speed. The baseline model is used to design our reblurring loss with pseudo-sharp images through ablation study in Table 3.
For reblurring operation, we use a simple residual networkMR without strides to avoid deconvolution artifacts. The baseline U-Net and the reblurring module architectures are shown in Figure 10. The detailed parameters for U-Net andMR are each specified in Table 5 and 7.
In addition to the U-Net, experiments were conducted with state-of-the-art deblurring models based on SRN (Tao et al., 2017) and DMPHN (Zhang et al., 2019). SRN (Tao et al., 2018) was originally designed to operate on grayscale images with a LSTM module. Later, the authors released the
sRGB version code without LSTM, exhibiting an improved accuracy. We adopted the revised SRN structure in our experiments.
The other model we chose is based on DMPHN (1-2-4-8) (Zhang et al., 2019). DMPHN performs hierarchical residual refinement to produce the final output. The model consists of convolutional layers with ReLU activations that are spatially shift-equivariant. In Zhang et al. (2019), each level splits the given image and performs the convolutional operation on the divided patches. As the convolutional weights do not differ by the input patches, the operations do not necessarily have to be done patch-wise. Thus, we remove the multi-patch strategy and perform the convolution on the whole given input without dividing the image into patches. We refer to the modified model as DHN. As shown in Table 6, convolution on the whole image compared with patch-wise convolution brings higher accuracy.
Metrics. To quantitatively compare the deblurred images in the following sections, we use PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018b), and NIQE (Mittal et al., 2012). In the image deblurring literature, SSIM has been measured by MATLAB ssim function on sRGB images with H × W × C. SSIM was originally developed for grayscale images and MATLAB ssim function for a 3-dimensional tensor considers an image to be a 3D grayscale volume image. Thus, most of the previous SSIM measures were not accurate, leading to higher values. Instead, we measured all the SSIM for each channel separately and averaged them. We used skimage.metrics.structural similarity function in the scikit-image package for python to measure SSIM for multi-channel images.
Training. For all the experiments, we performed the same training process for a fair comparison. On the GOPRO dataset (Nah et al., 2017), we trained each model for 4000 epochs. On the REDS dataset (Nah et al., 2019), the models are trained for 200 epochs. Adam (Kingma & Ba, 2014) optimizer is used in all cases. When calculating distance between images with Lp norm, we always set p = 1, using L1 distance. Starting from the initial learning rate 1×10−4, the learning rate halves when training reaches 50%, 75%, and 90% of the total epochs. We used PyTorch 1.8.1 with CUDA 11.1 to implement the deblurring methods. Mixed-precision training (Micikevicius et al., 2017) is employed to accelerate operations on RTX 2080 Ti GPUs.

heading: C DETERMINING THE REBLURRING MODULE SIZE
text: As our reblurring loss LR is realized byMR, the reblurring module design plays an essential role. As shown in Figure 10, theMR architecture is a simple ResNet. Table 8 shows the relation between the deblurring performance andMR size by changing the number of ResBlocks. For all deblurring moduleMD architectures, LPIPS was the best when the number of ResBlocks, n = 2. NIQE showed good performance when 2 ≤ n ≤ 3. PSNR and SSIM had tendency to
decrease when n ≥ 1. For larger number of ResBlocks, we witnessed sharper edges could be obtained but sometimes, cartoon artifacts with over-strong edges were witnessed.
Considering the trade-off between the PSNR and the perceptual metrics, we chose n ∈ {1, 2} in the following experiments. n = 1 finds balance between the PSNR and LPIPS and n = 2 puts more weight on the perceptual quality.

heading: D COMBINING REBLURRING LOSS WITH OTHER PERCEPTUAL LOSSES
text: Our reblurring loss is a new perceptual loss that is sensitive to blurriness of an image, a type of image structure-level information while other perceptual losses such as VGG loss (Johnson et al., 2016) and adversarial loss (Ledig et al., 2017) are more related to the high-level contexts. As VGG model (Simonyan & Zisserman, 2014) is trained to recognize image classes, optimizing with VGG loss could make an image better recognizable. In the GAN frameworks (Goodfellow et al., 2014), it is well known that discriminators can easily tell fake images from real images (Wang et al., 2020), being robust against JPEG compression and blurring. In the adversarial loss from the discriminator, the realism difference could be more salient than other features such as blurriness.
With the perceptual loss functions designed with different objectives, combining them could bring visual quality improvements in various aspects. Tables 9 and 10 show the effect of applying our reblurring loss jointly with other perceptual losses on GOPRO and REDS datasets. We omit the loss coefficients for simplicity. We used weight 0.3 for the VGG loss LVGG and 0.001 for the adversarial loss, LAdv. We witness LPIPS and NIQE further improves when our reblurring loss is combined with LVGG or LAdv.

heading: E TEST-TIME ADAPTATION DETAILS
text: We describe the detailed self-supervised test-time adaptation process. At test time, the learning rate is set to µ = 3 × 10−6. From the initial deblurring result L0, the self-supervised loss is iteratively minimized by updating the weights of MD. As the self-supervised loss in equation 6 only cares about image sharpness, the image may have color drifting issues. Thus, finally, we match the histogram of the updated image LN to the histogram of L0. The overall process is shown in Algorithm 1.
Algorithm 1 Optimization process in test-time adaptation 1: procedure TEST-TIME ADAPTATION(B,MD,MR) 2: Test-time learning rate µ← 3× 10−6. 3: θD ←Weights ofMD. 4: L0 =MD(B). 5: for i = 0 . . . N − 1 do 6: Li∗ =MD (B). 7: Lselfreblur = ‖MR(MD(B))− Li∗‖. 8: Update θD by∇θDLselfReblur and µ.
LN =MD (B). 9: LNAdapted =histogram matching(L N , L0∗)
10: return LNAdapted

heading: F PERCEPTION VS. DISTORTION TRADE-OFF
text: It is known in image restoration literature that the distortion error and the perceptual quality error are in trade-off relation (Blau & Michaeli, 2018; Blau et al., 2018). The relation is often witnessed by training a single model with different loss functions. In most cases, to obtain a better perceptual quality from a single model architecture, retraining with another loss from scratch is necessary. Our
test-time adaptation from self-supervised reblurring loss, in contrast, can provide the steps toward perceptual quality without full retraining.
In Figure 11 and 12, we present the perception-distortion trade-off from our test-time adaption. LPIPS and NIQE scores consistently improve from each adaptation step in both SRN and DHN models. While PSNR is moderately sacrificed from the adaptation, SSIM improves in the early steps as it more reflects the structural information. Our results show improved trade-off between the distortion and perception metrics over the baseline models trained with L1 loss.

heading: G TEST-TIME ADAPTATION EFFECTS
text: In Table 11, we quantitatively compare the deblurred results from test-time adaptation in terms of a no-reference metric, NIQE, and reference-based metrics, LPIPS, PSNR, and SSIMs. By performing TTA up to 30 steps as described in Algorithm 1, we show that LPIPS and NIQE could be improved to a degree. On both GOPRO and REDS datasets, NIQE has a tendency to improve further after LPIPS has stopped its improvement. This is due to the self-supervised nature of test-time adaptation that considers image sharpness without reference.
In Figures 13 and 14, we visually show the effect of test-time adaptation applied to SRN with a jointly trained reblurring module. By test-time adaptation, our model further improves the sharp edges of the images. In Figure 13, the building structures and the horizontal lines are better witnessed. Also in Figure 14, the vehicle’s pole are better recovered and the text are clearer. While the PSNR and SSIM have decreased by test-time adaptation in Table 11, perceptually, the results from test-time adaptation tend to be sharper.

Review summaries:
The paper proposes a novel approach to using deep networks for image deblurring. Since training with simply reconstruction losses usually leads to oversmoothed results, recent works have looked at using perceptual and adversarial losses.  The paper proposes an approach similar to an adversarial loss. If an image is not de-blurred entirely, it leaves within itself some tell-tale signs of the original blur. The paper proposes learning a “reblurring” network that given a “deblurred” image should yield the original “blurry” image, while given a true sharp image should output the image itself. The deblurring network is then trained in a sense to fool the re-blurring network, by generating images that would be left untouched by the reblurring network. The paper also proposes a “test time adaptation” approach.

A novel deblurring method is proposed by introducing the concept of reblurring loss. Conventional techniques deblur blurry images to some extent however there are some unremoved blur contents, where it can be used to reconstruct (reblur) blurry images. On the other hand, cleanly deblurred images have only sharp contents and it is difficult to reblur the images since no blurry cues are left. By using such observation, a reblurring module and a deblurring module with reblurring loss are proposed. Experimental results show that the proposed framework outperforms SOTA quantitatively and qualitatively on the GOPRO and REDS dataset. 

This work addresses the problem of image deblurring by training a deep model in an end-to-end fashion. Similar to recent work, the method makes use of large paired (blurry,sharp) training datasets to train the deblurring network.  The main idea of the work is that a correctly deblurred image should not contain any information about the original blur. The method exploits this observation by using an auxiliary network that tries to re-blur the deblurred image so it has the same blur as the original one. If the deblurred image was perfectly deblurred this shouldn't be possible.  These two networks end up having a "similar" role as a generator-discriminator on a GAN set-up. But even if there's this superficial connection with GANs the formalism is completely different. Here the reblurring network outputs an image that is compared to another image (i.e., reference-based).  The work introduces three different loss terms: L_blur, L_sharp and L_reblur as long as the typical regression loss L1 between deblurred and sharp images.  - L_blur: is mainly used to train the re-blurring module; - L_sharp: is used to avoid the re-blurring module to just apply the average blurring; - L_reblur: is used as a regression loss on the space of re-blurred images.  Additionally, the paper proposes a test time adaptation where the reblurring network is used to boost the deblurring results on a given image. This is done by approximately inverting the re-blurring network using gradient descent.   The work presents several experiments including ablation studies with two of the popular (synthetic) datasets used in image/video deblurring (GoPro and REDS). Several comparisons with SOTA methods and some results on a real image dataset (Lai et al. 2016). Different quantitative metrics are given (PSNR,SSIM, LPIPS, NIQE) as well as several figures showing visual comparisons.