Clean Images are Hard to Reblur: Exploiting the Ill-Posed Inverse Task for Dynamic Scene Deblurring

1 INTRODUCTION . Motion blur is a common photographic artifact that occurs when the camera moves and the scene changes during the exposure in dynamic environments . Dynamic scene deblurring is a challenging ill-posed problem as both the locally-varying blur kernel and the latent sharp image have to be found from a large solution space . Traditional approaches ( Hirsch et al. , 2011 ; Whyte et al. , 2012 ; Kim et al. , 2013 ; Kim & Lee , 2014 ) tried to alleviate the ill-posedness by using prior knowledge on the statistical properties of sharp images , such as gradient sparsity . Instead of using such handcrafted knowledge , recent methods take advantage of large-scale datasets as well as deep neural networks ( Nah et al. , 2017 ; Su et al. , 2017 ; Noroozi et al. , 2017 ; Nah et al. , 2019 ; Shen et al. , 2019 ) . Usually , the learning is driven by minimizing the pixel-wise distance to the ground truth , e.g. , L1 or L2 , so that the PSNR between the deblurred and the sharp reference can be maximized . By utilizing modern ConvNet architectures and training techniques , state-of-the-art approaches ( Nah et al. , 2017 ; Tao et al. , 2017 ; Gao et al. , 2019 ; Yuan et al. , 2020 ; Park et al. , 2020 ; Chi et al. , 2021 ) have been developed toward higher capacity and deblurring accuracy . Still , most methods tend to suffer from the blurry predictions due to the regression-to-mean behavior often witnessed in ill-posed problems with large solution space ( Ledig et al. , 2017 ; Menon et al. , 2020 ) . To overcome limitations of the conventional objectives , concepts of perceptual ( Johnson et al. , 2016 ) and adversarial ( Ledig et al. , 2017 ; Nah et al. , 2017 ; Kupyn et al. , 2018 ) loss terms from highlevel semantic tasks have been introduced to improve the visual quality of the deblurred results . Nevertheless , such high-level losses may not serve as optimal goals for blur removal as low-level structural properties , e.g. , blurriness , are not the primary features considered in their formulations . As illustrated in Figure 1 , results from the previous deblurring methods are still blurry to a degree and the VGG and the adversarial losses are not sufficient to obtain perceptually pleasing and sharp images across different architectures ( Tao et al. , 2018 ; Gao et al. , 2019 ; Kupyn et al. , 2019 ) . While the deblurred images look less blurry compared with the original input , it is still possible to find nontrivial blur kernels with directional motion information . From the observation , we introduce the concept of reblurring which amplifies the unremoved blur in the given image and reconstructs the original blur . We note that our reblurring operation aims to recover the original motion trajectory in the blurry input , rather than to synthesize arbitrary , e.g. , Gaussian , blurs . Therefore , an ideally deblurred clean image is hard to reblur as no noticeable blur can be found to be amplified , making reblurring an ill-posed task . In contrast , it is straightforward to predict the original shape of blur from insufficiently deblurred images as shown in Figure 1 . We propose to use the difference between non-ideally deblurred image and the ideal sharp image in terms of reblurring feasibility as the new optimization objective , reblurring loss for the image deblurring problem . The reblurring loss is realized by jointly training a pair of deblurring and reblurring modules . The reblurring module performs the inverse operation of deblurring , trying to reconstruct the original blurry image from a deblurred output . Using the property that the blurriness of a reblurred image depends on the sharpness quality of the deblurred result , we construct two types of loss functions . During the joint training , supervised reblurring loss compares the amplified blurs between the deblurred and the sharp image . Complementing L1 intensity loss , the supervised reblurring loss guides the deblurring module to focus on and eliminate the remaining blur . While our training strategy is similar to the adversarial training of GANs ( Goodfellow et al. , 2014 ) in a sense that our deblurring and reblurring modules play the opposite roles , the purposes and effects of the adversary are different . The reblurring loss concentrates on image blurriness regardless of image realism . Furthermore , in contrast to the GAN discriminators that are not often used at test time , our reblurring module can be used to facilitate self-supervised reblurring loss . By making the deblurred image harder to reblur , the deblurring module can adaptively optimize itself without referring to the ground truth . Our reblurring loss functions provide additional optimization directives to the deblurring module and can be generally applied to any learning-based image deblurring methods . With the proposed approach , we can derive sharper predictions from existing deblurring methods without modifying their architectures . We summarize our contributions as follows : • Based on the observation that clean images are hard to reblur , we propose novel loss functions for image deblurring . Our reblurring loss reflects the preference for sharper images and contributes to visually pleasing deblurring results . • At test-time , the reblurring loss can be implemented without a ground-truth image . We perform test-time adaptive inference via self-supervised optimization with each input . • Our method is generally applicable to any learning-based methods and jointly with other loss terms . Experiments show that the concept of reblurring loss consistently contributes to achieving state-of-the-art visual sharpness as well as LPIPS and NIQE across different model architectures . 2 RELATED WORKS . Image Deblurring . Classical energy optimization framework is formulated by likelihood and prior terms . Due to the ill-posedness of dynamic scene deblurring problem , prior terms have been essential in alleviating the optimization ambiguity , encoding the preference on the solutions . Sophisticated prior terms were carefully designed with human knowledge on natural image statistics ( Levin , 2006 ; Cho & Lee , 2009 ; Hirsch et al. , 2011 ; Whyte et al. , 2012 ; Sun et al. , 2013 ; Xu et al. , 2013 ; Kim et al. , 2013 ; Kim & Lee , 2014 ; Pan et al. , 2016 ) . Recently in Li et al . ( 2018 ) , learned prior from a classifier discriminating blurry and clean images was also shown to be effective . Deep priors were also used for image deconvolution problems ( Ren et al. , 2020 ; Nan & Ji , 2020 ) . On the other hand , deep learning methods have benefited from learning on large-scale datasets . The datasets consisting of realistic blur ( Nah et al. , 2017 ; Su et al. , 2017 ; Noroozi et al. , 2017 ; Nah et al. , 2019 ; Gao et al. , 2019 ; Jin et al. , 2019 ; Shen et al. , 2019 ) align the temporal center of the blurry and the sharp image pairs with high-speed cameras . Learning from such temporally aligned data relieve the ill-posedness of deblurring compared with difficult energy optimization framework . Thus , more attention has been paid to designing CNN architectures and datasets than designing loss terms . In the early work of Schuler et al . ( 2015 ) , the alternating estimation of blur kernel and restored image ( Cho & Lee , 2009 ) was adopted in CNN architecture . In Sun et al . ( 2015 ) ; Gong et al . ( 2017 ) , the spatially varying blur kernels are estimated by assuming locally linear blur followed by non-blind deconvolution . Later , end-to-end learning without explicit kernel estimation became popular . Motivated from the coarse-to-fine approach , multi-scale CNN was proposed ( Nah et al. , 2017 ) to expand the receptive field efficiently , followed by scale-recurrent architectures ( Tao et al. , 2018 ; Gao et al. , 2019 ) . On the other hand , Zhang et al . ( 2019 ) ; Suin et al . ( 2020 ) sequentially stacked network modules . Recently , Park et al . ( 2020 ) proposed a multi-temporal model that deblurs an image recursively . To handle spatially varying blur kernels efficiently , spatially non-uniform operations were embedded in neural networks ( Zhang et al. , 2018a ; Yuan et al. , 2020 ) . Perceptual Image Restoration . Often , L1 or L2 losses are used at training to achieve higher PSNR . However , such approaches suffer from blurry and over-smoothed outputs ( Johnson et al. , 2016 ; Zhang et al. , 2018b ; Menon et al. , 2020 ) as the learned models predict an average of all possible solutions under the ill-posedness ( Ledig et al. , 2017 ) . To deal with the issue , several studies utilize deep features of the pretrained VGG ( Simonyan & Zisserman , 2014 ) and other networks that are more related to human perception ( Johnson et al. , 2016 ; Zhang et al. , 2018b ) and with analysis on frequency space ( Tariq et al. , 2020 ; Czolbe et al. , 2020 ) . Recent methods introduce adversarial training ( Goodfellow et al. , 2014 ) so that outputs of the restoration models be indistinguishable from real samples ( Nah et al. , 2017 ; Nimisha et al. , 2017 ; Ledig et al. , 2017 ; Kupyn et al. , 2018 ; 2019 ) . Also , there were attempts to exploit statistical properties of images and features with contextual loss ( Mechrez et al. , 2018 ) and projected distribution loss ( Delbracio et al. , 2021 ) . Nevertheless , an inherent limitation of existing perceptual objectives is that they are not taskspecialized for image restoration . For example , the VGG features are learned for high-level visual recognition while the adversarial loss only contributes to reconstructing realistic images without considering the existence of motion blur . Therefore , blindly optimizing those terms may not yield an optimal solution in terms of image deblurring . In practice , we observed that those objectives still tend to leave blur footprints unremoved , making it possible to estimate the original blur . Our reblurring loss is explicitly designed to improve the perceptual sharpness of deblurred images by reducing remaining blurriness and thus more suitable for deblurring , acting as a learned prior . Image Blurring . As an image could be blurred in various directions and strength , image blurring is another ill-posed problem without additional information . Thus , intrinsic or extrinsic information is often incorporated . With a non-ideally sharp image , Bae & Durand ( 2007 ) detected the small local blur kernel in the image to magnify the defocus blur for bokeh effect . On the other hand , Chen et al . ( 2018 ) estimated the kernel by computing the optical flow from the neighboring video frames . Similarly , Brooks & Barron ( 2019 ) used multiple video frames to synthesize blur . Without such external information , Zhang et al . ( 2020 ) used a generative model to synthesize many blurry images . In contrast , Bahat et al . ( 2017 ) deliberately blurred an already blurry image in many ways to find the local blur kernel . Our image reblurring concept is similar to Bae & Durand ( 2007 ) in the sense that intrinsic cue in an image is used to amplify blur . Nonetheless , our main goal is to use reblurring to provide a guide to deblurring model so that such blur cues would be better removed .

Summary: This work addresses the problem of image deblurring by training a deep model in an end-to-end fashion. Similar to recent work, the method makes use of large paired (blurry,sharp) training datasets to train the deblurring network.  The main idea of the work is that a correctly deblurred image should not contain any information about the original blur. The method exploits this observation by using an auxiliary network that tries to re-blur the deblurred image so it has the same blur as the original one. If the deblurred image was perfectly deblurred this shouldn't be possible.  These two networks end up having a "similar" role as a generator-discriminator on a GAN set-up. But even if there's this superficial connection with GANs the formalism is completely different. Here the reblurring network outputs an image that is compared to another image (i.e., reference-based).  The work introduces three different loss terms: L_blur, L_sharp and L_reblur as long as the typical regression loss L1 between deblurred and sharp images.  - L_blur: is mainly used to train the re-blurring module; - L_sharp: is used to avoid the re-blurring module to just apply the average blurring; - L_reblur: is used as a regression loss on the space of re-blurred images.  Additionally, the paper proposes a test time adaptation where the reblurring network is used to boost the deblurring results on a given image. This is done by approximately inverting the re-blurring network using gradient descent.   The work presents several experiments including ablation studies with two of the popular (synthetic) datasets used in image/video deblurring (GoPro and REDS). Several comparisons with SOTA methods and some results on a real image dataset (Lai et al. 2016). Different quantitative metrics are given (PSNR,SSIM, LPIPS, NIQE) as well as several figures showing visual comparisons.