Cross-Scale Self-Supervised Blind Image Deblurring via Implicit Neural Representation

Authors: tianjing  zhang
yuhui  quan
hui  ji
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 11:29:35

Abstract: Blind image deblurring (BID) is an important yet challenging image recovery problem. Most existing deep learning methods require supervised training with ground truth (GT) images. This paper introduces a self-supervised method for BID that does not require GT images. The key challenge is to regularize the training to prevent over-fitting due to the absence of GT images. By leveraging an exact relationship among the blurred image, latent image, and blur kernel across consecutive scales, we propose an effective cross-scale consistency loss. This is implemented by representing the image and kernel with implicit neural representations (INRs), whose resolution-free property enables consistent yet efficient computation for network training across multiple scales. Combined with a progressively coarse-to-fine training scheme, the proposed method significantly outperforms existing self-supervised methods  in extensive experiments.

Main paper: 
 introduction: 
 Uniform blurring, a degradation commonly encountered in optics, leads to the loss of important details within a captured image. Uniform blurring usually can be described as the convolution:EQUATIONwhere ⊗ denotes the 2D discrete convolution operation, (y, x) is the pair of blurred and sharp (latent) images. k represents the blur kernel responsible for degradation, and n denotes measurement noise. For example, in digital photography, motion blur is caused by camera shake during exposure. When scene depth variation is small and camera movement is mainly translational within the image plane, motion blur can be approximated by the convolution model (1) , with k representing camera motion. In fluorescence microscopy, specimens stained with fluorescent dyes and exposed to specific wavelength light often exhibit blurring due to light diffraction and optical path imperfections, such as optical component misalignment or lens aberrations. These factors obscure critical details, like micro-tubule arrangements in cells. Such blurring can also be modeled by (1) . In either case, both the latent image x and blur kernel k are unknown and must be estimated from the blurred image y.BID aims at estimating (k, x), the pair of latent image and blur kernel, from the degraded image y.BID is a challenging non-linear inverse problem with many plausible solutions due to its inherent solution ambiguity. This ambiguity stems from the fact that the kernel can be decomposed into k = k 1 ⊗k 2 , suggesting that the pair (k 1 , k 2 ⊗x) are also viable solutions since y = (k 1 ⊗k 2 )⊗x = k 1 ⊗ (k 2 ⊗ x). One such example is the pair (δ, y = k, x), where δ denotes the delta kernel, which gives a trivial solution saying a blurred image is the convolution of a blurred image and a delta kernel. This ambiguity indicates the ill-posed-ness of the BID problem.The existing deep learning methods for BID can be roughly classified into the following categories:• Supervised deep learning for BID with paired data: Supervised learning methods (e.g., [50, 78, 68, 34, 69] ) trains a neural network (NN) using paired data of blurred images (or blur kernels) and GT images, to predict the latent image or/and blur kernel from an input blurred image. These methods mainly depend on a large amount of paired data for training, limiting their application to specific scenarios where such paired data is challenging to collect. • Supervised deep learning for BID with un-paired data: Some works (e.g. [36, 61] ) use generative adversarial networks (GANs) to train on unpaired blurred and latent images.Although effective for domain-specific images like faces and text, GANs perform poorly on general natural images due to domain shift. Additionally, GAN-based methods often suffer from mode collapse and training instability, resulting in suboptimal outcomes. • BID with pre-trained generative model: These methods employ pre-trained generative models, such as diffusion models [13] , which are well-suited for processing images with structural consistency features like human faces. However, their reliance on pre-trained models to generate outputs that conform to the learned image distribution can lead to inauthentic results. This is particularly problematic in fields that require high fidelity, such as medical imaging and microscopy. • Self-supervised learning for BID: To circumvent the challenges of data collection and mitigate potential biases and inauthentic outcomes from generative models, a growing body of research (e.g., [1, 47, 32, 33, 9, 18, 77] ) focuses on developing self-supervised deep learning approaches for BID that do not require training dataset or pre-trained model.In this paper, we focus on self-supervised BID, which is challenging due to the lack of GT images but offers many practical benefits. Most existing works on self-supervised BID are based on the deep image prior (DIP) [57] of convolutional NNs (CNNs), which introduces the implicit prior from CNN for preferring structured patterns over random noise during training. The DIP-based self-supervised BID methods typically use two NN-based generators, G k and G x , to re-parameterize the blur kernel k and the latent sharp image x. The generators are then trained to maximize the likelihood of the blurred image y, minimizing the following self-supervised reconstruction loss:EQUATIONFor instance, SelfDeblur [47] employed a CNN for G x and a Multi-Layer Perceptron (MLP) for G k . MCEM [33] used U-Net for both G x and G k , and Zhuang et al. [77] used INR for both G x and G k .Despite lacking access to GT images, these self-supervised BID methods perform competitively against supervised or pre-trained model based approaches, particularly with severely blurred images. However, their performance is less impressive on modestly blurred images and on real-world images.There is practical need for further studies to improve the performance of self-supervised BID across various blurring degrees and real-world images from different optics systems.Without GT images, self-supervised BID methods must address two vital questions for NN training:1. Without accessing GT images, how to formulate a self-supervised loss to teach the NN-based generators to accurately predict the latent image and kernel from only the blurred image? 2. The non-linear structure of BID makes training NN-based generators challenging. How can we efficiently train them to ensure accurate convergence to the latent images and kernels?Our answer to Question 1 is a cross-scale loss function that leverages the cross-scale consistency of the estimates from consecutive scales for regularization. Our answer to Question 2 is a progressive cross-scale training scheme to training the NNs, enhancing training efficiency and ensuring the convergence to GT image/kernel.Resolution-free INR for effective cross-scale interaction in BID: Our proposed method is built on re-parametrizatrion of both latent images and kernels by INR [53, 51] . INR represents signals as continuous functions rather than discrete valuel arrays, which in our case is expressed as follows,EQUATIONwhere (i, j) ∈ R 2 denotes the spatial co-ordinates of images/kernels, and Θ k , Θ x denotes the weights of NNs Φ k , Φ x for the kernel and image, respectively.Using INR for representing images/kernels is due to its inherent resolution-free property. By enabling the model to generate the prediction with higher/lower resolutions from the same learned model, INR facilitates seamless multi-scale processing and cross-scale interaction. In contrast, DIP-based NNs that directly maps noise to an image handle cross-scale interaction clumsily, requiring manual re-scaling or interpolation, which likely introduces artifacts in the prediction.Self-supervised cross-scale loss for BID: Without GT images, the sole constraint we have for training is y = k ⊗ x. Additional priors for regularization are necessary to alleviate over-fitting. Historically, the down-sampled version of y, denoted as y ↓s for scale s, has often been used to initiate the blur kernel estimate. However, it is important to note that the blurred image after down-sampling cannot be modeled by a convolution between the down-sampled image and a blur kernel:(x ⊗ k)↓ 2 ̸ = x↓ 2 ⊗ k↓ 2 .In this paper, we present a cross-scale constraint that accurately characterizes the connection between (y, x, k) at different scales. For instance, when s = 2, the cross-scale constraint is(x↓ 2 ) ⊗ (k↓ 2 ) = 1 4 (x ⊗ k)↓ 2) + 3 d=1 (x ⊗ g d )↓ 2 = 1 4 y↓ 2 + 3 d=1 (x ⊗ g d )↓ 2 ,where {g d } 3 d=1 denote three quadrature mirror filters (QMFs) [59] of the kernel k. This constraint can be easily implemented with INR-based resolution-free generators, and helps regularize the training of two generators to prevent likely over-fitting caused by the lack of GT images in loss function.Progressive cross-scale learning for BID: A well-established practice in traditional alternating iterative methods of BID to avoid convergence to trivial solutions involves solving the problem in a coarse-to-fine manner [67, 71] . This means the kernel is initially estimated for the blurred image at a coarse scale, followed by propagation of the kernel estimate to finer scales. This approach has proven effective in preventing iterations from converging to trivial solutions. Despite its effectiveness, the coarse-to-fine strategy remains under-exploited in existing selfsupervised BID methods, likely due to the resolution-fixed limitations of CNN or MLP based re-parametrization. Utilizing INR's resolution-free properties, we introduce a progressive learning strategy that begins training INR-based generators at a coarser scale and then refines training at finer scales. This scheme effectively addresses over-fitting and ensures convergence to the truth.Main contribution: Our main contributions are summarized as follows:• Leveraging the resolution-independent properties of INR for latent images/kernels, we propose a self-supervised cross-scale loss for training the NN without requiring GT images. • We introduce a progressive multi-scale learning approach for BID, specifically designed to mitigate potential over-fitting due to the non-linear nature of the BID problem.Extensive experiments conducted on a variety of datasets reveals that our proposed method outperforms existing self-supervised BID techniques as well as the supervised alternatives. 
related works: 
 Traditional non-learning methods for BID: Before deep learning became prevalent, regularization methods were the prominent approach for BID. These methods resolve solution ambiguities by imposing pre-defined priors on images and blur kernels, such as image gradient sparsity [8, 4, 24, 5, 67] , image patch recurrence [54, 38] , and Laplace priors in dark channels [42, 70] . These methods are based on some iteration scheme, and edge selection is an effective technique for better robustness and stability of the iteration [12, 65, 41, 15, 71] . Since regularization methods can be recast as Maximum-A-Posteriori (MAP) estimators in Bayesian inference, another class of BID methods is derived from variational Bayesian estimators [35, 14, 28, 63, 2] , as well as variational expectation maximization [29, 71] . The success of these methods requires rigorous tuning of hyper-parameters related to priors. In contrast, deep learning based methods can automatically learn priors from data.In recent years, there has been rapid progress in supervised deep learning methods for BID. By training over many pairs of blurred/truth images, these methods either explicitly estimate the blur kernel (e.g., [50, 7, 40, 34] or only estimate latent images (e.g., [66, 56, 25, 75, 20, 6, 11, 73, 72] ). The former is more efficient for handling uniform blurring. The latter is more general and can handle non-uniform blurring.Deep learning for BID with unpaired data or pre-trained model: There are also methods that are trained on unpaired dataset. Lu et al. [36] train the GAN for domain-specific deblurring. Wen et al. [61] propose a structure-aware deblurring method, and Chung et al. utilize generative priors from diffusion models to jointly estimate the blur kernel and latent images [13] . Nevertheless, these methods still require GT images in unpaired data for training or depend on pre-trained models, which can be challenging and present difficulties in adaptation. In contrast, our method goes through a self-supervised manner, addressing this limitation.Self-supervised deep learning for BID: These methods address the issues of data collection and dataset bias in supervised methods. Built on the DIP prior for image/kernel, Ren et al. [47] Most self-supervised BID methods are limited to handling uniform blurring, with Li et al. [33] being the only one capable of addressing both uniform and non-uniform blurring caused camera shake. Additionally, self-supervised super-resolution methods [3, 55] also involve the estimation of blur kernel. While both estimate the blur kernel. However, these methods differ in their input: low-resolution images versus high-resolution images. Moreover, the blur degree in BID usually is much more severe than in super-resolution.Coarse-to-fine estimation for BID: Coarse-to-fine schemes, which gradually refine the at different scales, have proven effective in traditional alternating iterative methods [65, 12, 52] . The traditional multi-scale methods typically use estimates from coarser scales solely as initial estimates for finer scales. In contrast, our approach for self-supervised deep BID leverages the cross-scale interaction of estimates with exact relations, rather than approximations. Coarse-to-fine scheme used in recent supervised NN-based methods for BID (e.g., [39, 56, 74, 11, 73] ) train the NN such that the estimates fit well the input image at different scales. Our multi-scale scheme is more for introducing a new scale consistency loss function specifically tailored for self-supervised BID. This allows for more precise and consistent refinement of the estimations across scales.INR for image recovery: In the domain of images, INR [10, 37, 43] encodes images as NN weights, mapping coordinates to pixel values for a compact, continuous representation. There are also multiscale extensions [27, 49] of INR for more efficient image compression. It has been used for image restoration tasks such as image in-painting, denoising, and super-resolution [51, 21, 64] . Besides images, INR also has been used for encoding defocus blur kernels [46] and motion blur kernels [77] .Our work is the first to exploit the resolution-free properties of INR for BID, leveraging these properties for a multi-scale approach and cross-scale interaction. 
methodology: 
 In this paper, we propose an INR-based progressive cross-scale self-supervised BID method. We first introduce the INR-based modeling for latent images and blur kernels, followed by the formulation of the self-supervised cross-scale loss function and the progressive learning strategy.In our approach, the blur kernel k and the latent image x are reparameterized by two INR models, Φ k and Φ x , respectively. Each model maps a spatial coordinate [i, j] to a pixel value. Let I k , I x ⊂ Z 2 denote the sets of spatial coordinates within the feasible domain for the blur kernel k and the latent image x, respectively. Then, they can be expressed asEQUATIONwhere Θ k , Θ x denote the NN weights of k, x. The INR-based re-parameterization (4) allows for the generation of kernel and image at any coordinates, providing a representation at arbitrary scales.INR-based multi-scale representation: Let ↓ m denotes the standard down-sampling operator:(k↓ m )[i, j] = k[i • m, j • m] and (x↓ m )[i, j] = x[i • m, j • m].Then, we can form both the kernel and the image in a dyadic pyramid, from the original scale and to coarser scales: Let k (0) = k, x (0) = x. Then defineEQUATIONFor any co-ordinate set I, we define its co-ordinate set at scale s asI (s) = {[i, j] : [2 s i, 2 s j] ∈ I}.Then, using INR-based model (4), k (s) and x (s) at scale s can be expressed asEQUATION) Such a multi-scale representation can facilitate the training of the INR-based generators for BID, as it allows for the generation of image details at arbitrary scales.Kernel generator: The kernel generator Φ k in our approach is a three-layer MLP with 128 feature nodes that takes coordinates normalized to [-1, 1] as input and adopts sinusoidal activation functions in every layer, following the Sinusoidal Representation Networks (SIRENs) [53] . The output layer employs a Softmax activation function to ensure the prediction satisfies the two physical constraints:EQUATIONThen, a Kernel Centering layer is applied to address possible positional shifts in estimated kernels for better training stability. For a kernel k ∈ R H×W satisfying (7) , calculating its centroids (c x , c y ) byc y = i,j i • k(i, j), c x = i,j j • k(i, j)The kernel k is then shifted by [⌊H/2⌋ -c y , ⌊W/2⌋ -c x ] to ensure it is centered geometrically.Image generator: The image generator Φ x adopts a U-Net structure with five blocks, separated by down-or up-sampling layers and connected by skip connections. Each block integrates a sequence of Convolution, Batch Normalization, and ReLU. The NN concludes with a 1 × 1 convolution layer followed by a Sigmoid layer to ensure the output image values remain within [0, 1]. To efficiently generate high-frequencies of images, following [51] , the input spatial coordinates are first transformed into a higher dimensional space using a high-frequency function γ(•) (sinusoidal):EQUATIONwhere p := (i, j) represents the normalized coordinate values within [-1, 1], and L is an positive integer. Note this encoding operation is used only in Φ x , not in Φ k .Without GT images, the only readily available loss function to train the generators is the fitting loss:EQUATIONwhere M f (•) is some distance metric. Such a fitting loss clearly is not sufficient to resolve solution ambiguities in BID. To address this, we introduce a scale consistency loss that regularize the training by enforcing cross-scale consistency of the estimation, which is based on the following proposition. Proposition 1. For a kernel (filter) k, let g 1 , g 2 , g 3 denote its associated QMF filters [59] defined byEQUATION) for any [m, n] ∈ I k . Then, we have the following relation between consecutive two dyadic scales:EQUATION)Proof. See Appendix A for the proof.As seen from Proposition 1, the down-sampled blurred image y↓ 2 does not equal to the convolution of the down-sampled latent image x↓ 2 and the down-sampled kernel k↓ 2 , incurring additional term3 d=1 (x ⊗ g d )↓ 2 .Therefore, down-sampled blurred images are fine for some initial estimation, not for regularizing the NN to obtain accurate estimation. To address this, based on Proposition 1, we introduce a scale consistency loss across two consecutive scales: for each scale s,EQUATION)where {g(s) d } 3d=1 denotes the QMF filter bank of the kernel k (s) defined by (10) , and by ( 6),EQUATIONThe scale-consistency loss ( 13)-( 14) enforces cross-scale consistency of the estimations at different scales, providing additional regularization for training two INR-based generators.To address the non-linear nature of BID and avoid convergence to trivial solutions, we introduce a progressive learning strategy that trains the INR-based generators at multiple scales. The training process consists of three stages at different scales. Define the fitting terms at different scale s byEQUATIONThe first stage serves as the initialization, operating at the coarsest scale with the fitting loss L (S0)fit . The second stage progressive refines the training from the scale S 0 to 0 with both the fitting loss and cross-scale consistency loss. Specifically, at scale s, the loss isL (s) fit + λL (s)cross where λ is a weight and L (s) cross is the cross-scale consistency loss defined by ( 13)- (14) . The third stage is the final tuning stage at scale 0 with L L (0) fit ; 6. Define k * = Φ k (I k ; Θ * k ) and x * = Φ x (I x ; Θ * x ).Structural Similarity Index Measure (SSIM) [60] is used for M f (•) in ( 15):M f (x, y) = (µ 2 x + µ 2 y + c 1 ) -1 (σ 2 x + σ 2 y + c 2 ) -1 (2µ x µ y + c 1 )(2σ x,y + c 2 ).For M c (•) in ( 13), note that convolution process in ( 13) can be efficiently computed by by transforming the convolution operation into pointwise multiplication in its discrete Fourier transform (DFT), denoted by F(•). Thus, we define M c (•) in frequency domain with ℓ 1 -norm: [26] . The methods marked with ∆ deblur the image by [23] using the estimated kernel, a standard protocol for evaluating kernel estimation accuracy in BID. The methods marked with * are retrained on the BSD-D dataset [48] .M c (x, y) = ∥F(x) -F(y)∥ 1 . 
experiments: 
 Implementation details: The training consists of 5000 iterations across three stages. The first stage operates at the coarsest scale S 0 with 500 iterations. The second stage refines training from scale S 0 to scale 0, with 500 iterations per scale. The final stage is tuning at scale 0 for the remaining iterations. The NN is trained using the Adam optimizer with a batch size of 1. The initial learning rates for the image and kernel generators are set to 5 × 10 -3 and 5 × 10 -5 , respectively, decreasing to half their values every 2000 iterations. The weight λ in the loss function is set to 0.001 to keep the values of the two loss terms L cross and L fit in the same order. For comparison, we use results from the literature when available; otherwise, we use pre-trained models or train from the provided code to achieve optimal performance. The code of the proposed method is available on Github. 1 .Two metrics, PSNR (peak signal-to-noise ratio) and SSIM, are used for performance evaluation.Following SelfDeblur [47] , we compute PSNR/SSIM after finding the best shift between GT and the result to handle shift ambiguity. In the tables, Bold in blue indicates the best among all supervised methods, Bold in black indicates the best, and underline the second-best among all GT-free methods.Synthetic dataset with uniform blurring from Lai et al. [26] : This dataset consists of 100 images categorized into five groups: Manmade, Natural, People, Saturated, and Text, and covers 4 different kernels whose size ranges from 31 × 31 to 75 × 75. For this dataset, S 0 is set to 2 . As the focus of BID is on accurately estimating the blur kernel. Thus, a two-stage evaluation protocol, as outlined by [47] , is also used in our test. Note that most supervised methods only generate clear images without kernels, and thus their evaluation is solely on their output images. supervised methods, except for [20] , we use their models pre-trained on the GoPro dataset [39] with both uniform and non-uniform blurring. For two recent methods, MPRNet [73] and Restormer [72] , we also include their models retrained on the BSD-D dataset [48] with only uniform blurring.The comparison in Tab. 1 shows that our method achieved the best performance. Note that for two supervised methods, MPRNet [73] and Restormer [72] , their models trained on a dataset with only uniform blur performs better on testing data with uniform blur, when compared to the ones trained on the dataset with non-uniform blurring. However, this improvement is still not enough to match the performance of our method. The main reason is that existing supervised methods target general blurring and overlook the physics prior of image formation, specifically the convolution model for uniform blurring. As a result, they underperform compared to our self-supervised method, which leverages this prior. This highlights the generalization issues inherent in supervised learning approaches, whose performance heavily depends the correlation degree between the training and testing data. Refer to Appendix F.1 for visual comparisons of different methods.Synthetic dataset with modest non-uniform blurring from Köhler et al. [22] : The benchmark dataset Köhler [22] comprises 48 motion-blurred images which blurring is not exactly uniform. This dataset is for evaluating the robustness of BID method to handle modest non-uniform blurring. Totally 17 methods are selected for comparison, and S 0 is also set to 2. Following the evaluation protocol in [33] , we use the average PSNR and MSSIM (multi-scale SSIM) as the evaluation metrics. Tab. 2 shows that our method outperforms all others in terms of both PSNR and MSSIM, indicating its robustness in handling modest non-uniform blur. Refer to Appendix F.2 for visual comparisons.Real-world dataset from [48] .Real-world dataset RealBlur [48] with small/modest blurring: Our proposed method is also evaluated in another real-world dataset published in [48] : RealBlur-J and RealBlur-R, both containing 980 real-world blurry images. See Tab. 3 for the results. It can be seen that, unlike the experiments on synthetic datasets, there is a gap between the proposed method and supervised methods. This is understandable, as our method cannot access GT images, whereas the supervised methods can. However, our method still outperforms all traditional methods and existing self-supervised methods. Refer to Appendix F.4 for visual comparisons of different methods.In microscopic imaging, acquired images often suffer from blur due to optical limitations, outof-focus elements, specimen motion, and the diffraction limit of light. Following [45] , the test dataset consists of 120 images, covering 24 images from the test subsets "Confocal_BPAE_B" and "TwoPhoton_MICE", and includes 3 Gaussian point spread functions (PSFs) and 2 Poisson PSFs as the blur kernels. For the compared supervised methods, Restomer [72] and INIKNet [46] , we retrained their models using the microscopic dataset [45] . It can be seen from Tab. 4 that our method outperforms all other self-supervised methods and is comparable to the supervised methods trained on the microscopic dataset. Refer to Appendix F.5 for visual comparisons of different methods. The ablation study is conducted on the Lai et al. dataset [26] . The results are shown in Tab. 5Effectiveness of self-supervised cross-scale consistency loss: To evaluate the gain from the proposed cross-scale consistency loss functions, we retrain the NN using only the fitting loss L fit for each scale, i.e., without (w/o) L cross . Tab. 5 shows an average gain of about 0.72 dB in PSNR with the cross-scale consistency loss, demonstrating its effectiveness.Effective of progressive coarse-to-fine training: We first examine the effectiveness of multi-scale training by performing the training only at the original scale. Tab. 5 shows about a 1 dB loss in PSNR, indicating multi-scale training's contribution. To evaluate the progressive training strategy, we train the NN with the sum of all loss functions at three scales, which resulting in a 3dB loss. This clearly indicates that progressive training is critical for effectively utilizing the multi-scale scheme. The reason is that the loss function at the coarser scale emphasizes lower frequencies since the image at coarser scale retains low but loses high frequencies. Thus, a NN trained on the sum of loss functions across 3 scales focuses more on low frequencies than one trained only at the finest scale, which fits both low and high frequencies. Therefore, a scale-progressive training scheme is more effective than a joint multi-scale loss. In scale-progressive training, the estimation at the coarser scale provides a strong initialization for the finer scales. The final result is obtained by applying the fitting term exclusively at the finest scale.This ablation study is to evaluate the necessity of using INR for re-parametrizatrion of image and/or kernel, instead of using the NN that maps an image to an image. In the study, we separately replace our INR-based kernel/image generator with MLP/CNN based representation adopted in [47] , whose down-sampled versions of images/kernels at different scales are generated by standard down-sampling process.Direct down-sampling vs. using multi-scale grid in INR: To verify the benefit of resolution-free property of INR, we also consider generating multi-scale versions of image/kernel by standard down-sampling using bilinear interpolation, on their high resolution version from INPs, same to a CNN-based representation. This is referred to as "Multi-scale by direct ↓". Tab. 5 shows that the performance of multi-scale by standard down-sampling process is very poor. Clearly multi-scale representation by standard downsampling process is not as effective as resolution-free INR.The computational effciency of the propoposed method is compared to three most related selfsupervised BID methods. The results are reported in terms of running time, number of parameters, and memory usage, when processing a 256 × 256 image with a 31 × 31 blur kernel on an NVIDIA 3090 RTX GPU. See Table 6 for the details. It can be seen that the proposed method achieves a balance between computational cost and deblurring performance.  
discussion and conclusion: 
 This paper introduces a self-supervised method for BID, which does not require GT images. Leveraging on the resolution-free representation of INR for image/kernel, we propose a self-supervised cross-scale consistency loss function and a progressive coarse-to-fine training strategy. The proposed method significantly outperforms existing methods in extensive experiments.There are two limitations of the proposed self-supervised method. The first is the computational cost for processing a large number of images, as the method requires training the model for each individual sample. A potential solution is to explore the usage of the proposed techniques in meta-learning or testing-time adaptation. This would allow the proposed technique to rapidly adapt a pre-trained model instead of to train the NN from scratch. The second limitation is that the proposed method is only applicable to handle uniform blurring, as it relies on the convolution model. Extending this approach to handle non-uniform blur will be another important direction for future research. 
appendix: 
 Below we provide additional details and results which are not presented in the main manuscript. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The paper utilizes several pre-existing datasets and codebases, which are properly cited within the manuscript.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. The proof is based on the convolution theorem and the properties of the DFT. Without loss of generality, we assume that the image x and the kernel k are of size 2M × 2N . Let x↓ 2 and k↓ 2 denote the down-sampled images of x and k, respectively. Let F denote the discrete Fourier transform (DFT). Recall that for a signal z ∈ R 2M ×2N , its DFT is defined as:F(z)[m, n] = 2M -1 ℓ=0 2N -1 j=0 z[ℓ, j]e -i2π m 2M ℓ e -i2π n 2N j .Then, for a down-sampled version of the signal z↓ 2 , whose DFT isF(z↓ 2 )[m, n] = M -1 ℓ=0 N -1 j=0 z[2ℓ, 2j]e -i2π m M ℓ e -i2π n N j .The DFTs of the down-sampled signals x↓ 2 and k↓ 2 can be expressed as follows: For each frequency [m, n], we haveEQUATIONEQUATIONandEQUATIONBy convolution theorem, for y = x ⊗ k, we have thatF(y)[m, n] = F(x)[m, n] • F(k)[m, n]Thus,EQUATIONApplying convolution theorem on x↓ 2 ⊗ k↓ 2 again, we haveEQUATIONPlugging in ( 16) and (17) into the above equation, we haveEQUATIONThen, theirs DFTs are given byEQUATIONThen, the expression of the right-hand side of Equation ( 21) can be rewritten as:EQUATIONBy convolution theorem, we have then(x↓ 2 ⊗ k↓ 2 ) = 1 4 ((x ⊗ k)↓ 2 ) + 3 d=1 ((x ⊗ g d )↓ 2 ) ,The proof is complete.In this ablation study, we investigate how different choices of scale S 0 impact the performance on motion deblurring tasks. The experiments are conducted using Lai et al.'s dataset. We compare results using the same framework with different S 0 values: single-scale, two-scale, three-scale, and four-scale (i.e., S 0 ∈ {0, 1, 2, 3}). Each setting undergoes a total of 5000 training epochs, with each stage except the last one consisting of 500 training epochs.As shown in Tab. 7, multi-scale frameworks consistently outperform single-scale setups. Among the multi-scale configurations, the three-scale model demonstrates the most significant improvement. However, the four-scale model shows only a slight improvement over the single-scale setup. This minimal gain can be attributed to the initial stages of the four-scale framework, where the target image, resized to 1/8 of the original size, is far from the true image. This highly erroneous initialization causes more harm than benefit, hindering the NN's convergence to the true image. The following Tab. 8 summarizes the hyper-parameters, specifically the range of kernel sizes and the coarsest scale S 0 used in our model across various datasets. These parameters are selected based on the specific characteristics of each dataset, including image size.Please see Tab. 9 for the details of the architectures used for image generative network Φ x . In Tab. 9, the "CBR" denotes the subblock for three successive layers: Conv2d+BatchNorm2d+ReLU. The "CBR/Down" means modifying the Conv2d layer in "CBR" with stride (2, 2), which performs downsampling operations. The "Up" means the upsampling layer with bilinear interpolation. The "Cat(m,n)" denotes the concatenation of the outputs from layers No.m and No.n. The proposed self-supervised learning method for deblurring images has the potential to impact a wide range of applications, including surveillance security, scientific research, and digital media restoration. By improving the accuracy and clarity of images, our research can facilitate deeper insights and more effective interventions in these fields.In surveillance, higher clarity images can enhance public safety by providing more detailed visual information. However, this also raises concerns about privacy and the potential for mass surveillance. In scientific research, improved image quality can lead to better data and more significant discoveries, though there is a risk that overly processed images could misrepresent the original data. In digital media restoration, while the technique helps preserve cultural heritage, it also poses the risk of altering historical records.Despite these possible concerns, our goal is to contribute to enhancing image clarity in critical areas such as public safety, scientific research, and cultural preservation. We emphasize the responsible application and continuous improvement of this technology to mitigate potential risks and maximize its positive impact.F Visual comparisons of the results in the experiments In this section, we visualize the results of different methods on some examples from Lai et al.'s dataset [26] , which is known for its severe blurring effects. Figs. 1 -6 demonstrate that our method consistently produces results with sharper details and fewer artifacts compared to existing methods, showcasing its effectiveness in addressing significant blur challenges. In contrast, supervised learning methods trained on external datasets yield poor quality results, highlighting the limited generalization performance of supervised approaches when dealing with complex real-world blurring. MPRNet [73] MIMO-UNet [11] Restormer [72] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT MPRNet [73] MIMO-UNet [11] Restormer [72] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT MPRNet [73] MIMO-UNet [11] Restormer [72] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT Figure 3 : Visual results on the dataset of Lai et al. [26] Blurry image Pan-DCP [42] MIMO-UNet [11] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT image MPRNet [73] MIMO-UNet [11] Restormer [72] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT Blurry image Pan-DCP [42] MIMO-UNet [11] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT image In this section, we display the visual results on Köhler et al.'s dataset [22] , which is characterized by its non-uniform blurring effects. From Fig. 7 -Fig. 8 , our method demonstrats satisfying performance on this challenging dataset, effectively handling the intricate and variable blurring effects. Furthermore, we compare our results with those obtained using the Diffusion-based method, BlindDPS [13] , revealing its difficulty in adapting to blurring effects and image types that deviate from its pre-trained model's training data.Kaufman & Fattal [20] BlindDPS [13] SelfDeblur [47] MCEM [33] Ours GT image Blurry image Cho &Lee [12] Kaufman & Fattal [20] BlindDPS [13] SelfDeblur [47] MCEM [33] Ours GT image In this section, we present a visual comparison of the results from state-of-the-art supervised learning methods [73, 11, 72] and our approach on the RealBlur dataset. Although our method achieves slightly lower average PSNR and SSIM values across the entire dataset compared to these representative supervised learning methods, the visual evidence presented in Fig. 10 demonstrates that our results are often sharper. In other words, the gap between our method and the supervised methods is smaller in terms of visual quality than indicated by the quantitative metrics.Blurry image MIMO-Unet [11] MPRNet [73] Restormer [72] Ours GT image This section provides a comparison between our method and other methods in the task of microscopic deconvolution. Our approach consistently restores images with more texture and details. Iteration=500 Iteration=1000 Iteration=1500 Iteration=5000Blurry image x (2) [I (2) x ]x (1) [I (1) x ] Apart from validating the efficacy of our approach through the quality of the final results, it is also interestting to see the intermediate status of the proposed method. To this end, we use "manmade_01" from Lai et al.'s dataset [26] as a case study, visualizing intermediate results to better understand the progress through different stages to the final output. As shown in Figure 13 , in the first stage, we obtain an initial representation at the smallest scale during initialization. In the second and third stages, we achieve results at larger scales with finer details. In the final stage, the neural network builds on the groundwork laid by the preceding stages to refine the output, ultimately generating the desired clear image.x (0) [I (0) x ] x (0) [I (0) x ] GT imageI Experiments on Levin et al.'s dataset [28] with small kernel size. Pan-DCP [42] MPRNet [73] Restormer [72] BlindDPS [13] SelfDeblur [47] MCEM [33] VDIP [18] Ours GT • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/tjzhang-nus/Deblur-INR 


Decision: This paper receives two borderline accept and two weak accept. This paper proposes a self-supervised method for blind image deconvolution. In the rebuttal, the authors have adequately addressed most of the reviewers' major concerns.  A more detailed motivation for why INR and the progressive cross-scale can benefit self-supervised learning is needed in the revised version.

Reviews: review: paper_summary: paper_summary: This paper presents an approach to solve blind image deconvolution. The authors use multiscale Implicit Neural Representations (INRs) to depict both the latent image and the blur kernel. In addition, they propose a method that incorporates a cross-scale consistency loss and a progressive scale optimization process. Experimental results demonstrate superior performance on small-scale simulated datasets and competitive results compared to state-of-the-art methods on large-scale and real-world datasets.
strength_weakness: strength_weakness: 1. There exist prior works on Multiscale Implicit Neural Representations. The authors should provide a more detailed discussion on the similarities and differences between their proposed method and these previous works. For instance:
    

- “PINs: Progressive Implicit Networks for Multi-Scale Neural Representations” by Landgraf, Zoe, Alexander Sorkine Hornung, and Ricardo S. Cabral, presented at ICML 2022.
    
- “Miner: Multiscale Implicit Neural Representation” by Saragadam, Vishwanath, et al., presented at ECCV 2022.
    

2. The method appears to require at least one hyperparameter (the number of scales) to be set based on the target dataset. This suggests that the method is not entirely blind.
questions: questions: \- Minor points:

    \- In Algorithm 1, please indicate whether the loop end condition in step 4 includes ‘0’.

    \- On line 259, the sentence “the test dataset consists of 120 images“ appears to be repeated.
limitations: limitations: Blind deconvolution, when limited to the uniform case, may not be particularly useful for real-world problems. While the authors have acknowledged this, it would be beneficial to include experiments demonstrating the effectiveness of the proposed method’s components on widely-cited non-uniform datasets, such as GoPro [40]. This would provide valuable guidance for future research in the field.
paper_summary: paper_summary: This paper proposed a self-supervised method for blind image deconvolution (BID). The main idea is to introduce the implicit neural representation (INR) technique for representing both the blur kernel and the image, such that they can be parameterized at different scales using a single model. With such a INR representation, supervision can be enforced in a cross-scale way to enhance the efficacy of the model. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The presentation of this manuscript should be improved. For example, a large part of Section 1.2 can be put into Section 3.
2. Experiments are insufficient in the following sense:
- A recent self-supervised BID method [1] is not compared.
- The cross-scale supervision training strategy can also be applied other self-supervised methods by with simple downscale operation, which is not tested.
- The use of the loss functions is not well justified. For example, it is not clear what the performance could be if other more popular ones are used, such as L2 and L1 loss.
- Model complexity comparison in addition to the running time in Appendix D should be provided.

References:

[1] D. Huo et al. Blind Image Deconvolution Using Variational Deep Image Prior. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 45(10): 11472-11483, 2023.
questions: questions: 1. To my knowledge, the performance of fully supervised methods highly depends on the training data. So I would like to know how the supervised methods were trained on the Lai, Kohler, and Levin datasets, since only test images are contained in them.
2. The method is constructed based on the uniform blurring assumption, and thus it should be discussed why it can perform well on the Kohler dataset which focuses on non-uniform blurring.
3. Typos: In Line 31, $\delta,y=k\otimes x$ should be $\delta\otimes y=k\otimes x$.
limitations: limitations: The authors discussed the limitations of the method, which are reasonable.
paper_summary: paper_summary: This paper introduces a self-supervised method for BID that does not require GT images. By leveraging an exact relationship among the blurred image, latent image, and blur kernel across consecutive scales, this paper propose an **effective cross-scale consistency loss** implemented by representing the image and kernel with **implicit neural representations** (INRs), whose resolution-free property **enables consistent yet efficient computation for network training at multiple scales**. The experimental part  verifies that proposed method outperforms some existing self supervised methods on several datasets.
strength_weakness: strength_weakness: - The INRs framework and progressive learning mechanism on which the paper is based are common in image restoration, the degree of innovation is average except for cross-scale estimation consistency constraint.
- With the development of BID or the larger blind image restoration area, the degradation settings of different methods are somewhat different. When describing the paper’s setting (Eq.1), in addition to comparison methods, introducing the existing blurring degradation settings (uniform kernel/non-uniform kernel/implicit modeling) and their differences and application value is necessary. The discussion of this part is missing, which causes readers/researchers confused by different method settings and applicability.
- The related work section of the paper does not fully cite and introduce recent learning-based self-supervised or frequency estimation degradation and some important works, such as KernelGAN(NeurIPS2019), FCA(AAAI2021), S2K(NeurIPS2021), etc. These solutions also involve deconvolution and propose estimation methods.
- Some descriptions of the method principles or experimental settings are vague or problematic, see Questions.
questions: questions: - There are some confusing points in Tab.1: (1) The results of the supervised method are of little reference value for they are trained on non-convolutional modeling datasets(GoPro). (2) According to the ablation experiment, even without the main contribution (1) mentioned in line 73, the PSNR(24.44) is almost the same as the second best method(MCEM) (24.55). The uniform blur setting is inconsistent with some existing methods (such as the non-uniform modeling used by MCEM). Are those self-supervised method retrained for comparison? 
- The proposed uniform blurring method is better than MCEM in the non-uniform deblurring experiment (Tab.2), which is a bit contradictory. Is this an advantage brought by individual sample training rather than the method itself?
- In the multi-scale training process, what is the effect of x and k estimated in the previous scale on the next scale? From the image, it can be understood that coarse-to-fine is getting clearer. How does k change in this process? In addition, in Tab.5, the average PSNR of training only at the original scale is two points higher than that of training with at three scales, indicating that the optimization direction of BID at different scales is inconsistent. In this case, why does training with progressive scales bring better performance?
- According to the existing conclusions in the BID field in lines 94-98 of the paper, the second contribution point of the paper summarized in lines 107-108 is not important. In addition, the use of multi-scale coarse-to-fine can alleviate the ill-posed problem of multiple solutions. There is no analysis or literature references provided though some work has been done before indeed.
- In Lines 191-192, it claims the role of the cross-scale estimation consistency is to limit the ambiguous solution of BID. Though a rational analysis of the approach and ablation experiments are provided, readers may wonder what is the motivation for designing this regularization term and its actual impact in BID process. If there is some intuitive analysis or explanation, it will be easier for them to accept.
- For the standard downsampling implemented by the coordinate index in line 165, is interpolation  need  when the image size is not even? Is the downsampling used in the ablation experiment to compare image-to-image NNs done in the same way or with more common interpolation methods such as bicubic or bilinear?
limitations: limitations: The authors have  adequately addressed the limitations and potential negative societal impact of their work as follows:
- Computational cost for processing a large number of images, as the method requires training the model for each individual sample.
- The proposed method is only applicable to handle uniform blurring, as it relies on the convolution model. 

Considering the practical application value and the single-image training and reasoning method proposed, the paper should compare the computational cost of the single-image reasoning results and the comparison with existing methods.

As mentioned above, it is necessary to add discussions about different setting and it is confusing that proposed method performs better than MCEM in non-uniform setting.
paper_summary: paper_summary: This paper proposes a blind image blurring method which reparametrizes the latent images the blurring kernel by the implicit neural representations (INRs). In addition, the authors also propose a cross-scale consistency loss. The authors validate the effectiveness of their method on several datasets.
strength_weakness: strength_weakness: 1) The idea of using a deep neural network for latent images and blurring kernel reparametrizatrion is not new. The current proposed method is very similar to the one in ref1. But I do not see the authors have discussed the differences between the proposed method and the method in Ref1. Thus, I doubt the novelty of this proposed method.

2) The current design, the authors use different network architectures to generate images and kernel. But how to control or balance the complexity between latent images generator and the kernel generator?  The authors need to make it more clear why use these specific network design for images and kernel.

3) If the batch size is 1, will it be better to use other norms instead of batch norm?

4) The authors should also compare their method with the one in Ref1 as the Ref1 seems to be the more recent one compared to other self-supervised learning method.

5) The authors should also show how the generated kernel looks like. Is the generated kernel feasible? Is it close to the ground-truth kernel?

Re1: Zhuang, Zhong, et al. "Blind image deblurring with unknown kernel size and substantial noise." International Journal of Computer Vision 132.2 (2024): 319-348.


=========
score has been revised after reading rebuttals.
questions: questions: I only see very limited novelty of this paper. The proposed idea is similar to the one in Ref1. The only contribution I can see is the cross-scale consistency loss. I list the details in the [Weaknesses].

Re1: Zhuang, Zhong, et al. "Blind image deblurring with unknown kernel size and substantial noise." International Journal of Computer Vision 132.2 (2024): 319-348.
limitations: limitations: Yes.

Comments: comment: Thank you so much for your rebuttal. I have carefully read the rebuttal as well as the comments and discussions. I raised my rating accordingly based on all the updated information (rebuttal, other available reviews, and discussions).
Dear reviewer, 

Thank you again for your time and effort in reviewing our work. We have carefully addressed all the concerns and questions you raised, including the point regarding the specific paper you mentioned for discussion.

We understand that the review process is demanding. However, as the deadline for discussion on the response is approaching, we wanted to ensure that our responses are clear and adequately address your concerns and questions. Any further clarification or specific questions related to our response would be invaluable to us. Thank you once again for your time and for considering our responses.
Thanks for replying and clearing up most of my concerns. From the additional experiments provided, self-supervised methods are indeed superior to supervised methods in cross-domain deblurring. It is an important step to make fair experiments clear about the degradation settings of different methods and compare them accordingly. I hope the paper will provide additional explanations in this regard. Based on the rebuttal, I give the final vote.
I appreciate the authors’ efforts to address the comments made by the reviewers. The responses made by the authors, the promised changes to the text, and the additional results provided seem to have addressed many of the concerns raised. I look forward to seeing the updated feedback from the other reviewers.