Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution

1 Introduction . As a special category of image Super-Resolution ( SR ) [ 11 , 6 , 22 ] , blind super-resolution [ 28 , 3 , 48 ] is an active research topic towards real-world restoration applications , and has attracted increasing attention . Blind SR aims at reconstructing a high-resolution image from its low-resolution counterpart which contains unknown and complex degradations ( e.g. , blur , noise ) . Recent deep-learning-based blind SR methods typically consist of two branches : one for degradation prediction and the other for conditional restorations [ 14 , 25 , 39 ] , as illustrated in Fig . 1 . This design incorporates human knowledge about the degradation process [ 8 , 23 ] and is consistent in classical blind SR methods [ 28 , 11 ] , thus implying a certain degree of interpretability . With the powerful representation ability of deep learning , it is natural to wonder whether a unified one-branch network can effectively address the blind SR problem without the delicate two-branch ∗Equal contributions . Liangbin Xie is an intern in ARC Lab , Tencent PCG . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . design from human prior knowledge . Thus , we conduct preliminary experiments on several state-ofthe-art methods [ 25 , 39 ] ( more details in Sec . 3 ) , and draw the observation that a unified one-branch network can achieve comparable results with similar computation budgets under proper training strategies , as shown in Fig . 1 . Despite its comparable performance , the one-branch network is more like a ‘ black-box ’ , and we are curious about its connection with delicate two-branch designs with higher interpretability . Specifically , the phenomenon mentioned above motivates us to raise two key questions : 1 ) Could one-branch networks automatically learn to distinguish degradations as what we specially design in two-branch methods ? 2 ) Are there any small sub-network ( i.e. , a set of filters ) existing for a specific degradation ? For these open questions , we lack systematic investigations and analysis tools to gain further understandings . In this paper , we make the first attempt to explore the underlying mechanism of blind SR networks . Our key findings are as follows : 1 ) In one-branch blind SR networks , we are able to find a very small number of ( at least to 1 % ) discriminative filters for each specific degradation ( e.g. , blur , noise ) . When we mask these discovered filters for a specific degradation , the corresponding function is eliminated while functions for other degradations are maintained ( as shown in Fig . 2 , more details in Sec . 5.2.1 ) . 2 ) Except for the filter weights , the locations and connections of the discovered filters also play an essential role in the network function for a specific degradation ( see Sec . 5.2.2 ) . 3 ) Based on these discovered filters , we could easily predict the degradation of input images without training in the supervision of degradation labels ( see Sec . 5.3 ) . In a word , a unified one-branch network is similar to a well-designed two-branch network in the working mechanism . The degradation prediction branch and several small sub-networks for various degradations could be automatically learned within the one-branch network , even it is trained without any supervision about degradation distinctions . To analyze the mysterious blind SR networks and draw the above key findings , we propose a novel Filter Attribution method based on Integral Gradient ( FAIG ) , aiming at finding core filters in a network that make the greatest contribution to the function of a specific degradation removal . Different from previous integral gradient methods [ 37 , 36 ] that cumulate gradients along paths in the pixel/feature space , our FAIG utilizes paths in the parameter space , which has a clearer meaning in attributing network functional alterations to filter changes ( i.e. , the changes of filter parameters rather than inputs result in the network function alteration ) . Built on FAIG , we further develop a simple yet effective method to predict the degradation of an input image ( see Sec . 4.3 ) . To summarize , the contributions are three-fold . 1 ) Our findings provide a better understanding of the mechanism under blind SR networks , especially , bringing insightful connections between popular two-branch methods and unified one-branch networks . 2 ) The discovered discriminative filters for specific degradations allow us not only to perform degradation prediction , but also achieve a controllable adjustment of restoration strength without introducing extra parameters . 3 ) Exploiting the interpretability of blind SR would bring great significance for future works in i ) designing more efficient architectures ; ii ) diagnosing an SR network , such as determining the boundary of network restoration capacity and improving algorithm robustness . 2 Related Work . The super-resolution field has witnessed a variety of developments since the pioneer work of SRCNN [ 6 , 7 ] . Different architectures are proposed , such as deeper networks [ 17 , 19 , 15 ] , residual blocks [ 20 , 22 , 51 , 43 , 41 ] , recurrent architectures [ 18 , 38 ] , and attention mechanism [ 50 , 4 , 24 , 27 ] . Most of these methods are unified one-branch networks and achieve excellent performance on SR with a fixed downsampling ( e.g. , bicubic ) kernel . Recently , blind SR has attracted increasing attention to solve real-world restoration problems with complicated degradations . Classical blind SR methods typically incorporate human understanding of degradation process [ 8 , 23 ] , and are divided into two stages : degradation estimation ( especially blur kernels ) [ 28 ] and super-resolution based on the estimated degradation [ 11 , 45 ] . Deep-learning-based methods inherit this design and develop two-branch blind SR networks . KernelGAN [ 3 ] leverages the internal distribution of image patches to estimate blur kernel . DASR [ 39 ] learns degradation representation and prediction with contrastive learning . SRMD [ 48 ] and [ 49 , 46 ] incorporate these estimated degradations into CNN networks for restoration . IKC [ 14 ] and DAN [ 25 ] further propose to jointly perform degradation prediction and restoration in an iterative manner . Recent works also attempt to address blind SR within a one-branch networks [ 40 , 47 ] . There are few related works investigating the relationship between two-branch and one-branch networks for blind SR . Moreover , the underlying mechanism behind blind SR is still mysterious . Network interpretation methods for high-level visions can be divided into two categories : gradientbased methods [ 44 , 33 , 35 , 34 , 1 , 53 , 32 ] and perturbation-based approaches [ 9 , 52 , 30 , 31 ] . The latter usually localizes the discriminative image regions by performing perturbation to the input or neurons [ 10 ] . They heavily depend on lots of samplings and resources to guarantee performance . The most relevant gradient-based method to ours is Integrated Gradients ( IG ) [ 37 , 36 ] , which accumulates gradients at all points along a path from the baseline input to target input . IntInf [ 21 ] and neuron conductance [ 5 ] further extend feature-important IG to neurons . Unlike these methods that cumulate gradients along paths in the pixel/feature space , our FAIG utilizes paths in the parameter space for attributing network functional alterations to filter changes . Recently , Gu and Dong [ 13 ] introduce an interpretation method with local attribution maps to SR networks , while we further try to uncover the underlying mechanism in blind SR networks . Interpolation in the network parameter space has been used to explore the generalization and flat property of loss landscapes [ 12 , 16 ] . It is also employed in image manipulation for continuous imagery effect transition [ 42 ] . In this work , we utilize the paths in parameter space ( with interpolation ) to find discriminative filters for specific degradations . 3 Preliminary . Blind SR aims to restore high-resolution images from low-resolution ones with unknown and complex degradations , e.g. , blur , noise and downsampling . The classical degradation model [ 8 ] is usually adopted to synthetic the low-resolution input . Generally , the ground-truth image xgt is first convolved with Gaussian blur kernel k. Then , a downsampling operation with scale factor r is performed . Finally , the low-resolution x is obtained by adding white Gaussian noise n : x = ( xgt ⊛ k ) ↓r +n . ( 1 ) Recent blind SR methods including IKC [ 14 ] , DAN [ 25 ] and DASR [ 39 ] typically adopt a two-branch design , as shown in Fig . 1 . We then compare the two-branch design with commonly-used one-branch networks on two state-of-the-art methods : DAN and DASR . For a fair comparison , all experiments are conducted with their corresponding officially released codes , and the settings remain unchanged except the network structures . Specifically , SRResNet without batch normalization [ 20 ] is employed as the one-branch network . We adjust the number of residual blocks to match a similar computation budget ( FLOPs and parameters ) to two-branch methods . We compare them on the blur and blur+noise settings ( more specifications are in the supp . material ) . The PSNR comparisons are shown in Tab . 1 and Fig . 1 . It is observed that a unified one-branch network can also achieve comparable results under similar computation budgets on both settings . This interesting observation motivates us to investigate the underlying mechanism of blind SR networks , especially one-branch networks . It is worth noting that we do not claim that one-branch networks are always better than two-branch designs . The two-branch methods that incorporate domain-specific knowledge may have an advantage in training stability and robustness . In this paper , this observation is our primal motivation for interpreting blind SR networks . Integrated Gradient ( IG ) method [ 37 , 36 ] is usually used in high-level tasks ( e.g. , classification network ) . It attributes the most important input components ( e.g. , pixels in input images ) that affect the network predictions . Suppose we have a deep classification network F : Rd 7→ [ 0 , 1 ] . Let x ∈ Rd be the input image and x̄ ∈ Rd be the baseline input . x̄ is usually a black image or blurry images . The IG method accumulates the gradients at all points along a straight-line path in Rd from the baseline x̄ to the input x . Formally , the integrated gradient along the ith dimension for an input x and baseline x̄ is defined as : IGi ( x ) = ( xi − x̄i ) × ∫ 1 α=0 ∂F ( x̄+ α× ( x− x̄ ) ) ∂xi dα , ( 2 ) where xi is its ith pixel value and ∂F ( x ) ∂xi is the gradient of F along the ith dimension at x . Each change in the input component adds up to the final prediction difference ∑ i IGi ( x ) = F ( x ) − F ( x̄ ) . Intuitively , the function F varies from a less informative value ( produced by the baseline x̄ ) to its final informative value . The gradients of F with respect to the image pixels explain each step of the variation in the value of F . The integration over the path gradients cumulates these small attributions and accounts for the difference of network outputs . In other words , the IG could interpret deep networks by finding out the most important input components ( or features ) that explain the network predictions .

Summary: Existing methods for blind super resolution have recently adopted a design consisting of two branches where one branch is responsible for predicting (or representing) the degradation and the other branch is responsible for reconstruction while conditioning on the degradation information of the first branch. This paper highlights that a single network with one branch can achieve a similar performance as the two-branch approaches and analyses this in detail. More specifically, the authors design a filter attribution method to find most discriminative filters as opposed to most discriminative input pixels or features. This analysis also allows the development of a simple predictor for degradation.