Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution

1 Introduction . As a special category of image Super-Resolution ( SR ) [ 11 , 6 , 22 ] , blind super-resolution [ 28 , 3 , 48 ] is an active research topic towards real-world restoration applications , and has attracted increasing attention . Blind SR aims at reconstructing a high-resolution image from its low-resolution counterpart which contains unknown and complex degradations ( e.g. , blur , noise ) . Recent deep-learning-based blind SR methods typically consist of two branches : one for degradation prediction and the other for conditional restorations [ 14 , 25 , 39 ] , as illustrated in Fig . 1 . This design incorporates human knowledge about the degradation process [ 8 , 23 ] and is consistent in classical blind SR methods [ 28 , 11 ] , thus implying a certain degree of interpretability . With the powerful representation ability of deep learning , it is natural to wonder whether a unified one-branch network can effectively address the blind SR problem without the delicate two-branch ∗Equal contributions . Liangbin Xie is an intern in ARC Lab , Tencent PCG . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . design from human prior knowledge . Thus , we conduct preliminary experiments on several state-ofthe-art methods [ 25 , 39 ] ( more details in Sec . 3 ) , and draw the observation that a unified one-branch network can achieve comparable results with similar computation budgets under proper training strategies , as shown in Fig . 1 . Despite its comparable performance , the one-branch network is more like a ‘ black-box ’ , and we are curious about its connection with delicate two-branch designs with higher interpretability . Specifically , the phenomenon mentioned above motivates us to raise two key questions : 1 ) Could one-branch networks automatically learn to distinguish degradations as what we specially design in two-branch methods ? 2 ) Are there any small sub-network ( i.e. , a set of filters ) existing for a specific degradation ? For these open questions , we lack systematic investigations and analysis tools to gain further understandings . In this paper , we make the first attempt to explore the underlying mechanism of blind SR networks . Our key findings are as follows : 1 ) In one-branch blind SR networks , we are able to find a very small number of ( at least to 1 % ) discriminative filters for each specific degradation ( e.g. , blur , noise ) . When we mask these discovered filters for a specific degradation , the corresponding function is eliminated while functions for other degradations are maintained ( as shown in Fig . 2 , more details in Sec . 5.2.1 ) . 2 ) Except for the filter weights , the locations and connections of the discovered filters also play an essential role in the network function for a specific degradation ( see Sec . 5.2.2 ) . 3 ) Based on these discovered filters , we could easily predict the degradation of input images without training in the supervision of degradation labels ( see Sec . 5.3 ) . In a word , a unified one-branch network is similar to a well-designed two-branch network in the working mechanism . The degradation prediction branch and several small sub-networks for various degradations could be automatically learned within the one-branch network , even it is trained without any supervision about degradation distinctions . To analyze the mysterious blind SR networks and draw the above key findings , we propose a novel Filter Attribution method based on Integral Gradient ( FAIG ) , aiming at finding core filters in a network that make the greatest contribution to the function of a specific degradation removal . Different from previous integral gradient methods [ 37 , 36 ] that cumulate gradients along paths in the pixel/feature space , our FAIG utilizes paths in the parameter space , which has a clearer meaning in attributing network functional alterations to filter changes ( i.e. , the changes of filter parameters rather than inputs result in the network function alteration ) . Built on FAIG , we further develop a simple yet effective method to predict the degradation of an input image ( see Sec . 4.3 ) . To summarize , the contributions are three-fold . 1 ) Our findings provide a better understanding of the mechanism under blind SR networks , especially , bringing insightful connections between popular two-branch methods and unified one-branch networks . 2 ) The discovered discriminative filters for specific degradations allow us not only to perform degradation prediction , but also achieve a controllable adjustment of restoration strength without introducing extra parameters . 3 ) Exploiting the interpretability of blind SR would bring great significance for future works in i ) designing more efficient architectures ; ii ) diagnosing an SR network , such as determining the boundary of network restoration capacity and improving algorithm robustness . 2 Related Work . The super-resolution field has witnessed a variety of developments since the pioneer work of SRCNN [ 6 , 7 ] . Different architectures are proposed , such as deeper networks [ 17 , 19 , 15 ] , residual blocks [ 20 , 22 , 51 , 43 , 41 ] , recurrent architectures [ 18 , 38 ] , and attention mechanism [ 50 , 4 , 24 , 27 ] . Most of these methods are unified one-branch networks and achieve excellent performance on SR with a fixed downsampling ( e.g. , bicubic ) kernel . Recently , blind SR has attracted increasing attention to solve real-world restoration problems with complicated degradations . Classical blind SR methods typically incorporate human understanding of degradation process [ 8 , 23 ] , and are divided into two stages : degradation estimation ( especially blur kernels ) [ 28 ] and super-resolution based on the estimated degradation [ 11 , 45 ] . Deep-learning-based methods inherit this design and develop two-branch blind SR networks . KernelGAN [ 3 ] leverages the internal distribution of image patches to estimate blur kernel . DASR [ 39 ] learns degradation representation and prediction with contrastive learning . SRMD [ 48 ] and [ 49 , 46 ] incorporate these estimated degradations into CNN networks for restoration . IKC [ 14 ] and DAN [ 25 ] further propose to jointly perform degradation prediction and restoration in an iterative manner . Recent works also attempt to address blind SR within a one-branch networks [ 40 , 47 ] . There are few related works investigating the relationship between two-branch and one-branch networks for blind SR . Moreover , the underlying mechanism behind blind SR is still mysterious . Network interpretation methods for high-level visions can be divided into two categories : gradientbased methods [ 44 , 33 , 35 , 34 , 1 , 53 , 32 ] and perturbation-based approaches [ 9 , 52 , 30 , 31 ] . The latter usually localizes the discriminative image regions by performing perturbation to the input or neurons [ 10 ] . They heavily depend on lots of samplings and resources to guarantee performance . The most relevant gradient-based method to ours is Integrated Gradients ( IG ) [ 37 , 36 ] , which accumulates gradients at all points along a path from the baseline input to target input . IntInf [ 21 ] and neuron conductance [ 5 ] further extend feature-important IG to neurons . Unlike these methods that cumulate gradients along paths in the pixel/feature space , our FAIG utilizes paths in the parameter space for attributing network functional alterations to filter changes . Recently , Gu and Dong [ 13 ] introduce an interpretation method with local attribution maps to SR networks , while we further try to uncover the underlying mechanism in blind SR networks . Interpolation in the network parameter space has been used to explore the generalization and flat property of loss landscapes [ 12 , 16 ] . It is also employed in image manipulation for continuous imagery effect transition [ 42 ] . In this work , we utilize the paths in parameter space ( with interpolation ) to find discriminative filters for specific degradations . 3 Preliminary . Blind SR aims to restore high-resolution images from low-resolution ones with unknown and complex degradations , e.g. , blur , noise and downsampling . The classical degradation model [ 8 ] is usually adopted to synthetic the low-resolution input . Generally , the ground-truth image xgt is first convolved with Gaussian blur kernel k. Then , a downsampling operation with scale factor r is performed . Finally , the low-resolution x is obtained by adding white Gaussian noise n : x = ( xgt ⊛ k ) ↓r +n . ( 1 ) Recent blind SR methods including IKC [ 14 ] , DAN [ 25 ] and DASR [ 39 ] typically adopt a two-branch design , as shown in Fig . 1 . We then compare the two-branch design with commonly-used one-branch networks on two state-of-the-art methods : DAN and DASR . For a fair comparison , all experiments are conducted with their corresponding officially released codes , and the settings remain unchanged except the network structures . Specifically , SRResNet without batch normalization [ 20 ] is employed as the one-branch network . We adjust the number of residual blocks to match a similar computation budget ( FLOPs and parameters ) to two-branch methods . We compare them on the blur and blur+noise settings ( more specifications are in the supp . material ) . The PSNR comparisons are shown in Tab . 1 and Fig . 1 . It is observed that a unified one-branch network can also achieve comparable results under similar computation budgets on both settings . This interesting observation motivates us to investigate the underlying mechanism of blind SR networks , especially one-branch networks . It is worth noting that we do not claim that one-branch networks are always better than two-branch designs . The two-branch methods that incorporate domain-specific knowledge may have an advantage in training stability and robustness . In this paper , this observation is our primal motivation for interpreting blind SR networks . Integrated Gradient ( IG ) method [ 37 , 36 ] is usually used in high-level tasks ( e.g. , classification network ) . It attributes the most important input components ( e.g. , pixels in input images ) that affect the network predictions . Suppose we have a deep classification network F : Rd 7→ [ 0 , 1 ] . Let x ∈ Rd be the input image and x̄ ∈ Rd be the baseline input . x̄ is usually a black image or blurry images . The IG method accumulates the gradients at all points along a straight-line path in Rd from the baseline x̄ to the input x . Formally , the integrated gradient along the ith dimension for an input x and baseline x̄ is defined as : IGi ( x ) = ( xi − x̄i ) × ∫ 1 α=0 ∂F ( x̄+ α× ( x− x̄ ) ) ∂xi dα , ( 2 ) where xi is its ith pixel value and ∂F ( x ) ∂xi is the gradient of F along the ith dimension at x . Each change in the input component adds up to the final prediction difference ∑ i IGi ( x ) = F ( x ) − F ( x̄ ) . Intuitively , the function F varies from a less informative value ( produced by the baseline x̄ ) to its final informative value . The gradients of F with respect to the image pixels explain each step of the variation in the value of F . The integration over the path gradients cumulates these small attributions and accounts for the difference of network outputs . In other words , the IG could interpret deep networks by finding out the most important input components ( or features ) that explain the network predictions .

Summary: Most existing blind super-resolution methods use a two-branch network for degradation prediction and conditional restoration. In this paper, authors show that a one-branch network can also automatically learn to distinguish degradations for blind SR with the proposed novel filter attribution method based on integral gradient (FAIG). By using FAIG, they could find 1% filters as the most discriminative filters for each specific degradation in the one-branch blind SR networks. Experimental results show that these selected discriminative filters do have big influence on the corresponding degradation, while having a minor impact on the other degradation. This finding is promising since it may help to design more efficient architectures for blind SR for future work.