Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Existing methods for blind super resolution have recently adopted a design consisting of two branches where one branch is responsible for predicting (or representing) the degradation and the other branch is responsible for reconstruction while conditioning on the degradation information of the first branch. This paper highlights that a single network with one branch can achieve a similar performance as the two-branch approaches and analyses this in detail. More specifically, the authors design a filter attribution method to find most discriminative filters as opposed to most discriminative input pixels or features. This analysis also allows the development of a simple predictor for degradation.
main_review: While early works in upscaling have focussed on fixed degradations, more recent upscaling work that considers the blind setting has helped making super resolution more practical and flexible. However, to my knowledge it has not yet been explored to what extent the solutions deployed in the blind methods are strictly required and in which way a single network can achieve similar performance. As such I find this paper to provide an interesting and fairly unique contribution and I certainly enjoyed reading the paper.

Through the analysis of filter attribution, the method highlights and dissects how certain filters of an otherwise 'black box' network exhibit a given functionality. It is an interesting insight, that even without any form of supervision about degradation distinctions, it can be observed that small sub-networks emerge that are responsible for different degradations.

The paper clearly lists limitations, which is first and foremost that it considers a toy example in blind SR. It only allows a single discrete level for each degradation and the degradations (blur and noise) do not exist at the same time in a given image. 

Nevertheless, I think this is an interesting direction to gain more understanding on and in follow up work, it would be interesting to analyse the behaviour for a wider range of continuous degradation settings. In terms of related work, in the space of blind super resolution [CORNILLÈRE et al., Blind Image Super-Resolution with Spatially Variant Degradations, Siggraph Asia 2019] could be cited.
limitations_and_societal_impact: Limitations are clearly described and I don't expect negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a Filter Attribution method based on Integral Gradient (FAIG) to analyze the behaviors of blind super-resolution networks on different degradations. It experimentally proves that a one-branch blind SR network is similar to a well-designed two-branch one in the working mechanism. The degradation prediction branch could be automatically learned in a unified one-branch network. These findings are interesting and can help us better understand the behaviors of blind SR networks.
main_review: The paper is well-written and easy to read. 

The finding that a one-branch blind SR network can achieve comparable performance with a well-designed two-branch one is exciting. The proposed FAIG is also interesting, and the adequate experiments also demonstrate the effectiveness of FAIG in discovering discriminative filers.

However, the reason why the one-branch network can be comparable with the two-branch one is still not well explained.  With the help of FAIG, we can know which filters are more important to the specific degradation, but still do not know why they do. It is better to give a further explanation of how the discovered filters work in removing the corresponding degradations.
limitations_and_societal_impact: 1. The overlap score OS(x, D)  can help in measuring the similarity of the degradation of x and D. However, the degradation space seems to be continuous and there should be an infinite number of different degradations. How do you predict the degradation by calculating OS(x, D)? I mean that there is an infinite number of D, and you can not calculate all of them to choose the largest one. 

2. Is it possible to use your findings to help design a better blind SR network? It will be great if you can provide some ideas.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Most existing blind super-resolution methods use a two-branch network for degradation prediction and conditional restoration. In this paper, authors show that a one-branch network can also automatically learn to distinguish degradations for blind SR with the proposed novel filter attribution method based on integral gradient (FAIG). By using FAIG, they could find 1% filters as the most discriminative filters for each specific degradation in the one-branch blind SR networks. Experimental results show that these selected discriminative filters do have big influence on the corresponding degradation, while having a minor impact on the other degradation. This finding is promising since it may help to design more efficient architectures for blind SR for future work.

  



main_review: However, there are still several issues to be addressed.
1. Authors claim that the proposed FAIG method is able to find a small number of filters as the most discriminative filters for degradation removal in blind SR networks. However, only one setting is validated, i.e., the bicubic downsampling with a scale factor of 2, Gaussian blur with a sigma of 2, and Gaussian noise with a level of 0.1. What about the experiments on other settings, like changing the scale factor, the sigma, or the noise level? By only using this same setting for all experiments may not be able to prove that the FAIG would work for various degradations. 

2. One important advantage about using the two-branch network for SR is that they provide an opportunity for training a network which can generalize well on images with multiple degradations, even though these degradations are unseen during training. For example, the two listed two-branch methods DAN[27] and DASR[41] directly apply their pretrained networks on some low-resolution images from real world to show their ability for generalization.
How would this finding be helpful for improving the ability for generalization of the one-branch network, especially on those LR images from real world which may be difficult to separate the blurry part and noisy part from the images?   
 
3. In Figure 3 (4), when masking a smaller number of the denoising filters (<20%) for blurry input, it seems that the random is better than the proposed FAIG, is there any particular reason for that?  

4. It is not clear how to determine how much proportion of filters should be selected in advance and obtain close performance as the upper bound.

limitations_and_societal_impact: Yes, they have addressed the limitation and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper aims to analyze blind SR models and gain a deeper understanding on how a one-branch CNN functions with different degradations. After empirically showing that a one-branch CNN performs on par with two-branch CNNs that are commonly employed in blind SR, the authors propose a filter attribution method based on integral gradients and make important key observations that a small number of discriminative filters ($\sim$1%) can be found for different degradations in one-branch models. 
main_review: Pros:
* This paper presents a simple idea for interpreting filters in deep networks that may otherwise be black-box modules. 
* Papers focusing on network interpretability are often geared towards high-level vision tasks such as image classification. However, this paper handles blind SR, and with strong empirical evidence that few discriminative filters exist for different degradations (blur and noise), the readers can gain a deeper understanding into how the internal filters are working for image restoration CNNs. I believe the presented analyses will be a great contribution to the low-level vision community.
* The visual results are very impressive and it can be directly observed that discriminative filters exist for different degradations.
* Quantitative comparisons to other analytical methods by masking and re-training 1% filters show that the proposed method based on filter attribution can find more discriminative filters.
* The authors present abundant results (visual results, comparisons, ablation studies, analyses) in the main paper and the supplementary material.
* The paper is clearly written and provides many insights on blind SR models.
* The authors discuss reasonable limitations of their work.

Suggestions/questions:
* For one-branch backbones, network depth (number of residual blocks) is adjusted to match the computational complexity of two-branch networks. Therefore, in Table 1, one-branch models may perform better due to the increased depth. How about increasing the network width / the number of output channels (e.g. 64 channels to 256 channels) to match complexity? Do the authors think results would be similar in this case? It would be more convincing to show results of one-branch models with increased network width as well.
* Urban100 and Manga109 are also widely used testsets in SR. Adding results on these two datasets would make the analysis more complete.
* In Table 3, more discriminative filters can be found for deblurring than denoising with the proposed method compared to the next best method. Any intuitions why?
* The original DAN model is trained iteratively. Is the one-branch DAN also trained iteratively?
* Minor: Fig. 2 could be mentioned in Section 5.2.1 to support its claims.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work in Section 5.5. I agree with the authors that societal impact is N/A as their work focuses on interpreting blind SR models.
review_title: null

Meta-review: All reviewers are enthusiastic about the paper's findings and gave it high scores. The use of one-branch instead of two for blind super resolution is interesting, and one reviewer was particularly interested in the analytical aspect of the paper. 
Decision: Accept (Spotlight)