Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a structure-regularized filter pruning strategy for efficient SISR. Considering the residual connections in the SR networks, the authors propose to prune the filters that are aligned across layers connected by the same residual. The weight of those layers are optimized using an L-2 regularization to avoid performance drop. Experiments on both lightweight and large SR networks are conducted, with superior performance both quantitatively and qualitatively.
main_review: Strength:
1 the paper is well-written and easy to read. 
2 the proposed method is simple yet effective, where the experimental results on both lightweight and large SR networks demonstrate state-of-the-art results.

Weakness:
1 In general, this paper lacks novelty. Filter pruning, as well as the regularization term and schedule, are common techniques in high-level vision tasks, and this paper applies them to the SR task. The difference might be the pruning criterion. Indeed, the extensive residual connections cause the main difficulty to directly apply these techniques to the SR task. However, the authors do not include deep investigations to this problem, e.g., experiments of direct application, insightful analysis about the intrinsic reasons, etc. In addition, to solve this problem, the authors randomly select a set of filters to be pruned and fix them during pruning. This may affect the stability of training, where no repetitive experiments are provided.
2 In Table 3, the SRPN is pruned from an extended version of RCAN (with 96 channels) and achieves state-of-the-art performance. This is logical since the extended RCAN is more powerful and contains redundant parameters. It is better to, the performance of the extended RCAN should also be provided. 
3 The authors claim a general idea to prune SR networks, especially for large networks. However, only the pruning of RCAN (without channel attention) is validated. To demonstrate the generalization capacity, more networks with different topologies should be included. 

summary_of_the_review: This paper is clear and well-written. However, the novelty of the paper is somehow limited. Also, there are limited insightful investigations about the idea and intrinsic motivations. Finally, some experiments are missed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose structure-regularized pruning (SRP) for efficient image super-resolution. They provide a general idea to structurally prune both lightweight and large image SR networks. Extensive results are provided to support their claimed contributions.
main_review: Strengths:
First of all, I like this idea, which jointly optimizes the network pruning and image SR. The results are impressive and show promising potential for future efficient works.
1. Making effective SR models more efficient is of interest to a broad audience in low-level vision. This paper tackles this issue thus potentially can have a big impact.
2. In terms of methodology, they present a new network pruning method (SRP) for efficient SR based on regularization, which is technically sound. 
3. Empirically, they apply their method to both large SR networks and lightweight networks. In both cases, they show their pruned networks achieve the best performance with fewer parameters or Mult-Adds. The results look pretty strong.
4. The supplementary material contains pretty much information. Most of my concerns, like differences with other related works, can be addressed by the supplementary file. 
5. The writing and organization are good. The whole paper, including the supplementary, is well prepared.
6. The authors provide demo code to reproduce the results in the paper, which makes the paper more reliable and convincing. 

Weaknesses:
In general, this paper is well-written and has extensive experiments with strong results. I have the following questions about the method and result details.
1. About the method, applying L2 regularization for sparsity appears not a common practice since it cannot make weights exactly zero. L1 regularization (i.e., lasso) is more normal in statistics in the sense of imposing sparsity. The discussion in “Regularization Form” seems not enough to treat this question properly. The authors are highly suggested to explain more.
2. I noted in Tab. 1, the listed “pruning ratio” seems not aligned with the parameter reduction. E.g., for pruning ratio 0.5, the compression ratio should be 2. While by the parameters, the compression ratio is 1369.9/381.8=3.6, much larger than 2. Why this? 
Btw, typos: Appendix “Line 756-760 in the main paper” seems not correct.
summary_of_the_review: The idea is simple yet efficient for both lightweight and large image SR networks. The ablation study, like the visualization of pruning process, demonstrates the effect of the method. The main comparison results with others are impressive.

--------------------------------
The authors addressed my concerns in the reply. After considering other reviews and response, I decide to keep my initial score and vote for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Structure-Regularized Pruning (SRP) for the super-resolution task. Authors found that residual connections in SR models make it difficult to develop filter pruning methods. To resolve it, the proposed method imposes constraints on the locations of pruned filters. The constraint is to align unimportant indexes across different layers. The proposed method was achieved.

main_review: Strength

- The proposed method achieves better performance compared to the previous lightweight SR methods. Also, the proposed method is successfully applied to larger SR models.

- The proposed method is simple and effective.

Weakness

- The proposed method is suitable when there is a residual connection and there are few Conv layers per block. On the other hand, if there is no residual block or there are many Conv layers per block, the effect may be somewhat insignificant. There is a need to compare the proposed method with the cases of the pruned model without residual block and of the pruned model with many Conv layers in the block.

- I think the comparison with the existing Purning method is somewhat lacking. Is there any comparison with unstructured pruning other than channel pruning? Also, Is there any comparison with the pruning technique used in semantic segmentation other than SR?
--> CAP: Context-Aware Pruning for Semantic Segmentation, WACV 2021

- There is some ambiguity in the details of the proposed method. In the Pruning criterion, how many random unimportant filters are selected? In the Regularization schedule, what is the initial alpha? How pre-defined constant is decided? 

- Performance improvement in Table1 is not significant. Also, the proposed method is only evaluated on only one backbone SR model. It is difficult to say that the proposed method has been generalized.

- In Table 3, there are no reports about Params and Mult-Adds.

- In Table 4, there are needs more comparisons on different datasets

summary_of_the_review: However, there are some ambiguity explanations, and the model design and hyperparameter were arbitrarily selected.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Although light weight SR models show promise in terms of quality while using moderate sized network, it does not extend from there into larger models, or into practical use.
Model compression techniques have also been used to reduce network size but consumes significant resources and computation.
Neural architecture search and knowledge distillation are some example techniques to use for compression but not sufficiently effective.
However, network pruning can be a viable option for effective model compression, but can be proved tricky due to its difficulty regarding pruning of the filter for residual blocks.
To mitigate the issue, the authors propose a structure regularized pruning which enforces a regularization on the pruned structure to make pruned regions aligned across different layers. For instance, the method works by selecting filters with the same indices which are connected by the same residual. 
The authors also employ a L2 regularization to drive weights to zero for unimportant filters and transfer its information to other parts of the network.
This is important for minimizing performance degradation.
The authors propose a derived network via using the structure regularized pruning, namely SRPN-L and SRPN.
The proposed methods show better quality  than the latest networks quantitatively and visually.

main_review: Strengths
- The problem definition is well defined, addresses the challenges and provides a possible solution
- The paper is well written, easy to follow, extensive explanations make the paper readable
- The descriptions and explanations are to the point
- Terms and concepts are well defined and explained
- Proposes a pruning method for structured pruning
- Gives two derived networks SRPN-L and SRPN and reports their results which shows quality improvement compared to other SOTA
- The SRP model can be applied to SOTA networks in a plug and play model
- Extensive experimental analysis
- Ablation study shows that the proposed method is effective at pruning and maintaining the quality

Weakness / Questions
- Were there any effort for the SR models to be trained using real-world downsampling kernels instead of bicubic?
- Why are the parameter counts not provided for the large network comparison table?
- Likewise for Mult-adds

summary_of_the_review: This paper is well written, containing extensive experiments and impressive results.
review_title: null

Meta-review: This paper receives positive reviews. The authors provide additional results and justifications during the rebuttal phase. All reviewers find this paper interesting and the contributions are sufficient for this conference. The area chair agrees with the reviewers and recommends it be accepted for presentation.
Decision: Accept (Poster)