Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a pruning method for training efficient super-resolution (SR) models. It is motivated by the fact that the pruned indices of SR networks should be aligned (due to residual connections) and the batch normalization layers are not commonly employed in top-performing SR models. They propose to apply L2 regularization to introduce weight normalization layers to align the sparsity structure. The proposed method is applied to top-performing SR networks, rendering a new efficient SR model (ASSLN), which is empirically shown to be more favorable than the other compared SR models.
main_review: Pros:

They propose a new pruning method to train efficient networks specifically for SISR. According to the paper, this is “the first attempt to leverage filter pruning for efficient image SR”.

The ideas of regularizing the gram matrices of masks and using the Sigmoid function to obtain soft masks are novel.

Good experimental results demonstrate the effectiveness of the proposed ASSL method.

Cons:

It has been shown in [51] that weight normalization can improve network performance. It is not sure in the paper that the performance superiority of ASSLN is because of the introduced weight normalization layer or the proposed ASSL method. An ablation study is needed to clarify this issue.

The proposed method has several hyper-parameters, such as the $\Delta$ and $T$ (Line 174). Is the proposed ASSL method robust to them? Hyper-parameter sensitivity analysis is needed.

I am a little bit confused with Eq. (6): What is the dimension of $\gamma_{th}$? Is it a vector or scalar?

In all, I like such an idea about network pruning and image SR. The score would be upgraded, if my concerns are well addressed in the rebuttal.

limitations_and_societal_impact: I have checked the limitations and societal impact of their paper. They are properly discussed. I do not have any more concerns in this regard.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes an Aligned Structured Sparsity Learning (ASSL) strategy for SISR. ASSL adds a weight normalization (WN) layer to a SR network and then enforces L2 regularization on the scale parameters in a WN layer. The proposed regularization term can align the final sparsity pattern across different constrained conv layers and deliver considerable speedup without seriously compromising the performance. Experiments show the network trained with ASSL outperforms the other SR networks in terms of performance-cost tradeoff.
main_review: The overall idea is appealing and the results are promising.

Strengths: (1) The proposed method applies regularization to the scale parameters in the weight normalization (WN) layers. This idea looks novel and logically straight considering that batch normalization (BN) layers are not typically used in SR networks. (2) Regularizing the gram matrix of masks (approximated by sigmoid) to impose sparsity structure is also good. (3) Besides, the empirical performance is strong in either PSNR or visual quality.

Weaknesses: (1) The real acceleration is usually not well-aligned with FLOPs reduction. The authors are encouraged to also report the runtime speedup of the ASSLN. (2) Currently, the paper only uses the same pruning ratio for different layers. This probably is sub-optimal. Is there any idea to do non-uniform pruning? How to decide the proper pruning ratios?

To further improve this submission, I hope the authors could provide sufficient explanations and address my questions.
limitations_and_societal_impact: The limitations and potential negative societal impact of their work have been discussed in their paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a filter pruning method for efficient image super-resolution (SR), named Aligned Structured Sparsity Learning (ASSL). Specifically, they first introduce the Weight Normalization layers into a SR network. Then they propose a regularization term (regularizing the Gram matrix of soft masks of different layers) to penalize the scale parameter in the Weight Normalization layers. The regularization term encourages the consequent sparsity of different layers to present the same structure. Moreover, to maintain expressive power during pruning, they employ a rising L2 regularization scheme to gradually drive the unimportant weights to zero. Experiments show the superiority of the compressed models over those by comparison SR methods.
main_review: Pros:
+ This work presents a new filter pruning method (ASSL) based on regularizing the scale parameters in Weight Normalization layers to learn lightweight image SR models.
+ Based on the new method, the authors manage to train a lightweight SR model ASSLN.
+ Strong experimental results -- both quantitative and qualitative results show that the compressed models by the proposed method are obviously better (higher PSNR and sharper details) than the other comparison models.

Cons:
- It would be better to see more recent filter pruning methods compared other than L1-norm pruning [37] in the experiments, such as HRank [1].
- Missing related works: [2, 3] also focus on efficient SR via network pruning. Please discuss how your work is different from theirs. 
- The authors mentioned that existing methods for lightweight SR models use heavy computation resources for training. To demonstrate that the proposed ASSL method is more resource-friendly, the authors are supposed to provide more training details, such as the wall-clock training time and specific GPU types.
- Based on the paper, the proposed ASSL method requires a *pretrained* SR model as base model to prune. Although this is very typical in the pruning papers, I am wondering if the method can learn a lightweight SR model *from scratch*.

[1] HRank: Filter Pruning Using High-Rank Feature Map, CVPR, 2020.

[2] DHP: Differentiable Meta Pruning via HyperNetworks, ECCV, 2020.

[3] Efficient Image Super Resolution Via Channel Discriminative Deep Neural Network Pruning, ICASSP, 2020.

------------------------------------------------------------------------------------------------------------------
The author has addressed all my concerns in the rebuttal, and I decide to keep my initial rating and vote for acceptance.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The paper presents an exciting new approach for image super resolution. All the reviewers are positive about the paper.  Please update the draft with reviewers' suggestions. Congratulations for a nice result!
Decision: Accept (Spotlight)