Title: Aligned Structured Sparsity Learning for Efficient Image Super-Resolution
Authors: Yulun Zhang
Huan Wang
Can Qin
Yun Fu
Venue: NeurIPS 2021
Keywords: Neural Network Pruning
Lightweight Image Super-Resolution
Aligned Structured Sparsity Learning

Abstract: Lightweight image super-resolution (SR) networks have obtained promising results with moderate model size. Many SR methods have focused on designing lightweight architectures, which neglect to further reduce the redundancy of network parameters. On the other hand, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable memory and computation resources. In contrast, network pruning is a cheap and effective model compression technique. However, it is hard to be applied to SR networks directly, because filter pruning for residual blocks is well-known tricky. To address the above issues, we propose aligned structured sparsity learning (ASSL), which introduces a weight normalization layer and applies $L_2$ regularization to the scale parameters for sparsity. To align the pruned locations across different layers, we propose a \emph{sparsity structure alignment} penalty term, which minimizes the norm of soft mask gram matrix. We apply aligned structured sparsity learning strategy to train efficient image SR network, named as ASSLN, with smaller model size and lower computation than state-of-the-art methods. We conduct extensive comparisons with lightweight SR networks. Our ASSLN achieves superior performance gains over recent methods quantitatively and visually.

TL;DR: Optimize image SR networks with network pruning simultaneously and achieve SOTA results

Review discussion:
The paper is well-written and easy to read.
The proposed ASSL properly addresses the problems (including lack of  BN layers and location mismatch) encountered in pruning SR networks. The details are well clarified and the experiments are adequate to demonstrate the effectiveness of ASSL.
No
Thanks for the valuable comments from Reviewer 3 (k8sL)
`Q3-1:` It seems that the SSA regularization term can only encourage m_i = m_j, and is not a strict constraint.
However, to my understand, only when m_i is exactly the same as m_j, the corresponding residual blocks can be pruned.
`A3-1:` Yes. The regularization term only encourages m_i = m_j instead of strictly guaranteeing it.
Yet, even if they are not exactly the same, when they are more similar, the induced damage is still much smaller than the case that m_i is very different from m_j.
Namely, the regularization still helps.
`Q3-2:` In Sec 4.4, it is better to add a comparison with EDSR + KD, since ASSLN is also developed over EDSR.
`A3-2:` As suggested, we add EDSR+KD results. We keep the final model structure the same between EDSR+KD and our ASSLN.
Specifically, in EDSR+KD, the student network has 48 channels in each residual block. The teacher network has 256 channels in each residual block. As seen, our method is consistently better than EDSR+KD on different test datasets.
Table 3-1: PSNR/SSIM comparisons on standard sets (scale = $\times$2). We apply our ASSL to EDSR baseline, resulting in ASSLN.
| Method | Set5 | Set14 | B100 | Urban100 | Manga109 |
| :-----: | :----: | :----: | :----: | :----: | :----: | 
| EDSR+KD      | 38.02/0.9605 | 33.56/0.9173 | 32.17/0.9001 | 32.11/0.9277 | 38.78/0.9774 |
| ASSLN (ours) | 38.12/0.9608 | 33.77/0.9194 | 32.27/0.9007 | 32.41/0.9309 | 39.12/0.9781 |
`Q3-3:` It is better to add comparisons on run-time.
`A3-3:` We add runtime results in Table 3-2, where the average running time is measured on Urban100 with scale $\times$2. The code and pretrained model are from the official websites. We run the testing on the same machine (3.60 GHz Core(TM) i7-6850K with 64 GB RAM and a Titan Xp GPU).
Table 3-2: Running time comparisons on Urban100 among lightweight SR networks (scale = $\times$2).
| Method | CARN | IMDN | ASSLN (ours) |
| :-----: | :----: | :----: | :----: |
| Time (ms)     | 74.3 | 51.8 | 51.0 |
[*1] Efficient processing of deep neural networks: A tutorial and survey, 2017, Proceedings of IEEE
[*2] Model compression and acceleration for deep neural networks: The principles, progress, and challenges, 2018, IEEE Signal Processing Magazine.
[*3] Recent advances in efficient computation of deep convolutional neural networks, 2018, Frontiers of Information Technology & Electronic Engineering
I have read the comments of other reviewers and the feedback of the authors. I understand the contribution of this paper on pruning filters of residual blocks.
But I agree that the novelty of the current submission is not highlighted enough, and the quantitative improvements are marginal (compared with LatticeNet and IMDN).
So I update my rates to 5.
1. Some motivations should be highlighted. This paper states that “it is hard to be applied to SR networks directly, because filter pruning for residual blocks is well-known tricky”. Some image SR methods (e.g., IPT) do not use residual blocks.
Therefore, it may not be the main reason that the network pruning is hard to be applied to SR.
2. The novelty of the proposed method should be highlighted.  The proposed method seems to combine the network pruning method and the image SR method. Actually, applying the network pruning into many tasks is difficult, not limited to the image SR task. The authors should highlight that such a combination is non-trivial.
3. Some technical details are not clear. In Line 146, why the global scheme cannot work for image SR networks? Why the main reason lies in the architecture difference between the image SR networks and classification networks.
However, some classification networks also use residual blocks.
4. The experiments can be improved. Why not compare with RCAN? Why not apply the proposed method to RCAN? Is the proposed method limited to the number of residual blocks? In Table 2, it would be better to report the results of EDSR.
In addition, it would be better to compare with another state-of-the-art image SR method (IPT).
No
Thanks for the valuable comments from Reviewer 2 (XysC).
`Q2-1:` Some motivations should be highlighted. This paper states that “it is hard to be applied to SR networks directly, because filter pruning for residual blocks is well-known tricky”. Some image SR methods (e.g., IPT) do not use residual blocks.
Therefore, it may not be the main reason that the network pruning is hard to be applied to SR.
`A2-1:` We understand transformer-based SR networks (e.g., IPT) do not use residual blocks.
However, they are not the mainstream SR methods currently. It is still very imperative to have an effective filter pruning method that can slim the current SOTA SR networks.
Besides, in terms of network architecture, transformer-based SR models are actually simpler than the SOTA SR networks due to the inexistence of residuals. Since our method can effectively prune SOTA SR networks, it can also be used to slim the simpler transformer-based SR models.
`Q2-2:` The novelty of the proposed method should be highlighted. The proposed method seems to combine the network pruning method and the image SR method. Actually, applying the network pruning into many tasks is difficult, not limited to the image SR task. The authors should highlight that such a combination is non-trivial.
`A2-2:` Our paper applies pruning to image SR, but it is beyond a trivial combination of pruning and SR. The reasons are (1) in terms of methodology, we do not apply an existing filter pruning method to SR networks;
instead, we propose a brand-new pruning algorithm specifically for SR networks tailored to its dense-residual-connection feature; (2) in terms of performance, our method achieves new SOTA complexity-performance tradeoff compared to many existing methods (Table 2, Table 4, Figure 5).
Thus, our paper joining filter pruning with image SR is non-trivial.
`Q2-3:` Some technical details are not clear. In Line 146, why the global scheme cannot work for image SR networks? Why the main reason lies in the architecture difference between the image SR networks and classification networks.
However, some classification networks also use residual blocks.
`A2-3:` In a global pruning scheme, a total sparsity ratio is given.
Then, how much sparsity assigned to each layer is decided by the pruning algorithm. E.g., in [41], L1 regularization is applied to the scale parameters in all the BN layers to learn sparsity for each layer. There is no guarantee that the two layers that will be added together are pruned by the same number of filters (L148-149). If the remaining filters of the two layers have different shapes, they cannot be added together -- This problem only occurs to networks with residuals. Unfortunately, residuals are intensively used in current SOTA SR networks. This is why “the global scheme cannot work for image SR networks”.
As analyzed above, the crux lies in the residuals. For classification networks (e.g., ResNet50), although they have residuals but not too many. We can skip pruning those layers connected with residuals but still can achieve descent acceleration (e.g., [37, 9, 58]). For SR networks, they use too many residuals. If we skip them, the best achievable acceleration will be marginal. This is why “the main reason lies in the architecture difference (i.e., number of residuals) between the image SR networks and classification networks” -- See L189-L206 for detailed analysis.
We will add more explanation to clear the possible confusion.
`Q2-4:` The experiments can be improved. Why not compare with RCAN? Why not apply the proposed method to RCAN? Is the proposed method limited to the number of residual blocks? In Table 2, it would be better to report the results of EDSR.
In addition, it would be better to compare with another state-of-the-art image SR method (IPT).
`A2-4:` We apply ASSL to each residual channel attention block in RCAN and denote this version as RCAN+ASSL. The comparison results with RCAN are shown in the following Table 2-1, where we provide PSNR/SSIM values with scale $\times$2.
Table 2-1: PSNR/SSIM comparisons on standard sets (scale = $\times$2). We apply our ASSL to RCAN.
| Method | Set5 | Set14 | B100 | Urban100 | Manga109 |
| :-----: | :----: | :----: | :----: | :----: | :----: | 
| RCAN | 38.27/0.9614 | 34.12/0.9216 | 32.41/0.9027 | 33.34/0.9384 | 39.44/0.9786 |
| RCAN+ASSL | 38.36/0.9620 | 34.30/0.9233 | 32.48/0.9033 | 33.53/0.9404 | 39.80/0.9799 |
The proposed method is not limited to the number of residual blocks.
During this short rebuttal, we cannot finish the code to apply our method to it.
Yet as explained above, in terms of architecture, IPT is actually simpler than EDSR and RCAN.
Thus, as far as we can see, it is entirely possible to apply our method to it (though we do not have solid numbers to support this claim). We will add this experiment later and release the code.
- Originality
The Novelty of this article is limited. The filter pruning method is widely used in previous lightweight networks. The idea of “the sparsity or redundancy in the Conv kernels” is out of date [1,2,3].
- Quality
The NAS-related method is time-consuming, and this is widely recognized. In lines 43-44, the authors show that some SR methods are time-consuming and need heavy computation resources through NAS[7].
This is inappropriate and unfair for most SR methods. The NAS-related method is not the mainstream of SR.
In the section of Sparsity Structure Alignment, the similarity-related method is a common idea in other methods, e.g. image transfer, KD, transfer learning. The authors should show their improvement.
The ablation analysis is not enough. The author needs to do the Basic module ablation analysis.
The authors should compare their method with SOTA lightweight SR method [4,5,6] in Table 2. They should also discuss the Lightweight SR method with the normal model，and show the gap between the lightweight method and other methods.
References:
[1] Exploiting kernel sparsity and entropy for interpretable CNN compression, CVPR 2019
[2] Sparse convolutional neural networks, CVPR 2015
[3] An information theoretic approach of designing sparse kernel adaptive filters, IEEE Transactions on Neural Networks 2009
[4] Madnet: A fast and lightweight network for single-image super resolution, IEEE Transactions on Cybernetics 2020
[5] LatticeNet: Towards Lightweight Image Super-Resolution with Lattice Block, ECCV 2020
[6] s-LWSR: Super Lightweight Super-Resolution Network,  IEEE Transactions on Image Processing 2020
No
Thanks for the valuable comments from Reviewer 1 (1kmf).
`Q1-1:` The Novelty of this article is limited. The filter pruning method is widely used in previous lightweight networks. The idea of “the sparsity or redundancy in the Conv kernels” is out of date [1,2,3].
`A1-1:` Admittedly (L48-52, L92-L105), filter pruning has been well-explored, yet mainly in the task of classification and regression. All the three references R1 suggested are either for classification or regression, not for image SR.
However, our paper is focusing on image SR. To our best knowledge, filter pruning on SR networks has not been well-explored to date (L106-L107). We propose a brand new filter pruning method specifically for SR networks, not applying an existing filter pruning method in classification to SR.
Thus, we believe our paper is novel, especially when the proposed method achieves the SOTA complexity-performance tradeoff (Table 2, Table 4, Figure 5).
`Q1-2:` The NAS-related method is time-consuming, and this is widely recognized. In lines 43-44, the authors show that some SR methods are time-consuming and need heavy computation resources through NAS[7]. This is inappropriate and unfair for most SR methods. The NAS-related method is not the mainstream of SR.
`A1-2:` We will use “NAS-based SR models” to make the statement more precise.
`Q1-3:` In the section of Sparsity Structure Alignment, the similarity-related method is a common idea in other methods, e.g. image transfer, KD, transfer learning. The authors should show their improvement.
`A1-3:` To our best understanding, by “the similarity-based method” (we did not use this term or any similar ones in our paper), R1 means the idea of regularizing the gram matrix of scale matrix (Figure 3). We understand the gram matrix, as a generic math concept, is broadly used in many areas (e.g., as pointed out by R1, it is used to describe image style in neural style transfer).
However, the relevance merely holds in the very top level (i.e., using the same math tool).
Besides this, these tasks are actually so different from image SR that they are basically incomparable. R1 does not provide specific related references overlapping with our idea but a few very broad research area names (“image transfer, KD, transfer learning”).
Thus, we do not know which paper/method should be compared to or how to “show the improvement”. R1 may suggest specific methods or papers you would like us to compare.
`Q1-4:` The ablation analysis is not enough. The author needs to do the Basic module ablation analysis. The authors should compare their method with SOTA lightweight SR method [4,5,6] in Table 1-1. They should also discuss the Lightweight SR method with the normal model，and show the gap between the lightweight method and other methods.
`A1-4:` We provide comparisons with MADNet-$L_{1}$ [4] (TCYB 2020), LatticeNet [5] (ECCV 2020),  s-LWSR$_{64}$ [6] (TIP 2020) in Table 1-1. We provide comparisons about parameter number, mult-adds, PSNR/SSIM results on Set5, Set4, B100, and Urban100 with scale $\times$4. All the results of compared methods are cited from their original papers.
According to the comparisons in Table 1-1, we can see our ASSLN has the smallest parameter number and mult-adds.
However, our ASSLN achieves the best results in most cases.
Table 1-1: Parameter, mult-adds, PSNR/SSIM results on standard sets (scale = $\times$4). We highlight the best results.
| Method | Params| Mult-Adds| Set5 | Set14 | B100 | Urban100
| :-----: | :----: | :----: | :----: | :----: | :----: | :----: | 
| MADNet-$L_1$ [4] | 1002K | 54.1G | 31.95/0.8917 | 28.44/0.7780 | 27.47/0.7327 | 25.76/0.7746 |
| LatticeNet [5] | 777K | 43.6G | **32.30**/0.8962 | 28.68/0.7830 | 27.62/0.7367 | 26.25/0.7873 |
| s-LWSR$_{64}$ [6] | 2277K | 131.1G | 32.28/0.8960 | 28.34/0.7800 | 27.61/0.7380 | 26.19/**0.7910** |
| ASSLN (ours) | 708K | 40.6G | 32.29/**0.8964** | **28.69**/**0.7844** | **27.66**/**0.7384** | **26.27**/0.7907 |
References:
[1] Exploiting kernel sparsity and entropy for interpretable CNN compression, CVPR 2019
[2] Sparse convolutional neural networks, CVPR 2015
[3] An information theoretic approach of designing sparse kernel adaptive filters, IEEE Transactions on Neural Networks 2009
[4] Madnet: A fast and lightweight network for single-image super resolution, IEEE Transactions on Cybernetics 2020
[5] LatticeNet: Towards Lightweight Image Super-Resolution with Lattice Block, ECCV 2020
[6] s-LWSR: Super Lightweight Super-Resolution Network, IEEE Transactions on Image Processing 2020

Meta-review:
role: PC
data: comment: This paper studies the issue of pruning for super-resolution networks.  The reviewers agree that the approach to pruning architecture-specific components, like resnet blocks, offers some interesting advances.  However the authors seem to agree that the paper could do a better job communicating the novelty of the proposed methods, and that experiments show fairly small improvements over existing methods for pruning networks of this type.



Decision: Accept (Spotlight)