Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning
Authors: Yulun Zhang
Huan Wang
Can Qin
Yun Fu
Venue: ICLR 2022 Poster (2022)
Keywords: image super-resolution

Abstract: Several image super-resolution (SR) networks have been proposed of late for efficient SR, achieving promising results. However, they are still not lightweight enough and neglect to be extended to larger networks. At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources. In contrast, network pruning is a cheap and effective model compression technique. However, it is hard to be applied to SR networks directly because filter pruning for residual blocks is well-known tricky. To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned filters are aligned across different layers. Specifically, for the layers connected by the same residual, we select the filters of the same indices as unimportant filters. To transfer the expressive power in the unimportant filters to the rest of the network, we employ $L_2$ regularization to drive the weights towards zero so that eventually, their absence will cause minimal performance degradation. We apply SRP to train efficient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN. We conduct extensive comparisons with both lightweight and larger networks. SRPN-Lite and SRPN perform favorably against other recent efficient SR approaches quantitatively and visually.

Published as a conference paper at ICLR 2022
LEARNING EFFICIENT IMAGE SUPER-RESOLUTION NETWORKS VIA STRUCTURE-REGGLARIZED PRUNING
Yulum Zhang!:
Huan Wang
Can Qin'
Yun Fu'
Northeastern University, USA
ABSTRACT
Several image super-resolution (SR) networks have been proposed for efficient SR, achieving promising results. However, they are still not lightweight enough and neglect to be extended to larger networks. At the same time, model compres- sion methods (i.e., knowledge distillation and neural architecture search) usually need heavy extra resources. Then, we turn to cheap and effective model com- pression way: network pruning. However, it is not easy to directly apply network pruning to image SR. It is well-known tricky to conduct filter pruning for resid- ual blocks commonly used in SR networks. To solve these problems, we propose structure-regularizze pruning (SRP), which imposes regularization structure to ensure the locations of pruned filters are aligned across different lay- ers. Specifically, for the layers connected by the same residual, we select the filters of the same indices as unimportant filters. To transfer the expressive power in the unimportant filters to the rest of the network, we employ L2 regularization to drive the weights towards zero so that eventually, their absence will cause mini- mal performance degradation. We apply SRP to train efficient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN. We pro- vide extensive comparisons with both lightweight and larger image SR networks. Our SRPN-Lite and SRPN perform favorably against other recent works.
the pruned
INTRODUCTION
Given low-resolution (LR) image, single image super-resolution (SR) aims to reconstruct a high- resolution (HR) output. Essentially, as a many-to-one mapping problemimage SR is ill-posed. To tackle this problem. plenty of deep convolutional neural networks (CNNs) (Dong al., 2014; 2016; Kim et al., 2016b; Zhang et al., 2018c; 2020; 2021) have been investigated to learn the accurate mapping from LR input to the corresponding HR target.
Deep CNN for image SR is first investigated in SRCNN (Dong et al., 2014) and has continuously shown promising SR performance. SRCNN consists of there convolutional (Conv) layers. constrain- ing its expressivity. Kim et al. adopted residual learning to increase network depth in VDSR (Kim et al., 201 and obtained notable improvements over SRCNN. Lim et al. (Lim et al., 2017) sim- plified residual blocks and built a much deeper network EDSR. Zhang et al. (Zhang et al., 2018b) proposed a even much deeper one RCAN with the residual in residual structure. Empowered by increased network size, deep SR models like EDSR (Lim et al., 2017) and RCAN (Zhang et al., 2018b) have seen remarkable SR performance. However, as a cost, the large model size brings about problems such excessive memory footprint, slow inference speed. It is thereby impractical to deploy them on resource-constraines platforms directly (Lee et al., 2020).
Aiming for efficient SR, more and more works introduce lightweight network architectures (Ahn et al., 2018; Luo et al., 2020). Ahn er al. proposed cascading residual network (CARN) (Ahn et al., 2018). Hui et al. proposed information multi-distillation network (IMDN) (Hui et al., 2019). Lee et al. introduced knowledge distillation (Hinton et al., 2014) for image SR (Lee et al., 2020). Besides, neural architecture search (NAS) (Zoph & Le, 2017; Elsken et al., 2019) was also utilized for lightweight SR model (Chu et al., 2019a). However, there are still several downsides to these networks: (1) There still an obvious performance gap between those lightweight models and the very deep models; (2) These methods can consume considerable computation resources for training. For example, (Chu et al., 2019a) train a single network with 8 Tesla V100 GPUs, and the training
"Euual contribution. The main work done when Yulun Zhang was at Northeastern University. Cooresponning author; Huan Wang avang.huan @ @northeastem.eduu

---

Published as a conference paper at ICLR 2022
IIR (img.099 002)
SRCNN
LapSRN
MemiNet
CARN
IMDN
SRPN-Lite (ours)
HR (ing U63)
EDSR
SRFBN
SAN
CSNLN
RFANet
SRPN (ours)
Figure I: Visual comparisons (x4) about lightweight and large SR networks on Urban 100 dataset. The first row is lightweight SR networks comparison. The second row is about larger models com- parison. SRPN-Lite and SRPN are our proposed lightweight and larger image SR networks.
takes about three days; (3) It is hard for most lightweight SR network designs to generalize to more large-scale networks and achieve superior performance at the same time. So, it is needed to explore more resource-friendly, effective, and generic lightweight SR networks.
On the other hand, neural network pruning is well-known as an effective technique to reduce model complexity (Reed, 1993; Sze al., 2017). For acceleration, filter pruning (a.k.a. structured prun- ing) (Li et al., 2017) attracts more attention than weight-element pruning (a.k.a unstructured prun- ing) (Han et al., 2015; 2016b). Introducing filter pruning into image SR is a promising way to achieve good trade-off between performance and complexity. However, it is not easy to apply fil- ter pruning techniques image SR networks directly. This is mainly because residual connections are well-known difficult to prune in structured pruning (Li et al., 2017). On the other hand, they are extensively used in state-of-the-arr (SOTA) image SR methods ge EDSR (Lim et al.,,, 2017) has 32 residual blocks; RCAN (Zhang al., 2018b) even has nested residual blocks).
To tackle the above issue, we propose Structure StructureReeularized Regularized Pruning (SRP), which imposes regu- larization on the pruned structure to ensure the locations of pruned filters are aligned across different layers. Specifically, for the layers connected by the same residual, we select the filters of the same indices as unimportant filters (i.e., those will remove finally). To transfer the expressive power in the unimportant filters to the remainder of the network, we employ L2 regularization to drive the that eventually, their absence will incur negligible performance
weights towards zero gradually loss. To the best of our knowledge, our SRP is one of the leading works (Zhang et al., 2021) to leverage structured pruning for efficient image SR. Our main contributions are as follows:
We propose network structure-regulariied pruning (SRP) method to learn efficient image SR networks. We try to jointly train image SR models with network pruning simultaneously to achieve high reconstruction performance as well as efficiency.
Our SRP provides general idea structurally prune networks, which consists of extensive residual connections. The introduction of regularization as a pruning tool manages to main- tain the expressivity of the original network while peeling off the unnecessary redundancy.
We employ SRP to train efficient image SR networks, resulting in lightweight network (named SRPN-Lite) and a very deep one (named SRPN). We achieve superior performance gains both lightweight and large image SR networks.
2 RELATED WORK
Deep Image SR Models. Dong el al. (Dong et al., 2014) firstly proposed SRCNN for image SR and achieved superior performance with only convolutional (Conv) layers. Residual learning was introduced in VDSR (Kim et al., 201 reaching 20 Conv layers and a significant improvement over SRCNN. Tai et al. later introduced memory block MemNet (Tai al., 2017b) for deeper network structure. Lim et al. (Lim et al., 2017) simplified the residual block (He et al., 2016) and constructed deeper and wider networks with large number of parameters. Zhang et al. (Zhang et al., 2018b) proposed an even deeper one, residual channel attention network (RCAN), where the attention mechanism was firstly introduced in image SR. Liu et al. proposed FRANet (Liu et al., 2020) to make the residual features focus critical spatial contents. Later, Zhang et al. (Zhang et al., 2019) extended to image restoration with residual non-local attention. Mei et al. proposed CSNLN (Mei et al., 2020) by combining local and non-local feature correlations. Most of those methods have achieved SOTA results. However, they suffer from huge model size (i.c., network parameter number) and/or heavy computation operations (i.e., FL.OPs.
2

---

Published as a conference paper at ICLR 2022
1+1
(b)
Figure 2: (a) Illustration of channel-wise and filter-wise pruning for single Conv layer. In this work, we adopt filter-wise pruning to learn efficient image SR networks. (b) Illustration of filter pruning within a residual block. We depict deep features F as 3d cubes. We expend the Conv kernel W (4d tensor) as 2d matrix here for easy illustration (each row represents a filter). Both green and blue color denote the pruned filters (or feature map channels): green denotes the pruned filters in free Conv layers; blue denotes the pruned filters in constrained Conv layers. The basic idea of our SRP is to apply the L2 regularization to the unimportant filters to make sure the pruned indices of F(i) are exactly the same as those of F((++2) (c) Illustration of filter pruning across different residual blocks. All the layers that are directly followed by the add operators are constrained Conv layers (in blue). Others are free Conv layers (in green).
Lightweight Image SR Models. Recent years have been rising interest in investigating lightweight image SR models. These approaches try to design lightweight architectures, which mainly take advantage of recursive learning and channel splitting. Kim er al. firstly decreased parameter num- ber by utilizing recursive learning in DRCN (Kim et al., 2016b). Ahn et al. proposed CARN by designing cascading mechanism upon residual network (Ahn et al., 2018). Hui et al. proposed IMDN by using distillation and fusion modules (Hui et al., 2019). Luo et al. designed the lattice block with butterfly structures (Luo et al., 2020). Recently, neural architecture search for image SR, like FALSR (Chu et al., 2019a). Also, model compression techniques have been ex- plored for image SR. He ef al. proposed knowledge distillation based feature-affinin for efficient image SR (He et al., 2020). Lee el al. distilled knowledge from a larger teacher network to a student one (Lee et al., 2020). Those lightweight image SR models have obtained great progress, but we still need to investigate deeper for more efficient image SR models.
applied
Neural Network Pruning. Network pruning aims to eliminate redundant parameters in neu- ral network without compromising its performance seriously (Reed, 1993; Sze et al., 2017). The methodology of pruning mainly falls into two groups: filter pruning (or more generally known as structured pruning) and weight-element pruning (also referred to as unstructured pruning). The former aims to remove weights by filters (i.e., 4-d tensors), while the latter removes weights by sin- gle elements (i.e., scalars). Structured pruning results in regular sparsity after pruning. It does not demand any special hardware features to achieve considerable practical acceleration. In contrast, unstructured pruning leads to irregular sparsity. Leveraging the irregular sparsity for acceleration typically demands special software supports, while past works have shown the practical speedup is very limited (Wen et al., 2016), unless using customized hardware platforms (Han et al., 201 20) In this paper, we tackle filter pruning instead of weight-element pruning for effortless acceleration. The major efforts in pruning (mainly in image classification) have been focusing on proposing more sound pruning criterion to select unimportant weights (Reed, 1993; Sze et al., 2017). Criteria based on weight magniudee (Han al., 2015; 2016b; Li al., 2017) are the most prevailing ones, which we will also employ to develop our method in this paper.
METHODOLOGY
We first show the overview of the problem setting about deep CNN for image SR. It is also observed that excessive redundancy exists in the SR deep CNNs. Then we move on to proposing our structure- regularized pruning (SRP) method attempting to achieve more efficient SR networks.
3.1 DEEP CNN FOR IMAGE SR
In image super-resolution (SR), high-resolution (a.k.a. super-resolved) image ISR is reconstructed from its low-resolution (LR) input Lt.. This image SR process can be described
Isn = (T. +),
FFlterr pruning is sub-coneept within structured pruning, while they are often used interchangeably in the literature. In this paper, we stick to this terminology practice.
3

---

Published as a conference paper at ICLR 2022
where F-s(.)) is the image SR network, parameterized by O. Meanwhile, the LR input ILR from the corresponding HR image can be formulated as
LRR = F1, (4HR):
(2)
where Fi, (.) downscales the original HR image IH by a scale factor 8. This downscaling pro- cess typically introduces additional compression, blurring, noise, and/or other unknown artifacts. Therein. the desired structural details could be removed more less. The job of image SR methods is to reconstruct the high-freeqenn, information as much as possible.
We aim to present structural sparsity inducing SR network learning method, which can deliver more efficient SR networks with fewer parameters and FLOPs.
3.2 STRUCTURE-REGULLRIZZZ PRUNING (SRP)
Pruned Index Constraint. Pruning filters in residual networks is well-known non-trivial as the Add operators in residual blocks require the pruned filter indices across different residual blocks must be aligned. A figurative illustration of filter pruning within a residual block is shown in Fig. 2(b). A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) consists of two convolutional layers. According to the mutual connection relationship, the convolutional layers can be categorized into two groups. One group is made up with the layers that can be pruned without any constraint, dubbed free Conv layers in this work; the other comprises Conv layers whose filters must be pruned at the same indices, dubbed constrained Conv layers. Concretely, the layer W in Fig. 2(b) is a free Conv layer, while the layer ((+1) is a constrained one.
Owing to the pruned index constraint problem, many filter pruning methods in image classification simply do not prune the last Cony layer in residual blocks (Li al., 2017: Wang ct al., 2021), which can still deliver considerable speedup of practical interest. Nevertheless, this doing- solution can barely translate to the image SR networks if we target a considerable speedup. The root cause is that image SR networks usually utilize many more residual blocks, and each block usually has only two Conv layers. In some oop-erforming SR networks (e.g., RCAN Zhang et al. (2018b)), there are even nested residual blocks. Taking EDSR for concrete example, it has 32 residual blocks with two Conv layers within each block. If the 2nd Conv layer in a residual block is spared from pruning, half of the Conv layers will not be pruned. In other words, at best, we can only obtain 2x theoretical acceleration measured by FLOPs reduction.
Given this issue, it imperative to prune all the Conv layers in residual blocks, thus calling for an approach to align the pruned indices of all constrained Conv layers. We then use regularization as promising solution considering that it has been widely used before to impose priors on the sparsity structure in classification (Reed, 1993; Wen et al., 2016; Wang et al., 2021).
Our SRP method is based on regularization to form desired hardware-friendlly sparsity structure. Primarily, there are three questions to answer in SRP: (1) which pruning criterion is used to choose unimportant filters; (2) which kind of regularization is chosen to induce sparsity; (3) how to schedule the pruming process to make the algorithm robust and easy to use.
(1) Pruning Criterion. When it comes to formulated pruning criteria, they mainly fall into three groups: magnitude based (Han et al., 2015; Li et al., 2017), 1st-order gradient based (Molchanov et al., 2017; 2019), and 2nd-order gradient based (LeCun et al., 1990; Hassibi & Stork, 1993; Wang al. 2019a). The Ist-order and 2nd-order gradient-based criteria are based Taylor approximation of the increased loss when a weight is removed, pioneered by (LeCun et al., 1990). They are more accurate than magnitude-based one right after pruning. However, pruning is typically followed by a finetuning process (Reed, 1993) to regain performance. After finetuning, the advantage of the Ist-order and 2nd-order gradient-based criteria usually becomes marginal. Meanwhile, they are typically more costly than the magnitude- based criteria. Taking the cost and flexibility into account, we simply employ magnitude-basee pruning criteria.
Specifically, given a pretrained SR network, for all the free Conv layers, we sort the filters in as- cending order by their -norms and select the filers with the least norms as unimportant filters, by a predefined pruning ratio E (0,1). For the constrained Conv layers, the problem becomes a little bit more complex. Given the pruned index constraint, idcally, we can only prune the filters in the intersection set (denoted by ((ee)) for the whole network,
S(ne))
n 5(1)
(3)
iEC
4

---

Published as a conference paper at ICLR 2022
where ( represents the set of unimportant filters selected by certain pruning criterion in the I-th Cony layer; denotes the set of constrained Conv layers. For top-perri uop-performed image SR networks, [C] ()- denotes cardinality of a set) is at the order of magnitude of hundreds (e.g.. [C] 200 for RCAN). Meanwhile, given a pretrained model, its pruned filter indices by I1-norm are usually random across different layers. As a consequence, [S(net)] will be rather small (in practice, on RCAN, we even observe |S(net)] = 0 when we try to prune 32 filters out of 64). This issue brings much trouble to user control when we want to prune layer to a pre-defined number of filters. To address this issue, we choose to randomly select a set of filters as unimportant and fixed during the whole pruning process. In practice, we find this simple scheme works pretty well. The reason enabling us to do so is that regularization for pruning is a gradual process. It allows ample time for the network recover from the introduced penalty term (this will be empirically verified in our experiments, see Fig. 3). Apart from aligning cross-layer sparsity structure, this is the second reason we choose regularization for pruning in our algorithm.
(2) Regularization Form. Although L1 regularization is well-known to induce sparsity in machine learning. it hard to tune the coefficient to realize desired trade-off between sparsity and perfor- mance. Instead, L2 regularization is thereby adopted in our method given its more tamable control over the sparsification process note the gradient of L2 regularization is proportion to the weight magnitude while the gradient of L, regularization is not. Specifically, given the loss function L of neural network O, the total error function E with adaptive L2 regularization term is formulated as
1 E(O:D) = (():)) + 2
Σ 0 15
where DD stands for the training dataset; W refers to the j-th filter in 1-th Conv layer; aj stands for the Ly regularization co-efficient for that filter. If filter falls into the set of unimportant filters st0,, (1) 0; otherwise, aj
(1) O since we do not restrict the learning of important filters.
(3) Regularization Schedule. As mentioned above, for the constrained Conv layers, select the unimportant filters randomly. To mitigate the side effect of this sub-optimal choice, the network should be provided with abundant training iterations adapt that it can transfer its expressive power to the remaining part of the network. To achieve this goal, we choose to gradually increase the penalty strength, drawing inspirations from (Wang et al., 2019b; 2021),
(5) where (a pre-defined constant) is a small amount that the penalty strength is increased each time. This design allows us to raise the penalty strength to large amount eventually (like 0.5 in our experiments; in contrast, note the normal L2 regularization co-efficient is at the scale of 10 10-4), As result, the unimportant weights are driven rather close to zero, and removing them will hardly hurt the performance. Besides, a is updated every T iterations during training for stability.
aj = =a + A,
(1)
(1)
To indicate when to terminate the pruning process, a ceiling limit T is introduced for the regulariza- tion co-efficient a. The pruning is over when all the a's for unimportant filters arise to T.
3.3 LEARNING EFFICIENT IMAGE SR MODELS VIA SRP
SRP can be applied to top-performing SR algorithms in a plug-and-pla manner - simply replace the loss function C in Eq. (4) with the loss objective function of an SR approach of interest. The proposed L2 regularization can be implemented on any deep learning framework very easily.
Upon finishing pruning, we take away the unimportant filters (not zeroing out them only, but literally removing them from the model), which will give us a compact SR model. Finally, the compact model will be finetuned to regain performance following the common practice in pruning (Reed, 1993).
3.4 IMPLEMENTATION DETAILS
We explain how to apply SRP to both lightweight and deep image SR networks to demonstrate the method can work with a wide range of SR networks under different computation budgets.
Lightweight Networks. First, we adapt the EDSR baseline (Lim et al., 2017) with 16 residual blocks and remove its final Conv layer to reduce model size. The reconstruction upscaling is realized by the pixel-shuffle layer (Shi et al., 2016) following common practice. The channel number in the revised EDSR baseline is first set as 256 and then pruned to 45. For x2 scale, we reduce the number of parameters from 19.5M 609K and name the compressed model as SRPN-Lite.
5

---

"hhe code is available at http://gihuab.commnnnnn--ccch httpp:////ttihhccc////ss//m////a Published as a conference paper at ICLR 2022
Pruning ratio
Params (K)
FLOPs (G)
Scratch
41-oorm pruning (Li et al,, 2017)
SRP (ours)
0.1
1,101.8
254.5
37.85
37.91
37.97
0.3
681.1
157.7
37.81
37.81
37.89
0.5
381.8
88.9
37.75
37.73
37.84
0.7
154.2
36.5
37.56
37.58
37.71
0,9
26.9
7.3
36.74
36.87
37.28
Table 1: PSNR (dB) values on the Sct5 (x2) between SRP and other two baselines to achieve exactly the same compact network. "Scratch" stands for training the small network from scratch. "Lormmmm" removes filters with the smallest -normss. The unpruned model is EDSR baseline (Params: 1,369.9K, FLOPs: 316.3G, PSNR: 37.99 dB). The best results are highlighted.
model.body. 12.ll..12bbbooo. b.ddo.
model.body.122 bt..boyd....b.....o. body.
Mean Li-norm
Mean Li-noru
0.06i
0.04
Pruned filters Kept filtens
0.04
Pruned filters
Kept filters
0.02
0.02
0.00
20
60 50
100
0
40
60
100
Iteration (k)
Iteration (k)
Figure 3: Plots of the mean Lı-norm of filters vs. iterations for "model.body. 1o.dy.1.bbbr body.'0 (free Conv layer) and "model.body wow.loo......d........ 1oddy... (constrained Conv layer) in EDSR baseline. As expected, the unimportant filters ("Pruned filters") are driven down (owing to strong regularization); interestingly, the important ones arise spontaneously.
Deep Networks. To apply SRP to the very deep network RCAN (Zhang et al., 2018b), representa- tive top top-performing deep SR network with over 400 Conv layers, we revise RCAN by removing all the channel attention modules (Zhang et al., 2018b). The channel number in the revised RCAN is chosen as 96 and then pruned to 64. For x2 scale, we reduce the number of parameters from 34.5M 15.3M and dub the compressed model as SRPN.
4
EXPERIMENTAL RESULTS
4.1 SETTINGS
Data and Evaluation. We use DIV2K dataset (Timofte et al., 2017) and Flickr2K Lim et al. (2017) as training data, following most recent works (Timofte et al., 2017; Lim et al., 2017; Zhang et al., 2018a; Haris et al., 2018). For testing, we use five standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban 100 (Huang al., 2015), and Mangal05 199 (Matsui et al., 2017). The SR results are evaluated with PSNR and SSIM (Wang al., 2004) on the Y channel in YCbCr space.
Training Settings. Following (Zhang et al., 2018b), data augmentation used in training - training images randomly rotated by 90% 180 270 and flipped horizontally. Image patches (patch size 48 x88 are cropped out to form each training batch. Adam optimizer (Kingma & Ba, 2014) is adopted for training with =0=0,9 32=0.999, and e=10-8 Initial learning rate is to 10-4 and then decayed by factor 0.5 every 2 10" iterations. We use PyTorch (Paszke et al., 2017) to implement our models with a Tesla V100 GPU
4.2 ABLATION STUDY
The EDSR baseline with 16 residual blocks (Lim et al., 2017) is used as the backbone for ablation study given its broad use in the community.
Comparison with Baseline Pruning Methods. To start with, we show that our SRP more effec- tive than the available baseline compact image SR network learning methods. Two baseline methods

---

Published as a conference paper at ICLR 2022
Params
Mult-Adds
PSNK
SSIM
PSNK
SSIM
PSNK
SSIM
PSNK
Urban
SSIM
SRCNN
57K
52.7G
36.66
0,9542
32,42
0.9063
31.36
0.8879
29.50
0.8946
FSRCNN
X2
12K
6.0G
37.00
0.9558
32.63
0.9088
31.53
0.8920
29.88
0.9020
VDSR
665K
612.6G
37.53
0,9587
33.03
0.9124
31,90
0.8960
30.76
0.9140
DRCN
X2
1,774K
17,974.3G
37.63
0,9588
33.04
0.9118
31.85
0.8942
30.75
0.9133
LapSRN
813K
29.9G
37.52
0.9590
33.08
0.9130
31,80
0.8950
30.41
0.9100
DRRN
X2
297K
6.796.9C
37.74
0.9591
33.23
0.9136
32.05
0.8973
31.23
0.9188
MemNNtt
677K
2.662.4G
37.78
0.9597
33.28
0.9142
32.08
0.8978
31.31
0.9195
CARN
X2
1,592K
222.8G
37.76
0.9590
33,52
0.9166
32.09
0.8978
31.92
0.9256
IMDN
694K
158.8G
38.00
0,9605
33.63
0.9177
32.19
0.8996
32.17
0,9283
SRPN-Lise (ours)
X2
609K
139,9G
38.10
0.96U8
33,70
0.9189
32.25
0.9905
32.26
0.9294
SRCNN
x3
57K
52.7G
32.75
0,9090
29.28 28
0.8209
28.41
0,7863
26.24
0.7989
FSRCNN
12K
5.0G
0,8080
VDSR
x3
665K
612.66
33.66
0.9213
29.77
0.83114
28.82
0.7976
27.14
0.8279
1.774K
0.8311
0.7963
27.15
0.8276
DRRN
X3
297K
6,79% 993
34.03
0.9244
29.96
0.8349
28.95
0,8000
27.53
0.8378
MemNett
677K
2.662.4G
0.8001
27.56
0.8376
CARN
X3
1,592K
118.8G
34.29
0.9255
30.29
0.8407
29.06
0.8033
28. 06
0.8499
IMDN
71.5G
0.8417
0,8046
0.8519
SRPN-Lite (ours)
X3
615K
62.7G
34.47
0,9276
30.38
0,8425
29.16
0.8061
28.22
0.8534
SRC
57K
52.7G
30.48
0.8028
27.49
0.7503
26,90
0.7101
24.52
0.7221
FSRONN
X4
12K
4.6G
30.71
0.8657
27.59
0.7535
26.98
0.7150
24.62
0.7280
VDSR
665K
612.66
31.35
0.8838
28.01
0.7674
27.29
0.7251
25.18
0.7524
DRCN
X4
1,774K
17,974.3G
31.53
0.8854
28.02
0.7670
27.23
0.7233
25.14
0.7510
LapSRN
813K
149.4G
31,54
0.8850
28.19
0.7720
27.32
0.7280
25.21
0,7560
DREN
X4
297K
6,796.9G
31.68
0.8888
28.21
0.7720
27.38
0.7284
25.44
0.7638
MemNer
677K
2,662.4G
31.74
0.8893
28.26
0.7723
27.40
0.728
25.50
0.7630
CARN
X4
1,592K
90.9G
32.13
0.8937
28.60
0.7806
27.58
0.7349
26.07
0.7837
IMDN
715K
40,9G
32.21
0.8948
28.58
0.7881
27.56
0.7353
26.04
0,7838
SRPN-Lite (ours)
x4
623K
35.8G
32.24
0.8958
28.69
0.7836
27.63
0.7373
26.16
0.7875
Table 2: Quantitative results of lightweight SR networks. Best results are highlighted.
Bicubic
SRCNN
ESRCNN
Urbun 100: img.0020 (x44
LapSRN
MemNet
CARN
IMDN
SRPN-Lite (ours)
IIR
Bicuhic
SRCNN
ESRCNN
VDSR
Urhan100: img 024
LapSRN
MemNet
CARN
IMDN
SRPN-L.te (ours)
Figure 4: Visual comparison of different lightweight SR approaches the Urban dataset (x4).
are compared to: training from scratch and the original Lı-norm pruning (Li et al., 2017). The re- sults are presented in Tab. As we can see, our pruned network delivers the best PSNR consistently across different pruning ratios. Notably, under greater pruning ratio, the advantage of our method over scratch training -i-norm pruning more evident in general, suggesting that our SRP is more valuable in aggressive pruning cases. Of particular note is that, our method SRP also utilizes the L -nomm as the scoring criterion to select unimportant filters, same as (Li al., 2017). Nevertheless, our method delivers significantly better results than theirs. The primary reason is that they do not impose any regularization on the pruned structure; the remaining feature maps are thus mismatched in residual blocks after pruning. In contrast, our method SRP is not bothered by this issue, showing the effectiveness of our proposed structure regularization.
Visualization of Pruning Process. In Fig. 3, visualize the pruning process by plotting the mean L1 Li-norm of filters in two layers of EDSR bascline during SRP training. The filters are split into two groups, kept filters and pruned filters. As is shown in the figure, the mean I--norm of the pruned filters goes down gradually because the penalty grows stronger and stronger, driving them towards zero. Interestingly, note the Li-norms of the kept filters arise themselves. Recall that there is no explicit regularization to promote them to grow. In other words, the network learns to recover by itself. akin to the compensation effect found in human brain (Duffau et al., 2003).
7

---

Published as a conference paper at ICLR 2022
Urban
Manga
Method
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
EDSR
X2
38.11
0.9602
33.92
0.9195
32.32
0.9013
32.93
0.9351
39.10
0.9773
DBPN
38.09
0.9600
32.55
0.9324
0.9775
RCAN
x2
38.27
0.9614
34.12
0.9216
32.41
0.9027
33.34
0.9384
39.44
0.9786
SRFBN
X2
38.11
0.9609
33.82
0.9196
32.29
0,9010
32.62
0.9328
39.08
0.9779
38.31
0.9620
0,9028
33.10
0.9370
HAN
x2
38.27
0.9614
34.16
0.9217
32.41
0.9027
33.35
0.9385
39.46
0.9785
IGNN
X2
38.24
0.9613
34.07
0.9217
32.41
0.9025
33.23
0.9383
39.35
0.9786
CSNLN
38.28
0.9616
34.12
0,9024
33.25
0.9386
RFANet
X2
38,26
0.9615
34.16
0.9220
32.41
0,9026
33.33
0.9389
39.44
0.9783
SRPN (Ours)
X2
38.34
0.9619
34.29
0.9232
32.47
0,9032
33,50
0,9401
39.76
0,9796
EDSR
34,65
0.9280
0.8093
0.8653
0,9476
RCAN
X3
34.74
0.9299
30.65
0.8482
29.32
0.8111
29.09
0.8702
34.44
0.9499
SREBN
X3
34,70
0.9 0.9292
30.51
0.8461
29.24
0.8084
28.73
0.8641
34.18
0.9481
34.75
0.9300
28.93
0.8671
HAN
X3
34.75
0.9299
30.67
0.8483
29.32
0.8110
29.10
0.8705
34.48
0.9500
IGNN
X3
34.72
0.9298
30.66
0.8484
29.31
0.8105
29.03
0.8696
34.39
0.9496
CSNLN
34.74
0.9300
29.13
0.8712
RFANet
X3
34.79
0.9300
30.67
0.8487
29.34
0.8115
29.15
0.8720
34.59
0.9506
SRPN (OurS)
X3
34.84
0,9303
30.76
0.8497
29.39
0.8120
29.36
0.8749
34.88
0,9515
EDSR
32.46
0.8968
0.7420
26.64
0.8033
31.02
0.9148
SRMDNF
X4
31.96
0.8925
28.35
0.7787
27.49
0.7337
25.68
0.7731
30.09
0.9024
RCAN
X4
32.63
0. 0,9002
28.87
0,7889
27.77
0.7436
26.82
0,80887
31.22
0.9173
SRFBN
32.47
28.81
SAN
X4
32,64
0.9003
28.92
0.7888
27.78
0.7436
26.79
0.8068
31.18
0.9169
HAN
X4
32.64
0.9002
28.
0,7890
27.80 80
0,7442
26.85
0,8094
31.42
0.9177
IGNN
32.57
0.7434
CSNLN
X4
32,68
0.9004
28.95
0.7838
27.80
0.7439
27.22
0.8168
31.43
0.9201
RFANet
X4
32.66
0.9004
28. 88
0,7894
27.79
0.7442
26.92
0.8112
31.41
0.9187
SRPN (ours)
32.72
27.87
0.7457
27.25
0.8172
Table 3: Quantitative results of large SR networks. Best results highlighted.
Bicuhic
Urban100: img (x4)
SRIBN
IGNN
CSNLN
SRPN (ours)
HR
Bicubic
EDSR
DRPN
RCAN
Urban100: 062 (x4)
SRIBN
IGNN
CSNLN
RFANer
SRPN (ours)
Figure 5: Visual comparison of different large SR networks on the Urban dataset (x4).
4.3 COMPARISONS WITH LIGHTWEIGHT NETWORKS
We compare our SRPN-Lite with lightweight SR networks: SRCNN (Dong al., 2014). FSR- CNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), LapSRN (Lai al., 2017), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), CARN (Ahn al., 2018) and IMDN (Hui et al., 2019). The compared results are produced from their released code.
Performance Comparisons. Table shows PSNR/SSIM comparisons for x2, x3, and x4 SR. When compared with previous leading models, our SRPN-Lite obtains the best performance on all benchmarks and scaling factors. Unlike most comparison methods, which achieve efficiency through careful network designs, work starts with the existing EDSR baseline (Lim et al., 2017) and prunes it to a much smaller network, showing the effectiveness of our proposed SRP.
Model Size and Mult-Adds, Our SRPN-Lite has the fewest parameter number in comparison to recent efficient SR works such as MemNet, CARN, and IMDN. The comparison in terms of Mult- Adds (measured when the output size is set 3x x,280 x 720) is also presented. As seen, SRPN- Lite costs fewer Mult-Ade Ault-Adds than most comparison methods. These results demonstrate the merits of SRP against other counterparts in striking better network performance-complesy trade-off.
8

---

Published as a conference paper at ICLR 2022
Method
Params
Mult-Adds
Set5
Set
B100
Urhan
MoreMNAS-A (Chu al., 2019b)
1,039K
238.6G
37.63
33.23
31.95
31.24
FALSR-A (Chu ct al., 2019a)
1,021K
234.7G
37.82
33.55
32.12
31.93
CARN+KD (Lee et al., 2020)
1,592K
222.8G
37.82
N/A
32.08
N/A
SRPN-Lite (ours)
609K
139.9G
38.10
33.70
32.25
32.26
Table 4: Model complexity comparisons (x2). Output size is 3 x,280x calculate Mult-Adds.
Visual Comparisons. The visual comparisons at x4 scale are shown in Fig. 4. It is easy to see that most of the comparison algorithms can barely recover the lost texture details in the correct direction; moreover, they suffer from blurring artifacts. In stark contrast, our SRPN-Lite effectively alleviates the blurring artifacts and recovers sharper structural fine textures in the right way, justifying the performance superiority of our SRP approach over other methods.
4.4 COMPARISONS WITH LARGER NETWORKS
Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks. As stated in Sec. 3.4, we extend our SRP method to larger network training and obtain a deeper network SRPN, In Tab. 3. we compare with large SR networks: EDSR (Lim al., 2017), DBPN (Haris et al., 2018), RCAN (Zhang et al., 2018b), SREBN (Li et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020), and RFANet (Liu et al., 2020). The compared results are produced from their released code.
Performance Comparisons. In Tab. 3, we observe that our SRPN achieves the best PSNR/SSIM values across different test sets and scales, except for SSIM value on Set5 (x2). Our SRPN utilizes slightly fewer parameters (15.3M vs. 15.4M) with the same network depth as RCAN (Zhang et al., 2018b). Notably, our SRPN removes the channel attention module in RCAN, but still achieves better results. This is mainly because our SRPN is initially trained from a wider and stronger network (with 96 channels). We train the much larger network and then prune to the target size.
Visual Comparisons. We further show visual comparisons with larger networks (e.g.. CSNLN, RFANet, and IGNN) in Fig. 5, we can also observe similar situations in Fig. 4. For example, in img. 001, other methods suffer from heavy blurring artifacts. In other cases, they may not recover clear structures img-044 044 or even wrong details img.062. Our SRPN achieves more visually pleasing results with more structural details. Those comparisons demonstrate that our SRP method can be applied to train larger networks and obtain comparable or even better performance.
4.5 MORE COMPARISON WITH MODEL COMPRESSION TECHNIQUES
We further compare our SRP to other representative efficient image SR approaches via model com- pression. Concretely, neural architecture search based methods (Chu et al., 2019b;a) and knowledge distillation (KD) based methods (Lee et al., 2020) are compared to. Quantitative results at x2 scale are presented in Tab. 4, where our SRPN-Lite delivers better PSNR results across different datasets with fewer parameters and Mult-Adds. With our SRP pruning method, there is no need to search massive network architectures or pretraining a teacher network, which usually consumes consider- able computation resources. These comparisons show that our SRP, as network pruning method, has as much potential (if not more) as other model compression techniques for efficient image SR.
CONCLUSION
Existing efficient image SR methods have achieved promising results with moderate model param- eters and Mult-Adds. However, they neglect to extend their approaches to larger networks, which usually perform much better with more parameters and/or operations with residual learning. Mean- while, although model compression approaches, like knowledge distillation and neural architecture search, have also been exploited for efficient image SR networks, they typically demand excessive computation resources. As another promising model compression method, network pruning is hard to be incorporated into lightweight SR models directly. To address these issues, we present structure- regularized pruning (SRP), which imposes L2 regularization on the pruned structure align the locations of pruned filters across different layers. It can transfer the expressive power in the unim- portant filters to the remaining part of the network. We employ SRP to train both lightweight and large image SR networks, resulting in a lightweight network SRPN-Lite and very deep one SRPN, delivering superior performance over recent SOTA efficient SR methods.
Acknowledgments. This research is supported by the U.S. Army Research Office Award W91 177-1--7-

---

Published as a conference paper at ICLR 2022
REFERENCES
Namhyuk Ahm, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV, 2018. 3,3,8 8
Marco Bevilaequa, Aline Roumy, Christine Guillemot, and Marie Line Alheri-Morel, Low-complexity single- image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 6
Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super- resolution with neural architecture search. arXiv preprint 1901.0020000 2019a. 1,3,9
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Hailong Ma. Multi-ohjective reinforced evolution in mobile neural architecture search. arXiv preprint /rXiv:/90..01007 2019b. 9
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu- Thu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-rosolution. In CVPR, 2019. 9
Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, 2014. 2,
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In ECCV, 2016. 1,8
Duffau, I. Capelle, D Denvil, N Sichez, P Gatignol, M Lopes, MC Mitchell, JP Sichez, and R Van Effenterre. Functional recovery after surgical resection low grade gliomas in eloquent brain: hypothesis of brain compensation. Journal of Neurology, Neurosurgery & Psychiatry, 74(7):901-907, 2003.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. JMLR, 2019. 1
Han, X. Liu, and H. Mao. FIF: efficient inference engine on compressed deep neural network. ACM Sigarch Computer Architecture News, 44(3):243-254 201 3
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. In NewrIPS, 2015. 3,4
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016b. 2,
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super- resolution. In CVPR, 2018. 6.9
B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NeurlPS, 1993. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.7
Zibin He, Tao Dai, Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Fcature-affnityy based knowledge distillation for efficient image super-resolution. ICIP, 2020. 3
Geoffrey Hinton, Oriol Vinyals, Jeff Dean. Distilling knowledge in a neural network. In NewrlPS Workshop, 2014. 1
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self- exemplars. In CVPR, 2015.
Zheng Hui, Xinho Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi- multi-distillation network. In ACM MM, 2019. 1,
Jiwon Kim, Jung Kwon Loc, and Kyoung Mu Lec. Accurate image super-resolution using very deep convolu-
tional networks. In CVPR, 2016a. 1, 2, 8
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super- resolution. In CVPR, 2016b. 3,
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 6
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2022
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In NewrlPS, 1990. 4
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution. In ECCV, 2020. 3,
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficieent convnets. In ICLR, 2017. 2, 3, 6,
Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image super-resolution. In CVPR, 2019. 9
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 1, 4, 5, 6, 8, 9
Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu, Residual feature aggregation network for image resouticso In CVPR, 2020. 2,
Xiaotong Luo, Yuan Xie, Yulum Zhang, Yanyun Qu, Cuihua Li, and Yun Fu. Latticenet: Towards lightweight image super-resolution with lattice block. In ECCV, 2020. 1,
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 6
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-hased manga retrieval using manga109 dataset. Multimedia Tools and Applications, 2017.
Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang. Thomas S Huang, and Humphrey Shi. Image super- resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In CVPR, 2020. 2,
Molchanov, S. Tyree, and T. Karras. Pruming convolutional neural networks for resource efficient inference. In ICLR, 2017. 4
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation neural network pruning. In CVPR, 2019.
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang. Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution holistic attention network. ECCV, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
R. Reed. Pruning algorithms survey. IEEE Transactions Neural Networks, 4(5):740-744 747, 1993. 3, 4,
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolu- tional neural network. CVPR, 2016.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Jocl S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295 2329, 2017, 2,
Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In CVPR, 2017a. 8
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In ICCV, 2017b. 8
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming- Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah. Kyoung Mu Lee, et al. Ntire 2017 challenge single image super-resofution: Methods and results. In CVPRW, 2017.
Chaoqi Wang. Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In ICML, 2019a. 4
Huan Wang, Xinyi Hu, Qiming Zhang, Yuehai Wang, Lu Yu, and Haoji Hu. Structured pruning for efficient convolutional neural networks via incremental regularization. IEFE Journal of Selected Topics in Signal Processing, 14(4):755-788, 2019b.

---

Published as a conference paper at ICLR 2022
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In JCLR, 2021. 4,5
Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from visibility to structural similarity. TIP. 2004.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NeurlPS, 2016. 3,4
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using ppars-reprrsentaation.. In Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning single convolutional super-resolution network for multiple degradations. CVPR, 2018a.
Proc. Int. Conf. Curves Surf., 2010.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. ECCV, 2018b. 2, 4, 6,
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super- resolution, In CVPR, 2018c.
Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. In ICLR, 2019. 2
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restora- tion. TPAMI, 2020.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. NeurlPS, 2021. 1,2
Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy. Cross-scale internal graph neural network image super-resolution. NeurIPS, 2020. 9
Barret Zoph and Quoc T.c. Neural architecture search with reinforcement learning. In ICTR, 2017.
12


Reviews:
[{"review_id": "yc_iRcEyR7t", "reviewer": "Reviewer_xzfp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- The proposed method achieves better performance compared to the previous lightweight SR methods. Also, the proposed method is successfully applied to larger SR models.\n\n- The proposed method is simple and effective.\n\nWeakness\n\n- The proposed method is suitable when there is a residual connection and there are few Conv layers per block. On the other hand, if there is no residual block or there are many Conv layers per block, the effect may be somewhat insignificant. There is a need to compare the proposed method with the cases of the pruned model without residual block and of the pruned model with many Conv layers in the block.\n\n- I think the comparison with the existing Purning method is somewhat lacking. Is there any comparison with unstructured pruning other than channel pruning? Also, Is there any comparison with the pruning technique used in semantic segmentation other than SR?\n--> CAP: Context-Aware Pruning for Semantic Segmentation, WACV 2021\n\n- There is some ambiguity in the details of the proposed method. In the Pruning criterion, how many random unimportant filters are selected? In the Regularization schedule, what is the initial alpha? How pre-defined constant is decided? \n\n- Performance improvement in Table1 is not significant. Also, the proposed method is only evaluated on only one backbone SR model. It is difficult to say that the proposed method has been generalized.\n\n- In Table 3, there are no reports about Params and Mult-Adds.\n\n- In Table 4, there are needs more comparisons on different datasets\n", "summary_of_the_paper": "This paper proposes a Structure-Regularized Pruning (SRP) for the super-resolution task. Authors found that residual connections in SR models make it difficult to develop filter pruning methods. To resolve it, the proposed method imposes constraints on the locations of pruned filters. The constraint is to align unimportant indexes across different layers. The proposed method was achieved.\n", "summary_of_the_review": "However, there are some ambiguity explanations, and the model design and hyperparameter were arbitrarily selected.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ngjblNaAeM", "reviewer": "Reviewer_yDGk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nFirst of all, I like this idea, which jointly optimizes the network pruning and image SR. The results are impressive and show promising potential for future efficient works.\n1. Making effective SR models more efficient is of interest to a broad audience in low-level vision. This paper tackles this issue thus potentially can have a big impact.\n2. In terms of methodology, they present a new network pruning method (SRP) for efficient SR based on regularization, which is technically sound. \n3. Empirically, they apply their method to both large SR networks and lightweight networks. In both cases, they show their pruned networks achieve the best performance with fewer parameters or Mult-Adds. The results look pretty strong.\n4. The supplementary material contains pretty much information. Most of my concerns, like differences with other related works, can be addressed by the supplementary file. \n5. The writing and organization are good. The whole paper, including the supplementary, is well prepared.\n6. The authors provide demo code to reproduce the results in the paper, which makes the paper more reliable and convincing. \n\nWeaknesses:\nIn general, this paper is well-written and has extensive experiments with strong results. I have the following questions about the method and result details.\n1. About the method, applying L2 regularization for sparsity appears not a common practice since it cannot make weights exactly zero. L1 regularization (i.e., lasso) is more normal in statistics in the sense of imposing sparsity. The discussion in “Regularization Form” seems not enough to treat this question properly. The authors are highly suggested to explain more.\n2. I noted in Tab. 1, the listed “pruning ratio” seems not aligned with the parameter reduction. E.g., for pruning ratio 0.5, the compression ratio should be 2. While by the parameters, the compression ratio is 1369.9/381.8=3.6, much larger than 2. Why this? \nBtw, typos: Appendix “Line 756-760 in the main paper” seems not correct.", "summary_of_the_paper": "The authors propose structure-regularized pruning (SRP) for efficient image super-resolution. They provide a general idea to structurally prune both lightweight and large image SR networks. Extensive results are provided to support their claimed contributions.", "summary_of_the_review": "The idea is simple yet efficient for both lightweight and large image SR networks. The ablation study, like the visualization of pruning process, demonstrates the effect of the method. The main comparison results with others are impressive.\n\n--------------------------------\nThe authors addressed my concerns in the reply. After considering other reviews and response, I decide to keep my initial score and vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "biWAhlFl8M", "reviewer": "Reviewer_t8M1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The problem definition is well defined, addresses the challenges and provides a possible solution\n- The paper is well written, easy to follow, extensive explanations make the paper readable\n- The descriptions and explanations are to the point\n- Terms and concepts are well defined and explained\n- Proposes a pruning method for structured pruning\n- Gives two derived networks SRPN-L and SRPN and reports their results which shows quality improvement compared to other SOTA\n- The SRP model can be applied to SOTA networks in a plug and play model\n- Extensive experimental analysis\n- Ablation study shows that the proposed method is effective at pruning and maintaining the quality\n\nWeakness / Questions\n- Were there any effort for the SR models to be trained using real-world downsampling kernels instead of bicubic?\n- Why are the parameter counts not provided for the large network comparison table?\n- Likewise for Mult-adds\n", "summary_of_the_paper": "Although light weight SR models show promise in terms of quality while using moderate sized network, it does not extend from there into larger models, or into practical use.\nModel compression techniques have also been used to reduce network size but consumes significant resources and computation.\nNeural architecture search and knowledge distillation are some example techniques to use for compression but not sufficiently effective.\nHowever, network pruning can be a viable option for effective model compression, but can be proved tricky due to its difficulty regarding pruning of the filter for residual blocks.\nTo mitigate the issue, the authors propose a structure regularized pruning which enforces a regularization on the pruned structure to make pruned regions aligned across different layers. For instance, the method works by selecting filters with the same indices which are connected by the same residual. \nThe authors also employ a L2 regularization to drive weights to zero for unimportant filters and transfer its information to other parts of the network.\nThis is important for minimizing performance degradation.\nThe authors propose a derived network via using the structure regularized pruning, namely SRPN-L and SRPN.\nThe proposed methods show better quality  than the latest networks quantitatively and visually.\n", "summary_of_the_review": "This paper is well written, containing extensive experiments and impressive results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NQEJMgINzPJ", "reviewer": "Reviewer_N1G5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1 the paper is well-written and easy to read. \n2 the proposed method is simple yet effective, where the experimental results on both lightweight and large SR networks demonstrate state-of-the-art results.\n\nWeakness:\n1 In general, this paper lacks novelty. Filter pruning, as well as the regularization term and schedule, are common techniques in high-level vision tasks, and this paper applies them to the SR task. The difference might be the pruning criterion. Indeed, the extensive residual connections cause the main difficulty to directly apply these techniques to the SR task. However, the authors do not include deep investigations to this problem, e.g., experiments of direct application, insightful analysis about the intrinsic reasons, etc. In addition, to solve this problem, the authors randomly select a set of filters to be pruned and fix them during pruning. This may affect the stability of training, where no repetitive experiments are provided.\n2 In Table 3, the SRPN is pruned from an extended version of RCAN (with 96 channels) and achieves state-of-the-art performance. This is logical since the extended RCAN is more powerful and contains redundant parameters. It is better to, the performance of the extended RCAN should also be provided. \n3 The authors claim a general idea to prune SR networks, especially for large networks. However, only the pruning of RCAN (without channel attention) is validated. To demonstrate the generalization capacity, more networks with different topologies should be included. \n", "summary_of_the_paper": "This paper proposes a structure-regularized filter pruning strategy for efficient SISR. Considering the residual connections in the SR networks, the authors propose to prune the filters that are aligned across layers connected by the same residual. The weight of those layers are optimized using an L-2 regularization to avoid performance drop. Experiments on both lightweight and large SR networks are conducted, with superior performance both quantitatively and qualitatively.", "summary_of_the_review": "This paper is clear and well-written. However, the novelty of the paper is somehow limited. Also, there are limited insightful investigations about the idea and intrinsic motivations. Finally, some experiments are missed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper receives positive reviews. The authors provide additional results and justifications during the rebuttal phase. All reviewers find this paper interesting and the contributions are sufficient for this conference. The area chair agrees with the reviewers and recommends it be accepted for presentation.
Decision: Accept (Poster)