GhostSR: Learning Ghost Features for Efficient Image Super-Resolution
Authors: Ying Nie
Kai Han
Zhenhua Liu
Chuanjian Liu
Yunhe Wang
Venue: Accepted by TMLR ()

Abstract: Modern single image super-resolution (SISR) systems based on convolutional neural networks (CNNs) have achieved impressive performance but require huge computational costs. The problem on feature redundancy has been well studied in visual recognition task, but rarely discussed in SISR. Based on the observation that many features in SISR models are also similar to each other, we propose to use shift operation for generating the redundant features (i.e. ghost features). Compared with depth-wise convolution which is time-consuming on GPU-like devices, shift operation can bring a real inference acceleration for CNNs on common hardware. We analyze the benefits of shift operation in SISR and make the shift orientation learnable based on the Gumbel-Softmax trick. Besides, a clustering procedure is explored based on pre-trained models to identify the intrinsic filters for generating corresponding intrinsic features. The ghost features will be generated by moving these intrinsic features along a certain orientation. Finally, the complete output features are constructed by concatenating the intrinsic and ghost features together. Extensive experiments on several benchmark models and datasets demonstrate that both the non-compact and lightweight SISR CNN models embedded with the proposed method can achieve a comparable performance to the baseline models with a large reduction of parameters, FLOPs and GPU inference latency. For example, we reduce the parameters by 46%, FLOPs by 46% and GPU inference latency by 42% of x2 EDSR model with almost lossless performance. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/GhostSR.

Published in Transactions on Machine Learning Research (11/2022)
GhostSR: Learning Super-Resolution
Ghost Features for Efficient Image
Ying Nie
ying. nie Whuoweeii.
Huowei Noah's Ark Lab
Kai Han
kai. Kai.han www.haaa@@huuuuecccc
Huawei Noah's 's Ark Lab
Zhenhua Liu
liu www.hhennua@hhuwvc..com
Huawei Noah's Ark
Chuanjian Liu
fiuchuanjjann@huuwvee.comm
Huawei Noah's Ark Lab
Yunhe Wang
waahh..ang@hhuwei.ccoo
Huowei Noah's Ark Lab
Reviewed on OpenReview:
openreviee. net/ forum? id= tbd f
Abstract
Modern single image super-resolution (SISR) systems based on convolutional neural net- works (CNNs) have achieved impressive performance but require huge computational costs. The problem on feature redundancy has been well studied in visual recognition task, but rarely discussed in SISR. Based on the observation that many features in SISR models are also similar to each other, propose to use shift operation for generating the redundant fea- tures (i.e. ghost features). Compared with depth-wise convolution which is time-consuming on GPU-like devices, shift operation can bring a real inference acceleration for CNNs on common hardware. We analyze the benefits shift operation in SISR and make the shift orientation learnable basod on the Gumbel-Softmax trick. Besides, clustering procodure explored based on pre-trained models to identify the intrinsic filters for generating corre- sponding intrinsic features. The ghost features will be generated by moving these intrinsic features along certain orientation. Finally, the complete output features are constructed by concatenating the intrinsic and ghost features together. Extensive experiments several benchmark models and dalasets demonstrate that both the non-compact and light SISR CNN models embedded with the proposed method can achieve a comparable perfor- mance to the basclinc models with large reduction of paramcters, FLOPs and GPU infer- ence latency. For example, we reduce the parameters by 46%, FLOPs by 46% and GPU infer- ence latency by 42% of x2 FDSR model with almost lossless performmnne Code will be avail- able at http::://git....cmmniiiitti http:///ee..riiioooo///////////s
1 Introduction
Single image super-resolution (SISR) is classical low-level computer vision task, which aims at recovering high-resolution (HR) image from its corresponding low-rosolution (LR) image. Since multiple HR images could be down-sampled to the same LR image, SISR is an ill-posed reverse problem. Recently, deep con- volutional neural network (CNN) based methods have made significant improvement on SISR task through carefully designed network architectures. The pioncer work SRCNN (Dong et al., 2014) which contains only three convolutional layers outperforms the previous non-deep learning methods by large margin. Subse- quently, the capacity of CNNs is further excavated with deeper and more complex architectures (Kim et al.

---

Published in Transactions on Machine Learning Research (11/2022)
Clustrrigg
Lp Left Shift
0.8-
Shift.
0.4
0.0
Inhinsic Filters
ghost
-0.-
intrinsic
-0.8-
Layerl
Layo2
Layer3
Input
Idantity
Output
Figure 1: Visualization of features generated by different layers in VDSR (Kim et al. 2016a), which obviously has many similar fcatures (Left). The heatmaps of fcatures' cosine similarity also indicate the phenomenon of redundancy. The redundant features (i.e. ghost features) can be generated by cheap operation such as shift based on intrinsic features. The intrinsic features are generated by intrinsic filters, which are selected via clustering pre-trained model (Right). The red dots and green rectangles in clustering represent the cluster centers and the selected intrinsic filters, respectively. Shifting to the upper left is taken as example to visualize the shift operation.
2016a; Lim et al. 2017 Zhang et al. 2018b cp, which significantly improve the performance. However, these networks usually involve a large number of parameters and floating point operations (FI.Oss), limiting its deployment on portable devices like mobile phones and embedded devices.
Many works have been proposed model compression for visual classificatinn task, including lightweight architecture design (Howard et al. 2017 Zhang et al. 2018a Ma et al. 2019 Han et al. 22020), pruming (Han ct al. 2016 Li et al.
2017 and quantization (Zhou et al. 2016 Nie et al. Li et al. 2021j. etc. Wherein, Ghost Net (Han et al.
2020 makes a deep analysis of feature redundancy in the neural network on classifica-
tion task, and then proposes to generate the redundant features (i.e. ghost features) with cheap operations based on the intrinsic features. In practice, the intrinsic features are generated by the regular convolution operation (i.e. expensive operation), and then the depth-wise convolution operation (tee cheap operation) is employed the intrinsic features for generating the ghost features. Finally, the complete output features are constructed by concatenating the intrinsic and ghost features together. GhostNet achieves competitive accuracy on the ImageNet dataset with fewer parameters and FLOPs. However, there are two major draw- backs in GhostNet that prevent its successful application to other vision tasks, especially the SISR task. Firstly, the so-called cheap operation is not cheap at all for the commonly used GPU-like devices. That is, depth-wise convolution can not bring practical speedup on GPUs due to its low arithmetic intensity (ratio of FLOPs to memory accesses) (Wu al., 2018 Zhang al.) 2018a)). In addition, GhostNet simply divides the output channel evenly into two parts: one part for generating intrinsic features and the other part for generating ghost features, which does not take into account the prior knowledge in the pre-trained model.
Compared with visual classification networks, the SISR networks tend to involve more number of FLOPs. For example, ResNet-50 He et al. (2016) and EDSR Lim et al. (2017) are typical networks in classification task and super-resolution task, respectively. Correspondingly, the FLOPs required for processing a single 224 x 224 image using ResNet-50 and x2 EDSR are 4.1G and 2270.9G, respectively. Therefore, the compression and acceleration of SISR networks more urgent and practical. To date, many impressive works have been proposed for compressing and accelerating the SISR networks (Ahn et al. 2018 Song al,, 2020 Hui et al.) 2018 Zhao et al. 2020 Zhang et al.) 2021b a Li et al. 2022). However, the basic operations in these works are still conventional convolution operation, which do not consider the redundancy in features and the practical speed in common hardware. Since most of the existing SISR models need to preserve the overall texture and color information of the input images, there are inevitably many similar features in each layer as observed in the left of Figure [ addition, considering that the size of images displayed on modern intelligent terminals are mostly in high-resolution format (2K-4K), the efficiency of SISR models should be maintained on the common platforms with large computing power (e.g. GPUs and NPUs).
2

---

Published in Transactions on Machine Learning Research (11/2022)
Considering that the phenomenon of feature redundancy also exists in the SISR, models, in this paper, we introduce a Ghost SR method to generate the redundant features with a real cheap operation, i.e. shift operation (Wu et al. 2018 Jeon & Kim 2018 Chen et al.) 2019). Compared with the depth-wise convolution in GhostNet (Han et al. 2020), shift operation can bring practical speedup in common hardware with negligible FL.OPs and parameters. Specifically, the shift operation moves the intrinsic features along a specific orientation for generating the ghost features. We conclude two benefits of the shift operation for the SISR task, including the ability to enhance high frequency information and the ability to enlarge the receptive field of filters. In addition, the Gumbel-Softmax trick (Jang et al. 2016) is exploited that each layer in the neural network can adaptively learns its own optimal moving orientation of the intrinsic features for generating the ghost features. The complete output features are then constructed by concatenating the intrinsic and ghost features logether. With the efficient CUDA implementation of shift. operalion, GhostSR brings a practical acceleration on GPUs during inference. Last but not least, the prior knowledge in the pro-trained model is taken into account in distinguishing which part is intrinsic features and which part is ghost fras Specifically, the intrinsic filters are identified via clustering the weights of pre-trained model based output channel, and the features generated by the intrinsic filters are taken as intrinsic features. The other part is taken as the ghost features. The extensive experiments conducted on several benchmarks demonstrate the effectiveness of the proposed method.
2 Related Works
2.1
Model Compression
In order to compress deep
CNNs, a series of methods have been proposed which include lightweight architec-
ture design (Howard et al.
2017 Zhang al.
2018a Ma et al. 2019 Han al. 2020), network pruning (Han)
2016 Li et al. 2017 et al.
Ding et al.) 2019). knowledge distillation Hinton et al. [015 Romero et al. 2015)
and liw-bit quantization
Courbariaux et al. 2015 Zhou et al. 2016: Nie et al. Li et al. 2021). etc. Han et
al.
(Han et al.
2016 remove the connections whose weights are lower than a certain threshold in network.
Ding et al.
Ding el al.; 2019) propose a centripetal SGD optimization met for network slimming.
Hinton et al.
(Hinton et al.) 2015) introduce the knowledge distillation scheme to improve the performance
of student model by inheriting the knowledge from a teacher model. Courbariaux al. (Courbariaux et al., 2015) quantize the weights and activations into 1-bit value to maximize the compression and acceleration of the network. In addition, lightweight network architecture design has demonstrated its advantages in constructing efficient neural network architectures. MobileNets (Howard et al. 2017 Sandler et al. 2018) are a series of lightweight networks based on bottleneck structure. Liu et al. (Liu et al.) 2022) propose to enlarge the capacity of CNN models by fine-grained FLOPs allocation for the width, depth and resolution on the stage level. Wu et al. (Wu et al. 2018 first propose the shift operation which moves the input features horizontally or vertically, then Jeon al. (Jeon Kim 2018) and chen et al. (Chen et 2019)
further make the shift operation learnable in visual classification task. ShiftAddNet You al. (2020) and AddNAS You et al. (2022) leverage the bit-wise shift for building energy-efficient networks. Recently, Han et al. (Han et al. 2020) analyze the redundancy in features and introduce novel GhostNet module for building efficient models. In practice, GhostNet generates part of features with depth- -iite convolution, achieving satisfactory performance in high-level tasks with fewer parameters and FLOPs. However, due to the fragmented memory footprints raised by wist convolution, GhostNet can not bring a practical acceleration on common GPU-like devices.
2.2 Efficient Image Super-Resolution
Numerous milestone works based on convolutional neural network have been proposed image super- resolution task (Dong et al.] 2014 Kim et al.) 2016a Tai ct al. 2017b; Lim et al. 2017 Zhang et al.) 2018c b). However, these works are difficult to deploy on resource-limited devices due to their heavy computation cost and memory footprint. To this end, model compression on super-resolution task is attracting widespread attention. FSRCNN (Dong et al. (2016) first accelerate the SISR network by a compact hourglass-shapp architecture. DRCN (Kim et al.] (2016bjand DRRN Tai et al., (017a)) adopt recursive layers to build deep network with fewer parameters. CARN (Ahn et al. 2018) reduce the computation overhead by combining the
3

---

Published in Transactions on Machine Learning Research (11/2022)
efficient residual block with group convolution. IMDN Huui al. 2019 construct the cascaded information
multiddistilatioon block for efficient, feature extraction. LatticeNet [Luo et al. 2020 utilize series connection
of lattice blocks and the backward feature fusion for building a lightweight SISR model. SMSR (Wang et al, 2021] develop a sparse mask network to learn sparse masks for pruning redundant computation. Attention mechanism also introduced to find the most informative region to reconstruct high-resolution image with
better quality (Zhao al. 2020; Muqeet et al.) 2020 Zhang al. 2018b Magid al.
2021 Nuu et al.
2020). To improve the performance of lightweight networks, distillation has been excavated to transfer the knowledge from experienced teacher networks to student networks (Hui et al. 2018 Gao et al. 2018
Lee et al., 2020b). Neural architecture search (NAS) is also employed to exploit. the efficient architecture
for image super-resolution task (Song et al. 2020 Guo et al., 2020 Lee et al. 2020a) Zhan et al.) 2021
Recently, based on the motivation that different image regions have different. restoration difficulties and can be processed by networks with different capacities, ClassSR (Kong et al. 2021) and FAD (Xie et al.) 2021) propose dynamic super-rcsolution networks. However, the dynamic SISR networks equipped with multiple processing branches usually contain several times the nunber of parameters of the original network, which limits its deployment on portable devices.
3 Approach
In this section, we describe the details of the proposed GhostSR method for efficient image super-resolution.
3.1 Shift for Generating Ghost Features
The CNN-based super-resolution models consist of massive convolution computations. For vanilla convo- lutional layer, producing the output features Y E Rexxxx requires hwc.c,s² FLOPs where Co, h, w, Ci X are the number of output channels, the height of output, the width of output, the number of input channels and the kernel size. The computational cost of convolution consumes much energy and inference time. On the other hand, we observe that some fcatures SISR network is similar to another ones, that is, these features can be viewed as redundant versions of the other intrinsic features, as shown in the left of Figure [ We term these redundant features ghost features. In fact, ghost features and high-frequency information, which cannot be directly removed (Han et al., 2020 Yuan et al. 2020). Instead of discarding, we propose to utilize a more efficient operator, i.c. shift operation, to generate them.
provide wealth of texture
Assuming that the ratio of ghost features is where 0 S A < 1, then the number of intrinsic and ghost features is (1 (1-Ai -)c, and Aco, respectively. The intrinsic features I (1 are generated by regular convolution, and the ghost features G E A x are generated by shift operation based on I since shift is cheap yet has many advantages on super-resolution task, which will be discussed in detail later. Formally, the vertical and horizontal offsets to be shifted are i, and jor where -d < io S d, -d jo d, and d is the maximum offset, then the element in position (u,x) of R can be obtained as:
d d
Gy.r, =
T Joti Wis1 (=-d-=-d
Σ
Σ
(1)
where W (0, (222+)))((2++) is one-hot matrix denoting the offset values, and a and c2 the channel index of ghost features and the corresponding index of intrinsic features, respectively. All the elements in W are 0 except that the value in the offset position is 1:
Woy = (1. -
-(1; 0. otherwise.
i is and j = jo:
(2)
Finally, we concatenate the intrinsic and ghost features together as the complete output: O = [1, GJ. Com- pared to original convolution layer, our method cuts the FLOPs directly by a ratio of À since shift operation is FLOPs-free. More importantly, with the efficient CUDA implementation of shift operation, we can bring practical inference acceleration GPU-like devices.
4

---

Published in Transactions on Machine Learning Research (11/2022)
Input:
Right shift one pixel: Fr
Ria
F22
F34
F22
Fi- Fr
E50
F92
F10
F42
F42
F43
F41
F40
F41
F42
543
(a) Enhance high frequency information
(b) Enlarge the receptive field of filters
Figure 2: Benefits of shifting the features for SISR task.
3.1.1 Benefits of Shift for Super-resolution.
Super-resolution aims at recovering a high-resolution image from its corresponding low-resolution image. The enhanced high frequency information such as texture could be helpful for improving the quality of recovered high-resolution image (Zhang et al. 2018b Zhou al. 2018). Given input feature map F7, we shift Fr one pixel to the right across all channels, and pad the vacant position with zeros to get the shifted feature maps Fr. The texture information In convolution neural network, the vanilla features and the shifted features concatenated together to be processed by next layer's convolution operation. The convolution operation can be seen as more complex operation involving subtraction, which can enhance the high frequency information to some exlent.
be enhanced by F, Fr, shown in the left of Figure [
addition, the combination of two spatially dislocated feature maps can enlarge the receptive field of CNNs, which is critical for super-resolution task (Wang al. 2019; He et al., (201b)). In other words, the shift operation of feature maps provides spatial information communication of convolution filters. For instance, as demonstrated in the right of Figure 2 when shift feature map pixel to the left, the receptive field of the same location on the next layer's 's feature map shift one pixel to the left correspondingly. The convolution operation performed on the combination of dislocation feature maps results in a wider receptive field.
Last but not least, as opposed to the low arithmetic intensity (ratio of FLOPs to memory accesses) of depth- wise convolution, shift operation is more efficient in terms of practical speed, which will compared in detail in the experiments section.
3.2 Make the Shift Learnable
During the training process, the shift operation in Eq. [ can be implemented by special case of depth-wise convolution where only one weight and the others are Figure 3 gives an example to illustrate how the shift operation works. In order to flexibly adjust. the offset of intrinsic features during training, the offset weight W need to be learnable. However, the one-hot values in W make it difficult to optimize the weights. Therefore, the Gumbel-Softmaa trick (Jang et al., 20166) adopted for addressing this issue. The Gumbel-Softmax trick foed-forwards the one-hot signal and back-propagates the soft signal, which solves the non-derivableness of sampling from categorical distribution.
We create a proxy soft weight W' E ((2++1)x(2++1) for representing the inherent values of one-hot W. A noise E IR (2)++))x(2+))) is randomly sampled from the Gumbel distribution:
Ni. log( (og(U))));
33)
5

---

Published in Transactions on Machine Learning Research (11/2022)
0
0
0
0
0
Rio
Fbi
F22
paad 0
Ro
Rm
Rz
0
0
0
Fio
F11
F12
x
:depth wise convolution
Flo
F11
F12
F20
F21
F22
0
0
O
F11
F12
0
0
0
Pa
P22
0
0
1
Input
Weight
Output
Figure 3: An example of the shift operation during training, which is implemented by a special case of disth-wise convolution where only one weight and the others are
where U1J ~ 1) is sampled from uniform distribution. The one-hot weight W is relaxed as
e N.55
S(W Ei= E
(4)
( +N,g)//'
where T is the temperature to control the sharpness of the softmax function, and the function approximates the discrete categorical sampling. Then, we can obtain the values of offset indices
in ja arg max S(W").
During feed-forward process, the values of W can be computed process, through estimator is utilized, that is, the derivative aw" approximated calculated using the derivative of Eq.
Eq. aw
As for the back pock-proppgation
aw as(w') OW aW
(6)
Then, the trainable shift weight W and other trainable parameters be trained end-to-end. After training, pick the position of the maximum value the shift offset position and construct the inference graph of GhostSR networks.
3.3 Intrinsic Features in Pre-trained Model
In the proposed method, we first generate (1-))c, features intrinsic features, then use the shift operation to generate the other features ghost features based the intrinsic features, and finally concatenate the intrinsic and ghost features together as the complete output features. If we train a GhostSR model from scratch, the indices 0 and 02 in Eq. ! are set simply by order. If a pre-trained vanilla SR model is provided, we can utilize the relation intrinsic and ghost features for better performance. Since the goal of this work is to reduce the inference latency, parameters and FLOPs of SISR models, and some other works such pruning or quantization (Li et al. 2021 Han et al. 2016 also require pre-trained models, pre-training here is reasonable.
Given a pre-trained SR model, we aim to replace part of the convolution operations with shift operations for generating ghost features. However, for the output features of certain layer in network, it is not clear which part intrinsic and which part is ghost. We address this problem by clustering the filters in pre- trained models, and the features generated by the filter which nearest the cluster centroid are taken as intrinsic features. Specifically, the weights of all convolution filters are firstly vectorized from [co, C₁ s, Ca, Gj x s] and obtain weight vectors {ʃ1, f2, Sc.) Then the weight vectors are divided into (1 X)c.

---

Published in Transactions on Machine Learning Research (11/2022)
clusters G (G1, Gz.... G(1-A)eal Any pair of points in one cluster should be as close to each other possible:
(1-AIco Σ
min g
Σ
1
(7)
iEGx
where MM is the mean of points in cluster G2. We use clustering method k-means for clustering filters with the objective function in Eq. Y For the clusters which contain only one filter, we take this filter as intrinsic filter. For the clusters which contain multiple filters, the centroid may not really exist in the original weight kernels, so we select the filter which is ncarest to the centroid as intrinsic filter, and the features generated by the intrinsic filters are laken as intrinsic features. The index of intrinsic filters can be formulated as
E Gk. arg min || - 1x
if 1
1g
(8)
otherwise.
iEGa
The set of intrinsic indicos = {I,, I2.... A(_,),,)) whose corresponding filters are preserved as convolution operations, and the other filters are replaced by the proposed shift operations.
After finding the intrinsic filters in each layers of the pre-trained model, we assign the corresponding weight in pre-trained model to the intrinsic filters. Thus, we can maximally utilize the information in pre-trained model to identify intrinsic features and inherit pre-trained filters for better performance.
4 Experiments
In this section, we conduct extensive experiments non-compact and lightweight networks. The detailed quantitative and qualitative evaluations are provided to verify the effectiveness of the proposed method.
4.1
Experimental Settings
4.1.1
Datasets and Metrics.
To evaluate the performance of the method, following the setting of (Lim et al., 2017 Zhang et al. 20 2018c
Ahn et al. 2018), we use 800 images from DIV2K (Timofte et al.) (017]] dataset train our models. In order compare with other state-of-the-art methods, we report our result on four standard benchmark datasets: Sct5 (Bevilacqua et al. 2012). Sct14 (Zeyde et al. 2010)), B100 (Martin et al. 2001 and Urban100 (Huang et al. 2015). The LR images are generated by bicubic down-sampling and the SR results are evaluated by PSNR and SSIM (Wang et al. 2004) on Y channel of YCbCr space.
4.1.2
Training Details.
We use four famous SISR models CARN (Ahn et al. 2018 and IMDN (Hui et al.) 2019). These models have various numbers of parameters ranging from 0.69M to 43.71M (million). To maintain the performance of the models embedded in the proposed method, we do not replace the regular convolution in the first and the last layers in these net- works [Courbariaux et al. 2015 Zhou et al. 2016). and the point-wiee wise convolution is also kept unchanged if encountered in the middle layers. The detailed architectures are summarized the Appendix. In addition, unless otherwise specified, the ratio À of ghost features set to 0.5, the temperature 7 in Eq. set to 1, and the maximum offset. d in Eq. is set to
basclines: EDSR (Lim al. 2017 RDN (Zhang et al. 2018c).
During training, crop 16 images with 48x48 patch size from the LR images every card for training,
The input examples are augmented by random horizontal flipping and 90º rotating. In addition, all the images are pre-processed by subtracting the mean RGB value of the DIV2K dataset. To optimize the model, we use ADAM optimizer (Kingma & Ba 2014) with 31 = 0.9, 32 = 0.999, and e = 10-8, 8 We train EDSR and RDN for 300 epochs by singlo-scale training scheme, and train CARN and IMDN for 1200 epochs by multi-scale and singlo-scale training scheme respectively. The initial learning rate is set to 1c-4 for all models
and reduced by cosine learning rate decay (Zhao al. 2020 Kong al. 2021

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: Quantitative results of baseline convolutional networks and their Ghost versions for scaling factor x2, x3 and x4. We assume the HR image size to be 720p (1280x720) to calculate the FLOPs.
Params
FLOPs
Set5
Set14
B100
Urban100
Scale
Model
Type
(M)
(G)
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Raw
40.73
9389
38.22/0.9612
33.86/0.9201
32.34/0.9018
32.92/0.9356
EDSR
Ghost
21.85
5038
10.01/40.0000
t0.07/10.0003
10.00/10.0000
40.04/10.0004
Raw
19.27
4442
38 28.25/099614
33.97/0.9205
32.34/0.9017
32.90/0.9355
RDN
Ghost
9.83
2265
40.06/10.0002
J0.01 00.01/00.005
10.00/10.0000
10.10/10.0009
x2
Raw
1.59
223
37 08.8900
33.34/0.9173
32.14/0.8990
31.96/0.92
CARN
Ghost
1.19
130
J0.00/10.0000
J0.00/10.0002
J0.03/L0.0003
J0.09/10.0000
Raw
0.69
160
37 37.91/0.9595
33. 33.59/0.9170
32.15/0.8988
32.14/0.9275
IMDN
Ghost
0.40
40.05/40.0002
J0. 10.18/40.0008
40.05/40.0006
J0.23/40.0021
Raw
43.68
4471
34.67/0.9293
30.56/0.8466
29.26/0.8094
28. 28.78/0.8649
EDSR
Ghost
24.80
2541
J0.03/40.0004
10.06/10.0007
10.03/10.0005
10.09/10.0018
Raw 19.37
1981
34.69/0.9296
30.54/0.8467
29.26/0.8093
28. 28.79/0.8654
RDN
Ghost
9.92
1016
10.09/40.0009
40.04/40.0004
40.04/40.0008
J.O. L0.15/40.0030
x3
Raw
1.59
119
34.35/0.9266
30.32/0.8415
29.08 29.08/0.800 80.433
28.05/ 28.05/0.8448 8499
CARN
Ghost
1.19
77
J0.08/40.0004
J0.02/40.0005
J0.01/40.0004
40.08/40.0018
Raw
0.70
72
34.32/0.9260
30.31/0.8410
29.07/0.8037
28.15 28.15/0.8511
IMDN
Ghost
0.41
10.14/40.0012
40.13/40.0024
40.07/40.0013
L0.22/40.0044
Raw
43.09
2896
32.46/0.8984
28.81/0.7874
27.73/0.7416
26.60/0.8019
EDSR
Ghost
24.21
1808
10.04/40.0005
10.05/10.0010
J0.06/10.0011
40.12/10.0027
Raw
19.42
1146
32.49/0 32.49/0.8987
28. 28.83/0.7875
27.72/0.7416
26.59/0. 06.55/008003
RDN
Ghost
9.97
602
10.10/10.0011
10.10/10.0024
10.05/40.0020
40.21/40.0063
x4
Raw
1.59
91
32.16/0.8943
28.59/0.7810
27.58/0.7355
26.03/0. 26.03/0.7830
CARN
Ghost
1.19
J0.05/L0.0007
10.02/10.0008
10.02/10.0007
J0.05/40.002
Raw
0.72
41
32.19 / 0.8938
28.57/0.7805
27.54/0.7344
26.03/0.7831
IMDN
Ghost
0.42
24
10.13/10.0019
10.08/10.0021
10.06/40.0017
10.14/10.0044
Table 2: Average inference latency (ms) for Urban100 dataset with x2 scale on a single V100 GPU platform.
Model
EDSR(Original / GhostSR)
RDN(Original / GhostSR)
Latency
720.20 420.71
450.73 304.75
Model
CARiN(riggnaal GhostSR)
IMDN((riginal / GhostSR)
Latency
49.52 37.26
35.05 / 25.64
4.2 Comparison with Baselines
4.2.1
Quantitative Evaluation.
In Table we report the quantitative results of the bascline convolutional networks and the proposed GhostSR versions for scaling factor x2, x3 and x4. The results raw type are obtained by our re-training, which are close to the results the original papers. The detailed results of our re-training and the results in the original papers are reported in the Appendix. In addition, following the experimental sottings IMDN (Hu et al. 2019), we report the practical average inference latency in Table 2 From the results that both the non-compact (EDSR and RDN) and lightweight (CARN and
Table and Table

---

Published in Transactions on Machine Learning Research (11/2022)
IMDN) SISR models embedded in the GhostSR module achieve comparable performance to that of their baselines with large reduction of parameters, FLOPSs and GPU inference latency. For instance, we reduce the FLOPs of x2 EDSR, x2 RDN, x2 CARN and x2 IMDN by 46%, 49%, 42% and 43%, respectively, with little performance loss. Similarly, for different networks, the number of parameters and inference latency are also greatly reduced. We attribute prior information in pro-trained models. Urban100 dataset contains 100 images of urban scenes, which is more challenging than other datasets. Therefore, slightly higher loss of PSNR is acceptable here.
the superiority of learnable shift and the clustering excavation of
HR (PSNR, SSIM)
RDN (23.37, 0.5766)
CARN (23.11, 0.5636)
(23.35, 0.5761)
(23.09, 0.5630)
Set14 (x4): baboon
Bicubic (22.44, 0.4528)
EDSR Chost (23.32, 0.5755)
RDN Ghost (23.35, 0.5764)
CARN Ghost (23.08, 0.5630)
IMDN Ghost (23.05, 0.5622)
MMDN_Ghost
HR. (PSNR, SSIM)
EDSR (22.38, 0.7800)
RDN (22.35, 0.7793)
CARN (21.67, 0.7042)
IMDN (21.90, 0.7573)
Urban100 (x4): i47_ 047
Bicubic (20.01, 0.6281)
RDN LDDN_Ghost (22.21, 0.7764)
CARN_Ghost Ghost (21.64, 0.7035)
IMDN OMDN_Ghost (21.83, 0.7562)
(22.27, 0.7789)
Figure 4: Visual comparisons for x4 images on Sct14 and Urban100 datasots. For the shown examples, the details and textures generated by shift operation are approximately the same those by regular convolution.
4.2.2 Qualitative Evaluation.
The qualitative evaluations on various datasets are shown in Figure ! We choose the most challenging x4 task to reveal the difference visual quality betwoen the original network and the GhostSR versions. From Figure 4 we can see that for both the non-compact and weight networks, the details and textures generated by GhostSR are basically the same as those by original network.
4.3 Comparison with State-of-
We compare the proposed GhostSR models with the state-of-the-arts including manual-designed efficient SISR. methods (VDSR Kim et al. 2016a CARN (Ah et al. 2018 CARN M (Ahn al.
2018
PAN (Zhao al. 2020), SelNet (Choi & Kim 2017 OISR (He et al. 2019a), BTSRN (Fan al.
2017
LapSRN [Lai et al. 2017 and NAS-based (ncural architecture scarch) efficient SISR methods (ESRN
(Song
et al. 2020), FALSR CChu et al. 2019), MoreMNAS (Chu et al., 2020 From the results in Figure 5 we can
see that the GhostSR models achieves a comparable PSNR with less FLOPs and inference latency. Since the
attention operation PAN (Zhao al. 2020) is very time-consuming, our GhostSR-IMDDS and Ghost
CARN faster. In addition, PAN achieves higher PSNR with the of additional Flickr2K [Lim al. 2017) training data, which is not used in method.

---

Published in Transactions on Machine Learning Research (11/2022)
38.0
38.0
ESRN-F
OISR
PAN
IMDN
IMDN
37,9
-3%
37.9
-25%
CARN
GlostsR
GhosSR
CARN
GhestSR
Ghost5R
PSNR
37.8
-MDN
CARNN
0.5M 1M
1.5M
PSNR
37.8
-IMDN
-CARN
BTSRN
HTSRN
37.7
BALSR-C
37.7
MoMMNAS-A
37.6
37.6
LapSRN
CARN
LagSRN VDSR
CARN
37.5
37.5
100
200
300
400
25
30
35
40
45
FLOPs (G)
Latency (ms)
Figure 5: PSNR U.8. FLOPs and PSNR n.s. Lateney. The PSNR is calculated on Set5 dataset with x2 scale, and the FLOPs is calculated with a 640 x 360 image. The average inference latency is calculated on single V100 GPU using Urban100 dataset with x2 scale.
Table Comparison of different. x3 CARN network under similar FL.OPs budget. The FLOPs is calculated with 426 240 image, and the average inference latency is calculated on single V100 GPU using Urban100 dataset with x3 scale. The best results in bold.
Params
FLOPs
Latency
Set5
Set.14
B100
Urban100
Type
(M)
(G)
(ms)
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Original
1.59
119
32.09
$4.35/0.9266
30.32/0.8415
29.08/0.8043
28. 28.05 /0.8999
Conv-0.80x
1.15
79
29.33
34.11 34.11/0.9242
30.13/0.8390
28.89/0.88
27.77 27.77/0.8466 /0.8460
C-SGD
1.15
79
29.05
34.15/0.922 /0.948
30.16/0.8395
28.92/0.800
27.84/0.8467
HRank
1.19
78
28.56
34.16 34.16/0.9255
00.9551
30.14/0.888
28.95 00.802
27.86/0.847/
GhostSR
1.19
77
23.28
34.27 34.27 /0.9262
30.30/ 0..410
10.30/0.8410
29.07 / 80.9
27.97/0.848
4.4
Comparison with Other Compression Methods
4.4.1
Comparison with Pruning Methods.
Pruning is a classical lype of model compression methods which aims to cut out the unimportant channels. Unlike image recognition that requires the high-level coarse feature, the detailed difference between feature maps is important for super-rosolution task. Our GhostSR does not remove any channel and ro-generate the ghost features by the learnable shift. Thus, compare GhostSR with the representative network pruming methods including C-SGD (Ding et al. 2019) and HRank (Lin et al. 2020). Tn table 3 we report the quantitative results of different methods of reducing FLOPs for x3 CARN network. Conv-0.80x means to directly reduce the mumber of channels in each layer of the network to 0.80 times the original number. We re-implement the C-SGD (Ding et al. 2019) and HRank (Lin et al.) 2020) and apply them for the STSR task. CARN itsclf is a lightweight network, and directly reducing the width of each layer of the network will bring greater performance drop. For instance, compared with the original CARN, the PSNR of Conv-0.80x is reduced by 0.28dB for x3 scale Urban100 dataset. When the pruning methods (C-SGD and HRank) are adopted, the PSNR/SSIM is improved significantly. However, there are still some gaps with the GhostSR method. For example, GhostSR exceeds HRank by 0.1 on Urban100 dataset with a faster inference speed.
Figure 0 visualizes the images generated by different CARN. The visual quality of the images generated by reducing the number of channels is obviously degraded, and the GhostSR maintains the visual quality of images. We aftribute it to the characteristies of shift operation, which can compensate for the performance
10

---

Published in Transactions on Machine Learning Research (11/2022)
degradation caused by the reduction in channels. For example, the left side of the enlarged area of img 008 generated by Conv-0.80x or C-SGD Ding et al. (2019) is blurred compared to that of GhostSR method.
HR
CARN
Conv-0.808 0.80x
HR
CARN
Conv-O.88 0.0v-0000x
Set14 (x3): zebra
Urban 00 (x3)) img. 008
SGD
Depthwise
G hostSR
C-SGD
Depthwise
G RostsR
Figure 6: Visual comparisons of different. model compression methods for x3 CARN.
4.4.2 Comparison with Depthwise Convolution.
In addition, we also compare the results of replacing the shift operation in our method with the depth-wise convolution, which denoted by Depthwise in Table 4 The depth-wise convolution utilized in GhostNet to build efficient models in classification task. When the shift operation in GhostSR model is replaced with depth-wise convolution, there is slight performance boost due to the introduction of a few more FLOPs. The visual quality between shift and depth-wise is similar in Figure G However, the average inference latency of depth-wise operation is greatly increased since it is time-consuming on GPU-like devices due to its fragmented memory footprints. In particular, the latency of Depthwise model is 31 1.6mms which is 8.28ms more than GhostSR method.
Table 4: Comparison of shift and depth-wise operations x3 CARN network.
FLOPs
Latency
Set5
Set14
B100
Urban100
Method
(G)
(ms)
PSNR/SSIM
PSNR/SSIM PSNR/SSIM PSNR/SSIM
Original
119
32.09
34.35
0.9266
30.32
0.8415
29.08/0.8043
28.05/0.8
Depthwise
80
31.56
34.27
0.9263
30.31
0.8410 29.09/0.8042 28.00/0.8485
GhostSR
77
23.28
34.27
0.9262
30.30
0.8410
29.07
29.07/0.8033 00.809
27.97 27.97/0.8480 00.8480
4.5
Ablation Study
4.5.1 Analysis on the Sub-Mar MobbMMd in GhostSR.
We first conduct series of comparison experiments to analyze the effects of sub-modules in GhostSR, and the results are reported in Table For the option that does not the learnable shift scheme, replace the shift operation with simple copy operation, that is, the ghost features are obtained directly by copying the intrinsic features. For the option that does not use the clustered information in pre-trained models, we train the models from scratch. When neither learnable shift scheme nor the clustering based on pre-trained models is used, the PSNR of GhostSR drops by 0.24dB on Urban100 dataset. When only the clustering basod on pre-trained models the learnable shift scheme used, the PSNR drops by 0.13dB and 0.08dB Urban 100 dataset, respectively.
Figure 7 visualizes the features generated at the same layer for three versions of CARN: copy operation, learnable shift operation and regular convolution operation. The copy version and the learnable shift version of CARN are both trained based on the pre-trained model using the fonementiioed clustering procedure. The features img_0000m 001 and lenna are generated in the first and third residual- block CARN, respectively. From Figure 7 the learnable shift operation extracts textures similar that of the regular convolution, and more than the copy operation. For example, the edge and texture of features in lenna generated by shift are clearer and richer than those by copy.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Table 5: Analysis on the sub-modules in GhostSR for x3 CARN network.
Architecture
Set5
Set14
B100
Urban100
Learnable Shift
Clustering (Fq.)
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM.
PSNR/SSIM
Exq.
5
7
x
34.04/0.9241
30.06/0.8378
28.80/0.8012
27. 27.73/0.8439
x
34.17/0.922
30.23/0.8399
28.96/0.8023
27. 27.84/0.8463
34.20/0.9255
00.21/0.8397
00.0000.8031
27.89 97.89/0.8471
34.27 /0.9262
30.30 30.30/0.8410
29. 29.07 80.9
27. 27.97 27.97/0.8480 00.4400
Urban 00:
img_00 001
Set14: lenna
Copy
Shift
Conv
Visualization of features generated at the same layer by different CARN. The learnable shift operation extracts more textures than copy.
Figure 7:
4.5.2 Analysis on the Ratio of Ghost Features.
We also conduct the experiments to analyze the effect of different ratio À of ghost features, and the results are reported in Table G A higher means more ghost features are generated by the shift operation. Since the number of features in each layer is fixed, the more ghost features are generated by shift operation, the greater the reduction in parameters, FLOPs and latency, and the slight negative impact on PSNR/SSIL is inevitable. Notably, when the ratio equals 0.25, we achieve a higher PSNR on all four datasets with fewer parameters, FLOPs and latency.
Table 6: Analysis on the ratio of ghost features for x3 CARN network.
Params FLOPs
Latency
Set5
Set14
B100
Urban100
(M)
(G)
(ms)
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
0.00
1.59
119
32.67
34.35/0.9266
30.32/0.8415
29.08/0.8045
28.05/0.84999
0.25
1.33
96
29.22
10.04/10.0003
10.05/10.0008
10.01/10.0000
10.04/10.0006
0.50
1.19
77
26.14
L0.08/L0.000
10.02/10.0005
10.01/10.000
10.08/L0.0019
0.75
1.01
60
22.54
00.15/40.0021
10.08/10.0016
10.10/40.0018
40.23/10.0047
12

---

Published in Transactions on Machine Learning Research (11/2022)
5 Conclusion
This paper proposes GhostSR method for efficient SISR models. We first study the feature redundancy in convolutional layers and introduce a learnable shift operalion to replace a part of convent filters for generating the ghost features in a cheap way. Then, we present a procedure of clustering pre-trained models to select the intrinsic filters for generating intrinsic features. Thus, we can effectively learn and identify the intrinsic and ghost features simultaneously. The final output features are construcled by concatenating the intrinsic and ghost features together. We empirically analyze the benefits of shift operation on SISR task and bring practical inference acceleration on GPUs. Extensive experiments demonstrate that both the non- compact and lightweight SISR models embedded in the GhostSR, method can achieve comparable quantitative result and qualitative quality to that of their baselines with large large reduction of parameters, FL.OPs and GPU inference latency.
6
Acknowledgement
We gratefully acknowledge the support of MindSpore (Huawei 2020 CANN(Compute Architecture for Neural Networks) and Ascend AT Processor used for this research.
References
Nambyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV, pp. 252-268, 2018.
Marco Bevilacqua, Aline Rouny, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single- image super-resolution based on nonnegative neighbor embedding. 2012.
Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. Spatial-adaptive network for single image denoising. In European Conference on Computer Vision, pp. 171 187. Springer, 2020.
Weijie Chen, Di Xie, Yuan Zhang, and Shiliang Pu. All you need is few shifts: Designing efficient convolutional neural networks for image classification. In CVPR, pp. 7241-7250 14410,25, 2019.
Jac-Seok Choi and Munchurl Kim. A deep convolutional neural network with selection units for super- resolution. In CVPR workshops, pp. 154 160, 2017.
Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. arXie preprint 1rXiv:1901.02661 2019.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Multi-objective reinforced evolution in mobile neural architecture scarch. ECCV, pp. 99-113. Springer, 2020.
Matthieu Courbariaux, Yoshua. Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural net- works with binary weights during propagations. In NewrIPS, pp. 3123 3131, 2015.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In CVPR, 2019.
Chao Dong, Chen Change Loy, Kaiming Hc, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV. Springer, 2014.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In ECCY, pp. 391 407. Springer, 2016.
Yuchen Fan, Honghui Shi, Jiahui Yu, Ding Liu, Wei Han, Haichao Yu, Zhangyang Wang, Xinchao Wang, and Thomas S Huang. Balanced two-stage residual networks for image super-resolution. In CVPR Workshops, pp. 161-168 2017.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Qinquan Gao, Yan Zhao, Gen Li, and Tong Tong. Image super-resolution using knowledge distillation. In ACCV, pp. 527-541. Springer, 2018.
Yong Guo, Yongsheng Luo, Zhenhao He, Jin Huang, and Jian Chen. Hierarchical neural architecture search for single image super-resolution. arXiv preprint arXit:2003.004 04619, 2020.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, pp. 1580-1589, 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn CVPR, pp. 770 778, 2016.
Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng, Ode-inspired network design for single image super-resohtion. In CVPR, pp. 1732-1741, 2019a.
Zewei He, Yanpeng Cao, Lei Du, Baobei Xu, Jiangxin Yang, Yanlong Cao, Siliang Tang, and Yucting Zhuang. Mrfn: Multi-receetiive-fele network for fast and accurate single image super-resolution. IEEE Transactions on Multimedia, 22(4):1042 1054, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.088 025331, 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint orXiv:1704.00 04861, 2017.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pp. 5197 5206, 2015.
Huawei. Mindspore.
cn/ 2020.
Zheng Hui, Xiumei Wang, and Xinbo Gao. Fast and accurate single image super-resolution via information distillation network. In CVPR, pp. 723-731, 2018.
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with informa- tion multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia. pp. 2024 2032, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiin with gumbel-softmax. arXiv preprinl arXiv:1611.01144 2016.
Yunho Jeon and Junmo Kim. Constructing fast network through deconstruction of convolution. In NewrIPS, pp. 5951-5961, 2018.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lec. Accurate image super-resolution using very deep convo- lutional networks. In CVPR, 2016a.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super- resolution. In CVPR, pp. 1637-1645, 2016b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.696 2014.
arXio preprint
Xiangtao Kong, Hengyuan Zhao, Yu Qiao, and Chao Dong. Classsr: A general framework accelerate super- resolution networks by data characteristic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12016 12016-12025, 2021.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuni Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, pp. (24-632, 2017.
Royson Lee, Lukasz Dudziak, Mohamed Abdelfattah, Stylianos I Venieris, Hyeji Kim, Hongkai Wen, and Nicholas D Lane. Journey towards tiny perceptual super-resoltton. arXiv preprint arXiv:2007.0p 04856, 2020a.
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolttion. European Conference on Computer Vision, pp. 465-482. Springer, 2020b.
Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for elficient convnets. In ICLR, 2017.
Yawei Li, Kai Zhang, Radu Timofte, Luc Van Gool, Fangyuan Kong, Mingxi Li, Songwei Liu, Zongcai Du, Ding Liu, Chenhui Zhou, et al. Ntire 2022 challenge on elficient super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognátion, pp. 1062-1102, 2022.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breoq: Pushing the limit of post-training quantization by block reconstruction. ar preprinl arXiv:2102.054 054.2, 2021.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Im- age restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833-1844 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. CVPR workshops, 2017.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529 1538, 2020.
Chuanjian Liu, Kai Han, An Xiao, Ying Nie, Wei Zhang, and Yunhe Wang. Network amplification with efficient macs allocation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1933-1942, 2022.
Xiaotong Luo, Yuan Xie, Yulun Zhang, Yanyun Qu, Cuihua Li, and Yun Fu. Latticenet: Towards lightweight image super-resolution with lattice block. In European Conference on Computer Vision, pp. 272-289. Springer, 2020.
Yinglan Ma, Hongyu Xiong, Zhe Hu, and Lizhuang Ma. Efficient super resolution using binarized neural network. In CVPR Workshops, 2019.
Salma Abdel Magid, Yulun Zhang, Donglai Wei, Won-Dong Jang, Zudi Lin, Yun Fu, and Hanspeter Pfister. Dynamic high-pass filtering and multi-spectral attention for image resoutteso.utio.. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4288-4297 2021.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume pp. 416 423. IEEE, 2001.
Abdul Mugeet, Jiwon Hwang, Subin Yang, Jung Heum Kang, Yongwoo Kim, and Sung-Ho Bae. Ultra lightweight image super-resolution with multi-attention layers. arXiv preprint arXto:2088.1290 2020.
Ying Nie, Kai Han, and Yunhe Wang. Multi-bit adaptive distillation for binary neural networks
15

---

Published in Transactions on Machine Learning Research (11/2022)
Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via holistic attention network. In European conference on computer vision, pp. 191 207. Springer, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chie Chig- Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 5510-4520, 2018.
Dehua Song, Chang Xu, Xu Jia, Yiyi Chen, Chunjing Xu, and Yunhe Wang. Efficient residual dense block search for image super-resolution In AAAT, pp- 12007-12014, 2020.
Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolttion via deep recursive residual network. In CVPR, pp. 3147-3155, 2017a.
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In ICCV, pp. 1539-4547 2017b.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuaa Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPR workshops, pp. 114-125, 2017.
Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. Explor- ing sparsity in image super-resolution for efficient inference. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4917-4926, 2021.
Ruxin Wang, Mingming Gong, and Dacheng Tao. Receptive field size versus model depth for single image super-resolution. IEEE Transactions on Image Processing, 29:1669-1682, 2019.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600) 2004.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative spatial convolutions, In CVPR, pp. 9127 9135, 2018.
Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. Learning frequency-awary dynamic network for efficient super-resolution. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4308 4317, 2021.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Tåu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. Advances in Neural Information Processing Systems, 33:2771-2783 2020.
Haoran You, Baopu Li, Shi Huihong, Yonggan Fu, and Yingyan Lin. Shiftaddnas: Hardware-inspired search for more accurate and efficient neural networks. In International Conference on Machine Learning, pp. 15566-25580 PMLR, 2022.
Pengcheng Yuan, Shufei Lin, Cheng Cui, Yuning Du, Ruoyu Guo, Dongliang He, Errui Ding, and Shumin Han. Hs-rosnct: Hierarchical-spiit block on convolutional noural network. arXiv preprint orXiv:2010.07621. 2020.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparsc-reprresenatiion.. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kacli, Bin Ren, al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEE/CWS International Conference on Computer Vision, pp. 2021.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflene: An extremely efficient convolutional neural network for mobile devices. In CVPR, pp- 6848 6856, 2018a.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu, Image super-resolution using very deep residual channel attention networks. In ECCV, 2018b.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018c.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. Advances in Neural Information Processing Systems, 34:2695 2706, 2021a.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Learning efficient image super-resolution networks via struccuur--rennlariied pruning. In International Conference Learning Representations, 2021b.
Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. Efficient image super-resolution using pixel atiention. Xiviv preprint orXii:2010.0109 2020.
Fuqiang Zhou, Xiaojie Li, and Zuoxin Li. High- figh-frequency details enhancing densenet for super-rosolution. Newrocompuling, 290:34-42, 42, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.0611 1606..666... 2016.
17

---

Published in Transactions on Machine Learning Research (11/2022)
A Appendix
A.1 Detailed Network Architectures
As we mentioned in the main body, to improve the performance of the models equipped with GhostSR, we do not replace the regular convolution in the first and the last layers in these networks, and the point-wise convolution is also kept unchanged if encountered in the middle layers. We take the x2 scale as an example and the modified network architectures are summarized in Table 0 Table g Table g and Table T0 Where (cin, cout, k, k) in layer size represents the input channel, output channel and kernel size of the convolutional layer are cin, cout and k x k, respectively. [c1_ _onv, c2 [__ghsst] in type represents the output channel of conventional convolution operation and ghost operation in GhostSR method are el and c2, respectively. Note that when c2 is equal to 0, the convolutional layer only involves the conventional convolution operation.
Table 7: The detailed archilecture of x2 EDSR (Lim et al.
2017 equipped with GhostSR.
layer name
layer size
type
head
(3, 256, 3, 3)
[256_cony 0 ghost]
ResBlock] x 32
(256, 256, 3, 3)
[128 conv, 128 ghost]
tail.upse
(256, 1024, 3, 3)
11044 cony, 0 ghost]
tail.conv
(256, 3, 3, 3)
B conv, ghost.
Table 8: The detailed architecture of x2 RDN (Zhang et al.
2018ch equipped with GhostSR.
layer name
layer size
type
SFENet1
(3, 64, 3, 3)
664 _cnov,, 0 ghost]
SFENet2
(64, 64, 3, 3)
3__cov, 32 ghost]
(64 x i, 64, 3, 3)
ResDenscBlock] x 16
[32 conv, 32 ghost)
i [1, 2, ... 8]
GFF.convO
(1024, 64, 1, 1)
(64_conv- 0 ghost]
.covv
(64, 64, 3, 3)
[64 _on_, 0 0_oost
UPNet.conv0
(64, 256, 3, 3)
2556 conv, 0_ ghost]
UPNet.convl
(64, 3, 3, 3)
[ conv, 0 ghost]
Table 9: The detailed architecture of x2 CARN (Ahn et al. 2018) equipped with GhostSR.
layer_name name
layer size
type
conv in
(3, 64, 3, 3)
664_ conv, 0 ghost]
Block1
(64, 64, 3, 3)
332 _o_v, 32 _2__host]]
C1
(128, 64, 1, 1)
[64 conv, 0_ 0ghost
Block2
(64, 64, 3, 3)
332 conv, 32 _
C2
(192, 64, 1, 1)
[64 conv, 0. ghost)
Block3
(64, 64, 3, 3)
[32 _on_, 32_ _2__tostt]
C3
(256, 64, 1, 1)
con_ ghost
upconv
(64, 256, 3, 3)
2256 conv, ghost]
conv_out
(64,3,3,3)
3 conv, 0 ghost]
18

---

Published in Transactions on Machine Learning Research (11/2022)
Table The detailed architecture of x2 IMDN (Hui et al. 2019 equipped with GhostSR.
layer name
layer size
lype
convl
(3, 64, 3, 3)
64 conv, ghost]
C1 (64, 64, 3, 3)
32 conv, 32 ghost]
C2 (48, 64, 3)
32 conv, 32 ghost]
C3 (48, 64, 3, 3)
[32 conv, 32 ghost]
IMDModule] x 6
C4 (48, 16, 3, 3)
conv, ghost]
C5 (64, 64, 1, 1)
664 conv, 0 ghost]
(384, 64, 1, 1)
[64 conv, ghost]
conv2
(64, 64, 3, 3)
[64 cony, ghost
upsampler
(64, 12, 3, 3)
[12 conv, ghost
-0.75
-0.60
-0.45
-0.30
-0.15
Figure 8: Visualization of the normalized depthwise filters.
A.2 Detailed Results of the Baseline Convolutional Networks
For fair comparison, report the results of the baseline convolutional networks in Table 11 including the results of our re-trained and the results in the original papers. The results of our re-trained are close to the results in the original papers.
A.3 Generalibity on Other Low-Level Tasks
In general, the proposed method is also applicable to other non-SISR low-level tasks. We conduct experi- ments on the SADNet (Chang et al., 2020) for the single image denoising task. Specifically, only the 3 x 3 conventional convolution filters are replaced with the Ghost version, and the convolution filters in the first and last layers are also kept unchanged. The results on the synthetic color BSD68 dataset with gaussian noise are reported in Table [2] The FLOPs are computed based on 480 x 320 color images. From the results in Table 12, the Ghost version significantly reduces parameters and FLOPs of original SADNet by 1.3M and 14.4G, respectively, while the PSNR is slightly reduced by 0.09 dB.
30
A.4 Visualization of Depthwise Filters
In Table 4 compare the results of replacing the shift operation in the proposed GhostSR method with the depthwise convolution. From Figure the output features (O in short) obtained by concating the intrinsic features (I in short) and ghost features (G in short) together, that is, O = [I, G7. For the shift operation, G = =hhif(()), and for the depthwise convolution operation (DWConv in short), , G = DW DWCone()). To compare the differences between shift and depthwise convolution, the learned depthwise convolution filters in the first GhostSR layer are randomly selected for visualization. Specifically, for filter W 3x33 every element w of W is first normalized by wl/maz([W]]] and the normalized depthwise convolution filters are visualized in Figure g From Figure
we that some filters (marked with green rectangle) very
19

---

Published in Transactions on Machine Learning Research (11/2022)
Table 11:
The detailed results of the baseline convolutional networks for scaling factor x2, x3 and x4.
Sct5
Sct14
B100
Urban100
Scale
Model
Type
PSNR//SIN
PSNR//SIL
PSNR/SSIM
PSNR/SSIM
Ours
38.22
0.9612
33.86
0.9201
32.34 / 0.9018
32.92 / 0.9356
EDSR
Paper
38.11 0.9601
33.92
0.9195
32.32 / 0.9013
32.93 / 0.9351
Ours
38.25
0.9614
33.97
0.9205
32.34 / 0.9017
32.90 / 0.9355
RDN
Paper
38.24
0.9614
34.01 / 0.9212
32.34 / 0.9017
32.89 / 0.9353
x2
Ours
37.88
0.9601
33.54 / 0.9173
32.14 / 0.8990
31.96 / 0.9267
CARN
Paper
37.76
0.9590
33.52
0.9166
32.09 / 0.8978
31.92 / 0.9256
Ours
37.91
0.9595
33.59
0.9170
32.15 / 0.8988
32.14 / 0.9275
IMDN
Paper
38.00
0.9605
33.63
0.9177
32.19 / 0.8996
32.17 / 0.9283
Ours
34.67
0.9293
30.56
0.8466
29.26 / 0.8094
28.78 / 0.8649
EDSR
Paper
34.65
0.9282
30.52
0.8462
29.25 / 0.8093
28.80 / 0.8653
Ours
34.69
0.9296
30.54
0.8467
29.26 / 0.8093
28.79 / 0.8654
RDN
Paper
34.71
0.9296
30.57
0.8468
29.26 / 0.8093
28.80 / 0.8653
x3
Ours
34.35
0.9266
30.32 / 0.8415
29.08 / 0.8043
28.05 / 0.8499
CARN
Paper
34.29
0.9255
30.29
0.8407
29.06 / 0.8034
28.06 / 0.8493
Ours
34.32
0.9260
30.31
0.8410
29.07 / 0.8037
28.15 / 0.8511
IMDN
Paper
0.9270 34.36
30.32 / 0.8417
29.09 / 0.8046
28.17 / 0.8519
Ours
32.46
0.8984
28.81
0.7874
27.73 / 0.7416
26.60 / 0.8019
EDSR
Paper
32.46
0.8968
28.80
0.7876
27.71 / 0.7420
26.64 / 0.8033
Ours
32.49
0.8987
28.83 / 0.7875
27.72 / 0.7416
26.59 / 0.8023
RDN
Paper
32.47
0.8990
28.81
0.7871
27.72 / 0.7419
26.61 / 0.8028
x4
Ours
32.16
0.8943
28.59 / 0.7810
27.58 / 0.7355
26.03 / 0.7830
CARN
Paper
32.13
0.8937
28.60 / 0.7806
27.58 / 0.7349
26.07 / 0.7837
Ours
32.19
0.8938
28.57
0.7805
27.54 / 0.7344
26.03 / 0.7831
IMDN
Paper
32.21
0.8948
28.58
0.7811
27.56
/ 0.7353
26.04 / 0.7838
Table 12: Comparison of the baseline SADNet (Chang et al.
2020 and the Ghost version.
Model
Params (M)
FLOPs (G)
PSNR (dB)
Original
4.3
50.1
30.64
Ghost 3.0
35.7
30.55
similar to shift operation, that is, only one value of 3x3 is equal and the others are significantly low, even close to 0.
A.5 Comparison with SwinIR
SwinlR Liang et al. 2021 is the representative work based transformers, which achieves impressive results many low-level vision tasks. The detailed comparisons are reported in Table 13 The average inference latency is calculated on Urban100 dataset with x4 scale on a single V100 GPU platform. From Table [3] compared with other models, SwinIR achieves the highest PSNRs on four datasets, however, SwinIR also has the highest latency at an astounding 153.4ms. The ultra-high latency limits the deployment of SwinIR in fal-time inference scenarios. The PSNRs of the 4x wider Ghost IMDN model are comparable SwinIR, however, the latency is only one-third of Rwnn
20

---

Published in Transactions on Machine Learning Research (11/2022)
Table 13: Comparison with SwinIR Liang et al.
2021
withx4 4 scale.
Params
FLOPs
Latency
Set.
Set14
B100
Urban100
Model
(M)
(G)
(ms)
PSNR
PSNR
PSNR
PSNR
SwinIR
0.90
49.6
153.4
32.44
28.77
27.69
26.47
IMDN
0.72
40.9
8.6
32.19
28.57
27.54
26.03
Ghost. IMDN
0.42
23.9
6.9
32.06
28.49
27.48
25.89
Ghost. IMDN (xd)
6.47
370.5
50.12
32.40
28.71
27.61
26.38
A.6 More Visual Results
In Figure more results are displayed to reveal the difference of visual quality between the original network and the GhostSR versions. For both the non-compact and lightweight networks, the details and textures generated by GhostSR are basically the same as those by original network.
21

---

Published in Transactions on Machine Learning Research (11/2022)
HR
EDSR
RDN
CARN
IMDN
Sct5: baby (x4)
Bicubic
EDSR GhostSR
RDN ChostSR
CARN GhostSR
IMDN GhostSR
HR
EDSR
RDN
CARN
IMDN
Bicubic
HDN_GhhsttTi
CARN SRo
B100: 69015 (x4)
EDSR
RDN
CARN
IMDN
Urban100: 006
006 (x4)
Bicubic
EDSR GhostSR
RDN GhostSR
CARN GhostSR
IMDN GhostSR
HR
EDSR
RDN
CARN
IMDN
Urban100: img_081 (x4)
Bicubic:
POSSRGGhostSR
GhostRh
CARN_GhostSR
TMDN_GhosttST
Visual comparisons for images on various datasets. For the shown examples, the details generated by ghost operation are approximately the same those by conventional convolution.
Figure
22


Reviews:
[{"review_id": "UtCbav9y4N", "reviewer": "Reviewer_mJsT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ The idea of creating redundant features as augmentation of the original model is pretty interesting. Especially in SISR, the high-frequency features are crucial and can be enhanced by ghost features. Therefore, the motivation is strong.\n\n+ The experimental results are competitive in efficiency for both model size compression and inference speed acceleration.\n\n+ This proposed method is agnostic to different pre-trained models based on CNN.\n\n+ The whole paper is well written, with clear logic to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper presents a new method for efficient single image super-resolution with the help of redundant features referred to as ghost features generated by the shift operation. To make the ghost features informative, the authors proposed the learnable shift operation based on Gumbel-softmax for diffusional computation. The original pre-trained model has been compressed by the clustering of filters, with the centroid filters preserved. The final model includes both intrinsic and ghost features for joint representation learning. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Cons:\n\n- The major concern is the novelty since GhostNet has already been proposed for high-level vision tasks, which makes this work incremental (like a simple extension for low-high vision tasks). The discussion of the difference between GhostNet and this work is not clear enough. On page 2, I am still confused why the “so-called cheap operation is not cheap”? Could the authors explain more about this point?\n- The minor concern is the visual effectiveness since I have not seen the significant improvements in Fig 4. More visual examples are welcome to be included.\n- Some important refs in the efficient SISR area are missing [1-3].\n\n[1] Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning. ICLR 21.\n\n[2] Aligned Structured Sparsity Learning for Efficient Image Super-Resolution. NeurIPS 21.\n\n[3] NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results. CVPRW 22.\n\n- K-means clustering for filters is not very convincing for me. The authors are suggested to discuss more of why choosing K-means and why it works.\n\nOthers:\n\nIn Eq. 4, why W^{'}{i,j} + N{i,j}?\n\nDuplicate references for the paper “Towards lightweight image super-resolution with lattice block”.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WEPUGw2U8V", "reviewer": "Reviewer_XCf9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a sound approach applied in the context of SISR, identifying a fast and simple operation that can replace more expensive multiply-add operations that learn similar features anyway due to low-level nature of the SISR task.  For the most part it is clearly described.  The performance of the method is robustly demonstrated with evaluations on multiple SISR methods and datasets.\n\nI have a few questions, which I think could strengthen the paper further:\n\n\n- Although the paper discusses feature redundancy, I don't see any measurements or explicit investigation of this.  Better establishing there is indeed shift redundancy before using this method would more precisely confirm the motivation.  A few possible measures:  How close are filters or feature maps to each other, taking minimum over all kxk shift offsets?  Or, if a SISR network uses depthwise-channelwise decomposition of a conv layer, how many of the depthwise conv filters are close to shifted copies of another filter?\n\n- What is offset range d used for W  (I did not see it in the text or appendix, but may have missed it) \n\n- How are offset weights W initialized?  For pretrained model filter clustering init: Are the shift weights W also initialized finding the shifts that best approximate the other (unselected) filters in each cluster from the selected one, using an initial greedy search over all shifts?  Or are they random?\n\n- 4.4.2:  this is an interesting evaluation, I would also be interested in seeing what the depthwise conv filters look like --- are a lot of them close to point shifts?\n\n\n- 4.5.1:  \"we replace the shift operation by simple copy\":  does this mean the features are copied directly with no shift at all (so they are the same features at the same offsets), or that there are shifts but these are not learned?  If there is no shift, how does this compare to removing the features entirely?  What is the next downstream op, and can it benefit from unshifted replicated features (if it's a linear conv layer, I don't see how it would benefit from copies).\n\n\n- While clear, the English writing is rough in several places, particularly in the abstract, which does not read very fluidly.  There are some unusual word choices and number/tense agreements that are off here, too.  This is not a major concern, as both the main body and abstract are still clear.  However, I would recommend trying to edit the abstract over again keeping this in mind.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to make single image super-resolution models more efficient, by replacing a subset of convolution filters with discrete shift operations.  Observing that intermediate layer feature maps can be very similar, after accounting for shifts this redundancy can be leveraged by reducing the number of convolution operations in favor of shifted replication.  The (x,y) shift amounts for each filter are learned using Gumbel-softmax on a dxd matrix of possible shifts.  To decide how many shifts to allow for each filter, two options are explored: (1) train from scratch with one shift for each filter (in lambda=0.5 case), or (2) cluster the learned filters in a pretrained SISR model, and assign as many shifts to each intrinsic filter as there elements in that filter's cluster.  Both methods arrive at good performance, with preinitializing better.  The method is evaluated on four datasets, incorporating into numerous existing SISR methods, finding about 40% reduction in flops for near-indistinguishable performance differences.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see above questions.  In my view, these could further strengthen the paper; none are critical.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VJ5G2DaRzr", "reviewer": "Reviewer_yxaN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n============\n\n1. The paper is easy to follow and well motivated. The demand for reduced complexity SISR model is interesting as the authors mentioned that they require even more FLOPs compared to standard classification tasks.\n\n2. The application of shift operations instead of DWC is well motivated as such DWCs often suffer from latency disadvantages in hardware like GPUs.\n\n3. There are some good results with the learnable shift operations.\n\n#### Weaknesses\n============\n\n1. The main weakness of the paper is its contribution, the authors have leveraged already published work to solve similar problem of DNNs, that is the complexity reduction of convolution operation.\n\n2. With the exhaustive research in the domain of Shift-add based CONV operations, for example: shift-add Net [1], Shift-add NAS [2], already showed efficacy of such approaches. This reduces the contribution and insights of the current manuscript.\n\n3. The implementation of Gumble softmax to make the opeation learnable is not new either as already mentioned by the authors. \n\nThus overall, I think its a good work with not enough new contribution and thus validating some of the well accepted efficient convolution approximations.\n\n[1] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS 2020.\n[2] ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks, ICML 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper leverages shift-based operation for generating ghost features for SISR tasks to yield benefits on hardware like GPUs. The paper then introduced learnable shift operations via Gumble softmax operator. The authors provided results on four standard benchmark datasets: set5, set14, B100, and Urban100 to show the efficacy of their proposed approach.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The authors need to add significant amount of contributions to motivate a new work, the current work of efficient convolution for ghost feature extraction is largely taken from existing literature.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zg1Ueq0qeQ", "reviewer": "Reviewer_kXZf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper investigates the feature redundancy of convolutional neural networks in the SISR task.\n2. This paper analyzes the benefits of shift operation on the SISR task and accelerates the practical inference on GPUs.\n3. Extensive experiments demonstrate that learning Ghost features helps achieve comparable performance with fewer parameters, FLOPs and GPU inference latency.\n\nWeaknesses:\n1. Some motivations should be highlighted in this paper. \n2. The model uses shift operation and Gumbel-Softmax trick is not novel.\n3. Some technical details of the proposed method are not clear. \n4. The experiment section should be improved in the paper.\n\nMore details can be found in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a problem on the feature redundancy of convolutional neural networks in the single image super-resolution (SISR) task. To address this, the authors propose an efficient SISR model by using shift operation to replace a part of conventional filters and generate the redundant features (i.e., ghost feature). To make the shift operation learnable, the authors use a Gumbel-Softmax trick and present a procedure of clustering pre-trained models to select the intrinsic filters for generating intrinsic features. Extensive experiments on some datasets demonstrate that both the noncompact\nand lightweight SISR models embedded in the GhostSR method are able to achieve comparable performance with fewer parameters, FLOPs and GPU inference latency.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The motivation for using shift operation should be discussed in the introduction. Additional features are important in SISR. However, why only use shift operation in the model? The shift operation may reduce top left information, as shown in Figure 1. Moreover, convolution has a shift-invariant property and obtains the same amount of information. There are some augmentation operations (e.g., scaling and ColorJitter) in existing computer vision tasks. Besides, some augmentation operations can increase additional information, e.g., different scale information. It would be better to exploit other augmentation operations to improve SR performance.\n\n2. In Figure 2, the benefits of shifting the features for the SISR task are not clear. (a) Small shifts can obtain high-frequency gradient information.  Is such gradient information concatenated in the features? (b) The shift operation seems cannot enlarge the receptive field of filters when the kernel size is fixed.\n\n3. Some baseline methods should be compared. For example, the authors should compare a baseline model by reducing the convolutional layer to a comparable parameter size and train this model with raw features.\n\n4. Different datasets have different performance gains. In Table 1, the model trained on Urban100 has a larger performance drop than other test sets. Could you please give more discussions in the paper?\n\n5. The compared state-of-the-arts are old. The authors should compare with recent SOTA methods (e.g., SwinIR) and conduct experiments by learning Ghost features.\n\n6. In Table 3, the performance of GhostSR is not significant. Compared with GhostSR, the Lightweight SwinIR model has a smaller model size of 886K and a higher PSNR of 28.66 db on Urban100. \n\n7. In Figure 5, it would be better to plot PSNR v.s. FLOPs v.s. Model size. In Figure 7, why shift operation can generate distinguishing feature maps? More discussions should be provided.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript develops a relatively efficient image super-resolution method, in the family of deep learning based ones, which uses a CNN equipped with learnable shift operations to generate the ghost features involved in the SR process. The proposed method has been demonstrated to achieve practical inference acceleration on GPU devices. The AE and three reviewers recognize the main technical contribution that leads to a non-trivial gain in the computational efficiency of SR. 

To be accepted by TMLR, the AE would ask the authors to strengthen their work to address the following issues: 
1)  Proofread the entire paper to fix typos and grammatical errors. 
2)  Give more explanations on Figure 8 which showcases depthwise convolution kernels.
3)  Add a fair comparison against Lightweight SwinIR, which is vital and requested by the reviewer. 
4)  Make your technical contribution more insightful other than just stating the hardware benefits. 

Decision: Accept (poster)