Title: UNDERSTANDING OPPORTUNITIES FOR EFFICIENCY IN SINGLE-IMAGE SUPER RESOLUTION NETWORKS

Abstract: A successful application of convolutional architectures is to increase the resolution of single low-resolution images -a image restoration task called super-resolution (SR). Naturally, SR is of value to resource constrained devices like mobile phones, electronic photograph frames and televisions to enhance image quality. However, SR demands perhaps the most extreme amounts of memory and compute operations of any mainstream vision task known today, preventing SR from being deployed to devices that require them. In this paper, we perform a early systematic study of system resource efficiency for SR, within the context of a variety of architectural and low-precision approaches originally developed for discriminative neural networks. We present a rich set of insights, representative SR architectures, and efficiency trade-offs; for example, the prioritization of ways to compress models to reach a specific memory and computation target and techniques to compact SR models so that they are suitable for DSPs and FPGAs. As a result of doing so, we manage to achieve better and comparable performance with previous models in the existing literature, highlighting the practicality of using existing efficiency techniques in SR tasks. Collectively, we believe these results provides the foundation for further research into the little explored area of resource efficiency for SR.

1. INTRODUCTION:

Rapid progress has been made in the development of convolutional networks (Dong et al., 2015) that are capable of taking a low-resolution image and producing an image with a significant increase in resolution. This image restoration task is referred to as super-resolution (SR) and has many potential applications in devices with limited memory and compute capacity. The fundamental problem however is that the state-of-the-art networks (Lim et al., 2017; Zhang et al., 2018; Zhang et al., 2018) consist of thousands of layers and are some of the most resource intensive networks currently known. Furthermore, due to the spatial dimensions of feature maps needed to maintain or up-scale the input, the number of operations are counted in the billions as opposed to millions in models for discriminative tasks. As a result, there is a need for a general systematic approach to improve the efficiency of SR models.

The challenge of the system resource requirements for deep learning models for tasks other than SR have been carefully studied in previous works (Zhang et al., 2017b; Howard et al., 2017; Ma et al., 2018; Sandler et al., 2018) , achieving massive gains in size and compute with little to no loss in performance. These reductions are achieved with a wide variety of methods being developed grounded in primarily architecture-level changes and techniques grounded in the use of low precision and quantized model parameters. However, how these efficiency methods behave when applied within SR have not yet been studied in significant depth, with very few results published in the literature. Extrapolating from prior results for other tasks is problematic given that predominantly existing studies are applied to discriminative tasks with substantially different architectures and operations. Due to the up-sampling structure of SR models, these efficiency methods may therefore produce potentially stronger side-effects to image distortion.

In this paper, we detail a systematic study that seeks to bridge current understanding in SR and known approaches for scaling down the consumption of system resources by deep models. By examining the impact on image distortion quality when performing various efficiency techniques, we provide the following new insights:

• The effectiveness of low rank tensor decomposition and other convolution approximations, which are comparable and successful in discriminative tasks, can vary considerably in SR.

(See section 4.1).

• Unlike image discriminative networks, SR networks suffer from a worse trade-off between efficiency and performance as more layers are compressed. (See section 4.2)

• The practicality of adopting compression techniques for other tasks to SR as our best models are better or comparable to existing literature. For instance, our best model achieves significantly better performance and 6x less compute than MemNet (Tai et al., 2017b) and VDSR (Kim et al., 2015b) . Additionally, it also performs better and is 4.1x-5.8x smaller than SRMDNF (Zhang et al., 2017a) . (See section 4.3)

• Successful quantization techniques used in image discriminative tasks are equally successful in SR. (See section 5)

2. RELATED WORK:

We focus on using neural networks for SR as they have shown to achieve superior performance against previous traditional approaches ( Timofte et al., 2013; Kim & Kwon, 2010; Chang et al., 2004 ). An SR image can either be evaluated using standard image distortion metrics, such as PSNR, SSIM (Wang et al., 2004) and IFC (Sheikh et al., 2005) , or using perception metrics, such as Ma et al. (2016) , NIQE (Mittal et al., 2013) , and BRISQUE (Mittal et al., 2012) . Blau & Michaeli (2017) provided theoretical backups on the trade-off between image distortion and perception.

Distortion SR: In the distortion line of work, models favour pixel-to-pixel comparisons and are usually trained on either the L1 or L2 (MSE) loss. These models have been known to produce more visually pleasing outcomes on structural images Blau et al. (2018) than perceptual SR models. Dong et al. (2015) first proposed using convolutional networks for SR, leading to a surge in using neural networks for SR. These networks differ in their building blocks for feature extraction and up-sampling. For instance, Dong et al. (2016) proposed a faster convolutional network by taking the down-sampled low-resolution image as an input. Other variations include using more layers (Kim et al., 2015b) , recursive layers (Kim et al., 2015a; Tai et al., 2017a) , memory blocks (Tai et al., 2017b; Ahn et al., 2018) , DenseNet (Huang et al., 2016) blocks (Tong et al., 2017) , residual (He et al., 2015) blocks (Ledig et al., 2016; Lim et al., 2017; Kim & Lee, 2018) , and multipleimage degradations (Zhang et al., 2017a) . Additionally, more recent models use attention (Bahdanau et al., 2014) mechanisms (Liu et al., 2018; Zhang et al., 2018) , back-projection (Haris et al., 2018; Navarrete Michelini et al., 2018) , and other non-conventional non-linear layers (Choi & Kim, 2017; Gu et al., 2018) .

Perceptual SR: Perceptual SR models, on the other hand, are better at reconstructing unstructured details with high perceptual quality (Blau et al., 2018) . These models usually adopt popular models for image distortion and train them using a variety of different loss functions, such as the perceptual loss (Johnson et al., 2016) , contextual loss (Mechrez et al., 2018b) , adversarial loss (Goodfellow et al., 2014) , and the Gram loss (Gatys et al., 2015) . For instance, Choi et al. (2018) adopted EUSR Kim & Lee (2018) and Ledig et al. (2016) ; Wang et al. (2018) ; Mechrez et al. (2018a) adopted SRResNet (Ledig et al., 2016) by making slight architecture changes and replacing the objective. Although these perceptual models are able to generate more visually pleasing results on certain images, they do not seem to work well as inputs for image classification (Jaffe et al., 2017) .

Efficient SR: As models in both tracks are resource-intensive, the recent PIRM 2018 Challenge for mobile (Ignatov et al., 2018) presented a range of high efficiency models that were designed to run faster and perform better than SRCNN (Dong et al., 2015) . These models are complementary to our work and can follow our best practices to achieve greater efficiency gains. A work closely related to our work is done by Ahn et al. (2018) who systemically investigate the impact of using grouped convolutions. Due to the massive design space caused by the variability of training and evaluating these models, we focus on the trade-offs between performance measured by the image distortion metrics and efficiency and leave the rest as future work.

3. SYSTEMATIC STUDY OF LOW-RESOURCE SUPER RESOLUTION NETWORKS:

The key step in our work is to build understanding towards building resource-efficient architectures for super-resolution. While there is a lot of understanding of how these efficiency-saving techniques work in classification problems, there is a lack of experimental studies and systematic approaches to understand their practicality in super-resolution. To our knowledge, this is the first systematic study of wide range efficiency methods on super-resolution.

We measure performances using PSNR and SSIM (Wang et al., 2004) and measure efficiency of memory and compute using the number of parameters and the number of multiply-add operations (Mult-Adds), both of which dictate which platform these models can run on. However, these metrics alone do not reflect the trade-off between performance and efficiency. Therefore, we introduce two new metrics that measures the number of Giga Mult-Adds saved and the number of parameters saved for every 0.01dB PSNR loss in the test sets: Set5 (Bevilacqua et al., 2012) , Set14 (Yang et al., 2010) , B100 (Martin et al., 2001) , and Urban100 (Huang et al., 2015) . These metrics are calculated by taking the difference between the compressed model and the uncompressed model. All Mult-Adds are calculated by upscaling to a 720p image.

We decide to use RCAN Zhang et al. (2018) as our baseline model as it proves to be the state-ofthe-art and has the best performance in the image distortion metrics at the time of writing. We take its simplest building block and build a shallower network and use that as a basis for exploring the use of a variety of techniques.

Implementation Details: We train our models in section 4 and section 5.1 in the same manner as that of EDSR Lim et al. (2017) . In particular, we use 48×48 RGB patches of LR images from the DIV2K dataset Timofte et al. (2017) . We augment the training data with random horizontal flips and 90 degree rotations and pre-process them by subtracting the mean RGB value of the DIV2K dataset.

Our model is trained using the ADAM Kingma & Ba ( 2014) optimizer with hyper-parameters β 1 = 0.9, β 2 = 0.999, and = 10 -8 . The mini-batch size is 16, learning rate begins with 1e -4 and is halved at 200 epochs, and the model is trained for 300 epochs using L1 loss. We train x2 models from scratch and use them as pre-trained models to train x3 and x4 models for faster convergence. Lastly, for ternary quantization in section 5.2, we further train the model with quantization enabled in each forward pass for 40 epochs, starting at a learning rate of 5e -5, and then fix the quantized ternary weights and further train for another 10 epochs at a learning rate of 2.5e -5.

4. EFFICIENT NETWORK ARCHITECTURES FOR SUPER RESOLUTION:

We begin our evaluation by conducting a series of experiments: (i) we explore the effects of applying different resource-efficient architectures to our baseline model (section 4.1), (ii) we consider two best techniques and present trade-off solutions while applying them to different parts of our baseline model (section 4.2), (iii) and lastly, we compare our best results with previous SR architectures (section 4.3).

4.1. EFFECTS OF VARIOUS RESOURCE-EFFICIENT TECHNIQUES:

Motivation: Resource-efficient architectures use various low rank tensor decomposition and other convolutional approximation techniques, which is agnostic and is not specifically designed for any particular task, to build fast and accurate image discriminative models. We first develop an initial understanding of the trade-off solution by replacing and modifying 3x3 convolution layer blocks in the baseline model.

Approach: We explore the use of known techniques such as the bottleneck design, separable/grouped convolutions, and channel shuffling. We take the feature extraction unit from resourceefficient architectures and remove all batch normalisation layers as they were previously shown to reduce performance and increase GPU memory usage (Lim et al., 2017) . For our first set of experiments, we replace all 3x3 convolution layers in the residual groups of our baseline model.

bl: Our baseline model from RCAN (Zhang et al., 2018) . We reduce the number of residual groups (RG) from 10 to 2 and the number of residual channel attention block (RCAB) in each RG from 20 to 5. We use a feature map size of 64. Making the network shallower and small in parameters allow us to clearly understand each architectural changes as opposed to having a deep network which may cause other effects and interplay.

blrn(r)::

We adopt the residual bottleneck design from ResNet (He et al., 2015) with a reduction factor of r. Specifically, a 1x1 convolution is used to compress information among channels by a reduction factor, resulting in a cheaper 3x3 convolution. Another 1x1 convolution is then used to recover the dimension of the output channel and a skip connection is used to pass on information that may have been lost.

blrxn(r,g): We replace the 3x3 convolution in blrn to a 3x3 grouped convolution, forming a block that is similar to that of ResNeXt (Xie et al., 2016) with an additional group size of g. Computation cost is further reduced by the use of grouped convolutions (Krizhevsky et al., 2012) .

blm1: In order to further improve efficiency of the 3x3 grouped convolution, we can maximise the group size, forming a convolution that is known as depthwise convolution. Following this idea, we adopt the MobileNet v1 (Howard et al., 2017) unit which uses depthwise separable convolutions, each consist of a 3x3 depthwise convolution followed by a 1x1 convolution, also known as a pointwise convolution.

bleff(r)::

We can further approximate the 3x3 depthwise convolution by using a 1x3 and a 3x1 depthwise convolution, a technique that is used in EffNet (Freeman et al., 2018) . We adopt the unit from EffNet by removing the pooling layers.

bls1(r, g): We group both 3x3 and 1x1 convolutions and added channel shuffling in order to improve the information flow among channels. In order to test the effects of channel shuffling, we adopt the ShuffleNet v1 (Zhang et al., 2017b) unit.

blclc(g1, g2): Channel shuffle is also used in Clcnet (Zhang, 2017) to further improve efficiency of blm1. In order to maximise efficiency from our adoption of ClcNet units, we follow the group size guidelines recommended by the authors for both the group sizes of the 3x3 (g1) and 1x1 (g2) grouped convolution.

bls2: Apart from using grouped convolutions, Ma et al. (2018) proposed splitting the flow into two, which is termed as channel splitting, and performing convolution on only half of the input channels in each unit at each pass. Channel shuffle is then used to enable information flow between both branches.

blm2(e): Inverted residuals can be used to enable skip connections directly on the bottleneck layers. Therefore, we adopt the MobileNet v2 (Sandler et al., 2018) unit in our experiments Results: Our results in Table 1 show that techniques that result in a better trade-off between memory and performance will have a better trade-off between compute and performance.1 . Overall, the use of bottlenecks alone (blrn) result in the best trade-offs followed by the use of separable/grouped convolutions.

Reducing the number of features to accommodate inverted bottlenecks (blm2) severely impact the performance and thus we omit the results from the table. We speculate that doing so would result in insufficient features at the up-sampling layer to fully capture the up-sampled image representation. Thus, we use the same number of feature maps as our bottleneck. Although the use of inverted residuals in our experiments seem worse off, it may perform better on models that use a larger feature size or multiple smaller up-sampling layers.

Lastly, the use of 1x1 grouped convolution or channel splitting with channel shuffling further reduces the evaluation metric. Although doing so can drastically reduce size, the trade-off does not seem to justify its advantages. Therefore, we recommend using bottlenecks for building resourceefficient SR architectures. If the budget for memory and efficiency is tight, we recommend the use of depthwise separable convolutions instead.

In image discriminative tasks, the proposed architecture changes are comparable in terms of efficiency and accuracy trade-offs. In our work, we show that the sole use of low rank tensor decomposition (bottleneck architectures) provide the best trade-offs, followed by the use of separable/grouped convolutions and the use of both channel splitting and shuffling.

4.2. EFFECTS OF ARCHITECTURAL LAYERS BETWEEN THE INPUT AND OUTPUT LAYER:

Motivation: Bhattacharya & Lane (2016) and Kim et al. (2015c) have shown that it is possible in image classification to maintain a similar or slight drop in performance by decomposing tensors of known models. However, our models suffer a significant drop in performance. (Table 1 ). Therefore, in order to further understand the extent of their applicability in SR, we apply the top two best techniques, which are bottleneck reduction (blrn) and depthwise separable convolutions (blm1), on various different parts of our baseline model.

Approach: Our preliminary experiments with applying some of these techniques on the first and last convolution layer led to worse trade-offs. Therefore, we apply our techniques between them.

We replace the sub-pixel convolution upsampling layer to the enhanced upscaling module (EUM) as proposed by Kim & Lee (2018) to allow the use of skip connections. Using EUM leads to an increase in performance at a slight cost of both memory and compute. Results: Our results in Table 2 reinforce our findings in section 4.1 that the adoption of bottleneck reduction alone leads to the best trade-offs, followed by the use of group convolutions. Therefore, we recommend taking gradual steps to compress the model. For instance, we suggest gradually changing convolutions to use bottleneck reduction, avoiding the up-sampling, first, and last convolutions until a budget is reached. If further compression is needed, we suggest changes to the up-sampling layer or the use of group convolutions.

4.3. COMPARISONS WITH PREVIOUS SR MODELS:

We take our derived best models based on different budgets from our first two experiments (See section 4.1 & 4.2) and compare them with the existing literature, which is shown in Table 3 . For fair comparisons, we omit models that are way bigger by several magnitudes as their performances are much better. Likewise, we exclude models that are way smaller as their performance are much worse. Regardless, our techniques can be applied to any model for further trade-offs between performance and efficiency.

Although our main objective is not to beat previous models but to understand and recommend techniques that can be applied to any existing model, we manage to derive models that are better or comparable to other models in the literature. For instance, in terms of size and evaluation metric, our best model (blrn-e[rb]) outperforms all models that have a count of 1,500K parameters and below. By comparing compute and evaluation, our best model performs better and has roughly x6 less operations than MemNet (Tai et al., 2017b) . It is also comparable with the CARN model in the number of operations, trading a slightly worse performance with a 2.5x size reduction. Overall, our best model is better than earlier models such as VDSR (Kim et al., 2015b) and later models such as SRMDNF (Zhang et al., 2017a) for 3x and 4x scales. Our second and third best models also outperform earlier models in performance with huge savings in the number of operations for 3x and 4x scales. Our results show that these techniques which are designed for image discriminative tasks can be effective in SR. Visual comparisons for some of these models can be found in the appendix.

5. QUANTIZATION AND LOW-PRECISION UNDER SUPER RESOLUTION:

In our next set of experiments, we examine the viability of quantization and the use of extreme low-precision (ternary/binary) as mechanisms to reduce system resource for SR.

5.1. INTEGER QUANTIZATION:

Motivation: With the success of low precision on neural networks on classification problems, we aim to show initial understanding of applying 8-bits integer quantization on our baseline model as described in section 4.1. Moving from 32-bits to 8-bits will result in a 4x reduction in memory and allow support for low-power embedded devices.

Approach: We train the model in full precision and apply the quantization scheme in Tensorflow-Lite for integer-only arithmetic (Jacob et al., 2017) and retrain for an additional 5 epochs with the a learning rate of 5e -5.

Results: Our results show that applying quantization lead to a slight evaluation loss in 2x scaling and a slight improvement in 4x scaling. Our results are similar to that of classification Jacob et al. (2017) . Furthermore the results show that deep neural networks are robust to noise and perturbations caused by quantization. Therefore, we strongly recommend quantization especially on hardware that can further utilise its benefits. 

Motivation::

The success of using binarized (Courbariaux & Bengio, 2016; Rastegari et al., 2016; Lin et al., 2017) and ternarized neural networks (Li & Liu, 2016; Tschannen et al., 2017) to approximate the full-precision convolutions in image discriminative tasks motivates us to experiment the effectiveness of these techniques in SR.

Approach: We adapt the baseline SR architecture used in prior experiments in section 4.1 but modify it structurally by replacing every convolution layer with sum-product convolution layers proposed in StrassenNets (Tschannen et al., 2017) . These sum-product convolution layers represent a sum-product network (SPN) that is used to approximate a matrix multiplication. Specifically, each convolution layer is replaced with a convolution layer that outputs r feature maps, followed by a element-wise multiplication and a transpose convolution layer. As both the convolution layers hold ternary weights, the number of multiply operations required is determined by the number of element-wise multiplication which is controlled by r. Besides outlining the trade-off of tuning r, we aggressively use group convolutions.

Results: Similar to section 5.1, the results in Table 5 are similar to that of image discriminative tasks. Specifically, the higher the width of the hidden layer of the SPN, r, the better the performance at a cost of additional multiplications and additions. When r = 6c out, we achieve an evaluation score that is close to the uncompressed model for 2x scales and suffer a slight drop for 3x and 4x scales. Any further attempts to increase r do not improve evaluation metric.

As proposed by Tschannen et al. ( 2017), we use group convolutions to reduce the number of additions. We take a step further and experiment with a wide range of groups as well. We found that the reduced number of additions do not justify the evaluation drop; the use of a lower r is better than the use of groups. Additionally, since multipliers are more costly and take up more area on chip than adders, we suggest lowering r instead of using grouped convolutions. Through an extensive set of experiments, we show that some of the previous efficiency techniques that are successful in image discriminative tasks can be successfully applied to SR. Although these techniques are comparable in the former tasks, we highlight their varying effectiveness in SR and derive a list of best practices to construct or reduce any model that are designed to reduce image distortion:

• The sole use of low rank tensor decomposition (bottleneck design) results in the best tradeoffs between performance and efficiency. If further compression of memory and/or compute is needed, separable/grouped convolution is recommended. If efficiency on conventional hardware is the topmost priority, we recommend reducing the number of layers or adopting the use of both channel splitting and shuffling (Ma et al., 2018) .

• The fewer resource-efficient architecture changes applied, the better the trade-off. Therefore, we recommend a mixture of convolution and resource-efficient units unless further compression is needed.

• Avoid architecture changes on the first and last convolution layers.

• We strongly recommend using any form of quantization if the hardware supports it. 

:

Results for 3x and 4x upscaling show similar performance and efficiency trade-offs

Figures and Tables:

Thus, in order to maintain the memory cost, we use recursion, forming the enhanced recursive upscaling module (ERUM) shown in figure 1. The number of ERUMs is the same as the scaling factor and each ERUM recurses twice or thrice for x2, x4 or x3 scales respectively. Experiments that use ERUMs for up-sampling are indicated with a postfix -e. We calculate our trade-off metrics based on our baseline model with ERUM as its up-sampling layer instead bl-e. We modify all 3x3 convolution layers as such: rb: Changes are made in residual blocks/modules. rg: Changes are made in residual groups, therefore including those in rb. (Experiments in section 4.1 are done in this setting.) rg+u: Changes are made in both rg and the up-sampling layers (ERUMs).

Figure 1: Our proposed ERUM for image upscaling.

Figure 2: Visual comparisons with state-of-the-art models on Set5 and Set14. VSDR and LapSRN are comparable to our models with regards to model size and/or number of operations and RCAN is x22.8-x30.5 larger and has x8.4-x9.3 more operations.

Quantitative results of applying resource-efficient techniques. bold/italics indicates best/second-best trade-off.

Table content: <table><tr><td>Scale</td><td>Model</td><td>Params (K)</td><td>Mult-Adds (G)</td><td>Set5 PSNR/SSIM</td><td>Set14 PSNR/SSIM</td><td>B100 PSNR/SSIM</td><td>Urban100 PSNR/SSIM</td><td>Mult-Add Saved (G)</td><td>Params Saved (K)</td></tr><tr><td/><td>bl</td><td>1006</td><td colspan="5">231.2 37.86/0.9600 33.39/0.9159 32.06/0.8982 31.74/0.9248</td><td>-</td><td>-</td></tr><tr><td/><td>blrn(r=2)</td><td>464</td><td colspan="5">106.5 37.61/0.9591 33.18/0.9137 31.90/0.8961 31.09/0.9173</td><td>0.9819</td><td>4.27</td></tr><tr><td/><td>blm1</td><td>265</td><td colspan="5">60.7 37.45/0.9586 33.00/0.9122 31.79/0.8948 30.65/0.9128</td><td>0.7894</td><td>3.43</td></tr><tr><td/><td>blrxn(r=2, g=4)</td><td>305</td><td colspan="5">69.9 37.45/0.9585 33.01/0.9123 31.80/0.8948 30.71/0.9131</td><td>0.7755</td><td>3.37</td></tr><tr><td/><td>blrn(r=4)</td><td>258</td><td colspan="5">59 37.42/0.9583 32.96/0.9120 31.76/0.8943 30.66/0.9123</td><td>0.7653</td><td>3.32</td></tr><tr><td>2×</td><td>blclc(g1=32, g2=2)</td><td>232</td><td colspan="5">52.9 37.40/0.9581 32.94/0.9118 31.73/0.8939 30.53/0.9108</td><td>0.7278</td><td>3.16</td></tr><tr><td/><td>bls2</td><td>211</td><td colspan="5">48.4 37.37/0.9580 32.96/0.9117 31.71/0.8936 30.49/0.9102</td><td>0.7254</td><td>3.15</td></tr><tr><td/><td>bleff(r=2)</td><td>257</td><td colspan="5">58.7 37.33/0.9580 32.94/0.9114 31.71/0.8936 30.41/0.9098</td><td>0.6485</td><td>2.82</td></tr><tr><td/><td>blm2(e=2)</td><td>561</td><td colspan="5">128.9 37.43/0.9586 33.08/0.9130 31.81/0.8949 30.77/0.9138</td><td>0.5219</td><td>2.27</td></tr><tr><td/><td>bls1(r=2, g=4)</td><td>188</td><td colspan="5">42.9 37.06/0.9568 32.74/0.9096 31.52/0.8912 29.94/0.9031</td><td>0.4968</td><td>2.16</td></tr><tr><td/><td>blm2(e=3)</td><td>763</td><td colspan="5">175.4 37.56/0.9589 33.11/0.9133 31.86/0.8954 30.93/0.9155</td><td>0.3509</td><td>1.53</td></tr></table>

Quantitative results of applying techniques on different parts of the model. We took the best three derived models given three different budgets and compared them with previous models in Table3bold/italics indicates best/second-best trade-off.

Table content: <table><tr><td>Scale</td><td>Model [Changes]</td><td>Params (K)</td><td>Mult-Adds (G)</td><td>Set5 PSNR/SSIM</td><td>Set14 PSNR/SSIM</td><td>B100 PSNR/SSIM</td><td>Urban100 PSNR/SSIM</td><td>Mult-Add Saved (G)</td><td>Params Saved (K)</td></tr><tr><td/><td>bl-e</td><td>1006</td><td colspan="5">265.3 37.92/0.9602 33.43/0.9162 32.09/0.8988 31.83/0.9256</td><td>-</td><td>-</td></tr><tr><td/><td>blrn-e(r=2)[rb]</td><td>535</td><td colspan="5">156.8 37.75/0.9596 33.30/0.9153 32.00/0.8973 31.48/0.9218</td><td>1.4662</td><td>6.3649</td></tr><tr><td/><td>blm1-e[rb]</td><td>363</td><td colspan="5">117 37.65/0.9592 33.19/0.9143 31.92/0.8964 31.13/0.9181</td><td>1.0746</td><td>4.6594</td></tr><tr><td>2×</td><td>blm1-e[rg]</td><td>265</td><td colspan="5">94.7 37.59/0.9590 33.12/0.9132 31.87/0.8959 30.91/0.9158</td><td>0.9584</td><td>4.1629</td></tr><tr><td/><td>blrn-e(r=2)[rg]</td><td>464</td><td colspan="5">140.5 37.64/0.9592 33.20/0.9142 31.93/0.8967 31.18/0.9185</td><td>0.9455</td><td>4.1061</td></tr><tr><td/><td>blrn-e(r=2)[rg+u]</td><td>370</td><td colspan="5">97.1 37.56/0.9589 33.10/0.9131 31.86/0.8954 30.99/0.9164</td><td>0.9557</td><td>3.6136</td></tr><tr><td/><td>blm1-e[rg+u]</td><td>137</td><td colspan="5">35.4 37.35/0.9580 32.94/0.9117 31.72/0.8939 30.45/0.9103</td><td>0.8181</td><td>3.0925</td></tr><tr><td/><td>bl-e</td><td>1080</td><td colspan="5">156.3 34.35/0.9269 30.29/0.8415 29.06/0.8046 28.13/0.8521</td><td>-</td><td>-</td></tr><tr><td/><td>blrn-e(r=2)[rb]</td><td>609</td><td colspan="5">108.1 34.19/0.9257 30.18/0.8394 29.00/0.8026 27.89/0.8469</td><td>0.8456</td><td>8.2632</td></tr><tr><td/><td>blm1-e[rb]</td><td>437</td><td colspan="5">90.5 34.09/0.9249 30.13/0.8379 28.95/0.8015 27.69/0.8419</td><td>0.6784</td><td>6.6289</td></tr><tr><td>3×</td><td>blrn-e(r=2)[rg+u]</td><td>397</td><td colspan="5">57.6 33.97/0.9236 30.04/0.8362 28.89/0.7997 27.50/0.8376</td><td>0.6902</td><td>4.7762</td></tr><tr><td/><td>blrn-e(r=2)[rg]</td><td>538</td><td colspan="5">100.9 34.13/0.9251 30.14/0.8380 28.97/0.8016 27.72/0.8421</td><td>0.6368</td><td>6.2299</td></tr><tr><td/><td>blm1-e[rg]</td><td>339</td><td colspan="5">80.6 34.08/0.9244 30.03/0.8356 28.92/0.8005 27.55/0.8386</td><td>0.6056</td><td>5.928</td></tr><tr><td/><td>blm1-e[rg+u]</td><td>146</td><td colspan="5">21.4 33.73/0.9221 29.83/0.8320 28.78/0.7970 27.10/0.8283</td><td>0.5644</td><td>3.9079</td></tr><tr><td/><td>bl-e</td><td>1154</td><td colspan="5">135.5 32.08/0.8942 28.58/0.7815 27.56/0.7360 26.16/0.7872</td><td>-</td><td>-</td></tr><tr><td/><td>blrn-e(r=2)[rb]</td><td>683</td><td colspan="5">108.3 32.10/0.8938 28.51/0.7795 27.51/0.7340 25.95/0.7808</td><td colspan="2">0.8774 15.1935</td></tr><tr><td/><td>blm1-e[rb]</td><td>511</td><td colspan="5">98.4 31.98/0.8921 28.45/0.7778 27.47/0.7328 25.80/0.7754</td><td>0.5456</td><td>9.4559</td></tr><tr><td>4×</td><td>blrn-e(r=2)[rg+u]</td><td>424</td><td colspan="5">50 31.84/0.8898 28.34/0.7750 27.38/0.7295 25.52/0.7673</td><td>0.6577</td><td>5.6154</td></tr><tr><td/><td>blm1-e[rg]</td><td>413</td><td colspan="5">92.8 32.02/0.8921 28.44/0.7765 27.44/0.7310 25.67/0.7715</td><td>0.5272</td><td>9.1481</td></tr><tr><td/><td>blrn-e(r=2)[rg]</td><td>612</td><td colspan="5">104.3 32.02/0.8925 28.47/0.7779 27.48/0.7323 25.79/0.7755</td><td>0.5032</td><td>8.7419</td></tr><tr><td/><td>blm1-e[rg+u]</td><td>156</td><td colspan="5">18.7 31.47/0.8847 28.12/0.7697 27.26/0.7252 25.16/0.7538</td><td>0.4928</td><td>4.211</td></tr></table>

We extend the table that is provided by Ahn et al. (2018) and compared our best three models (in order). For fair comparisons, we do not include models that are much bigger or much smaller than our derived models.

Table content: <table><tr><td>Scale</td><td>Model</td><td>Params (K)</td><td>Mult-Adds (G)</td><td>Set5 PSNR/SSIM</td><td>Set14 PSNR/SSIM</td><td>B100 PSNR/SSIM</td></tr></table>

Quantitative results of applying 8-bit integer quantization in TF-Lite.

Table content: <table><tr><td colspan="2">Scale Model</td><td>Params (K)</td><td>Mult-Adds (G)</td><td>Set5 PSNR/SSIM</td><td>Set14 PSNR/SSIM</td><td>B100 PSNR/SSIM</td><td>Urban100 PSNR/SSIM</td></tr><tr><td>2×</td><td>bl bl q</td><td>961</td><td>231.2</td><td colspan="4">37.82/0.9599 33.36/0.9160 32.05/0.8981 31.67/0.9237 37.68/0.9582 33.34/0.9146 32.01/0.8966 31.64/0.9226</td></tr><tr><td>3×</td><td>bl bl q</td><td>1191</td><td>122.4</td><td colspan="4">34.20/0.9257 30.15/0.8392 29.01/0.8028 27.91/0.8467 34.17/0.9259 30.13/0.8393 29.00/0.8030 27.92/0.8474</td></tr><tr><td>4×</td><td>bl bl q</td><td>1154</td><td>93</td><td colspan="4">32.01/0.8927 28.41/0.7793 27.49/0.7337 25.87/0.7792 31.99/0.8922 28.43/0.7794 27.51/0.7337 25.88/0.7790</td></tr><tr><td colspan="4">5.2 TERNARY PRECISION</td><td/><td/><td/></tr></table>

Quantitative results of applying ternary-weighted SP convolutions. We omit the use of group convolutions as it leads to worse results. c out refers to the number of output channels in each SP convolution layer.

Table content: <table><tr><td colspan="3">Scale Model r</td><td>Reduction in Mult (%)</td><td>Reduction in Add (%)</td><td>Set5 PSNR/SSIM</td><td>Set14 PSNR/SSIM</td><td>B100 PSNR/SSIM</td><td>Urban100 PSNR/SSIM</td></tr><tr><td/><td>bl</td><td>-</td><td>-</td><td colspan="4">-37.86/0.9600 33.39/0.9159 32.06/0.8982 31.74/0.9248</td></tr><tr><td/><td/><td>c out</td><td>99.82</td><td colspan="4">-15.96 37.59/0.9588 33.14/0.9138 31.88/0.8959 31.02/0.9171</td></tr><tr><td>2×</td><td>ST-bl</td><td>2c out 4c out</td><td>99.64 99.28</td><td colspan="4">-132.1 37.73/0.9595 33.30/0.9152 31.98/0.8973 31.37/0.9210 -364.38 37.81/0.9598 33.31/0.9151 32.03/0.8978 31.59/0.9229</td></tr><tr><td/><td/><td>6c out</td><td>98.92</td><td colspan="4">-596.65 37.85/0.9600 33.41/0.9162 32.06/0.8981 31.68/0.9240</td></tr><tr><td/><td>bl</td><td>-</td><td>-</td><td colspan="4">-34.24/0.9260 30.26/0.8405 29.03/0.8033 27.96/0.8479</td></tr><tr><td/><td/><td>c out</td><td>99.81</td><td colspan="4">-35.58 33.79/0.9215 29.92/0.8340 28.82/0.7983 27.25/0.8318</td></tr><tr><td>3×</td><td>ST-bl</td><td>2c out 4c out</td><td>99.64 99.28</td><td colspan="4">-171.33 33.99/0.9236 30.07/0.8365 28.91/0.8005 27.54/0.8389 -442.83 34.16/0.9250 30.11/0.8380 28.97/0.8021 27.73/0.8435</td></tr><tr><td/><td/><td>6c out</td><td>98.92</td><td colspan="4">-714.33 34.17/0.9253 30.15/0.8383 28.99/0.8023 27.83/0.8454</td></tr><tr><td/><td>bl</td><td>-</td><td>-</td><td colspan="4">-32.06/0.8930 28.49/0.7787 27.50/0.7337 25.87/0.7788</td></tr><tr><td/><td/><td>c out</td><td>99.81</td><td colspan="4">-26.04 31.46/0.8829 28.15/0.7709 27.27/0.7265 25.24/0.7566</td></tr><tr><td>4×</td><td>ST-bl</td><td>2c out 4c out</td><td>99.64 99.28</td><td colspan="4">-152.24 31.72/0.8871 28.31/0.7743 27.37/0.7296 25.49/0.7662 -404.65 31.90/0.8901 28.40/0.7770 27.45/0.7319 25.67/0.7723</td></tr><tr><td/><td/><td>6c out</td><td>98.93</td><td colspan="4">-657.06 31.95/0.8907 28.43/0.7777 27.46/0.7324 25.77/0.7752</td></tr><tr><td colspan="7">6 BEST PRACTICES FOR EFFICIENT SUPER-RESOLUTION</td></tr></table>

Peer review (iclr2019):
 The paper proposes a detailed empirical evaluation of the trade-offs achieved by various convolutional neural networks on the super resolution problem. The paper provides an extensive evaluation of different architectural changes and the trade-off between savings in terms of memory and computational cost and performance, measured in terms of PSNR and SSIM. This is an empirical paper, thus it does not provide technical contributions. I do think that the insights obtained from such an empirical evaluation could be of interest for practitioners and researchers working on the problem. My main concern is the method only evaluates the trade-offs between model efficiency (in terms of memory and/or computation) and performance measured using metrics that are known to not be well correlated with perceptual quality. Thus it is not obvious to me that the insights obtained in this work would translate to the other case. It is well known that PSNR favors blurry solutions over perceptually more appealing solutions. This comes from the fact that there is no information in the low resolution image to produce the missing high resolution details. Filling up plausible details in a way that is different from the original image would lead to high PSNR. Models that treat the super resolution problem as a regression task using similarity in pixel space, tend to produce blurry solutions and require very large models to improve the score. In recent years, many works have been studying the use of perceptual losses to mitigate this issue or simply treating the super resolution problem as conditional generative modeling. For instance, models using L2 losses in a perceptually more relevant (or learned) feature spaces [A, B], or including GAN losses [C, D] (to list a few). To my knowledge, these models are the current state of the art in terms of perceptual quality. This has been evaluated empirically via perceptual tests [D]. This line of work needs to be cited. In my view, the paper needs to provide a detailed justification on why models using these losses are not considered. Would the conclusions drawn on this work transfer to that setting? Furthermore, it would be good to perform perceptual tests to perform this evaluation. It would be good to provide some canonical examples in the appendix. The overall writing of the paper could be improved. Several sentences are difficult to read, due to typos or the construction of the sentences. The paper evaluates many architectural modifications proposed by other works. It would be good to add an appendix with a small description of what these are. This would make the paper self-contained an easier to read (I had too look up a few of them). The authors mentioned that they first train models for scaling factor of x2 and then use them for training settings higher magnification. How is this exactly done? Please provide details. I am curious of weather using some for of distillation techniques would be useful here. Did you try scaling factors larger than x4? Scaling factors of x2 does not seem very relevant, as simpler methods can achieve already quite competitive results (such as simple interpolation methods) The authors seem to be citing Zhang et al (2018) as a reference to attention mechanisms. To my knowledge the paper that proposed these mechanisms is [E]. The citation style is not used properly throughout the manuscript. As an example: “… proposed in StrassenNets Tschannen et al (2017).” Should be “… proposed in StrassenNets (Tschannen et al, 2017).” Or “… proposed in StrassenNets proposed by Tschannen et al (2017).” [A] Johnson, J. et al. "Perceptual losses for real-time style transfer and super-resolution." ECCV, 2016. [B] Bruna, J. et al "Super-resolution with deep convolutional sufficient statistics." ICLR 2016. [C] Ledig, C. et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network." CVPR. Vol. 2. No. 3. 2017. [D] Sønderby, C. K., et al. "Amortised map inference for image super-resolution." arXiv preprint arXiv:1610.04490(2016). [E] Bahdanau, D. et al "Neural machine translation by jointly learning to align and translate." arXiv (2014).