CRNet: Image Super-Resolution Using A Convolutional Sparse Coding  Inspired Network
Authors: Menglei Zhang
Zhou Liu
Jingwei He
Lei Yu
Venue:  (2020)
Keywords: Convolutional sparse coding
LISTA
image super-resolution

Abstract: Convolutional Sparse Coding (CSC) has been attracting more and more attention in recent years, for making full use of image global correlation to improve performance on various computer vision applications. However, very few studies focus on solving CSC based image Super-Resolution (SR) problem. As a consequence, there is no significant progress in this area over a period of time. In this paper, we exploit the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and it can be implemented using CNN architectures. Then we develop a novel CSC based SR framework analogy to the traditional SC based SR methods. Two models inspired by this framework are proposed for pre-/post-upsampling SR, respectively. Compared with recent state-of-the-art SR methods, both of our proposed models show superior performance in terms of both quantitative and qualitative measurements.

Under review as a conference paper at ICLR 2020
CRNET: IMAGE SUPER-RESOLUTION USING A CON- VOLUTIONAL SPARSE CODING INSPIRED NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional Sparse Coding (CSC) has been attracting and more attention in recent years, for making full use of image global correlation to improve perfor- mance on various computer vision applications. However, very few studies focus on solving CSC based image Super- Resolution (SR) problem. As a consequence, there is no significant progress in this area over a period of time. In this paper, we exploit the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and it can be implemented using CNN architectures. Then we develop a novel CSC based SR framework analogy to the traditional SC based SR methods. Two models in- spired by this framework are proposed for pre-/post-upsampile Compared with recent state-of-the art SR methods, both of our proposed models show superior performance in terms of both quantitative and qualitative measure- ments.
SR, respectively.
INTRODUCTION
Single Image Super-Resolution (SISR), which aims to restore visually pleasing High-Resolution (HR) image from its Resoluettttion (LR) version, still a challenging task within computer vision research community (Timofte et al. 2017: 2018). Since multiple solutions exist for the mapping from LR to HR space, SISR highly ill-posed. To regularize the solution of SISR, various priors of natural images have been exploited, especially the current leading learning-based methods (Wang
al. 2013 Dong et al. 2016 Mao al.
2016 Kim et al.) 2016a b. Tai et al.] 2017a Lim al.)
2017
Ahn et al. 2018 8 Haris al. [0188] Li et al. 2018 Zhang al. 2018) are proposed directly learn the non-lincar LR-HE mapping.
By modeling the sparse prior in natural images, the Sparse Coding (SC) based methods for SR (Yang et al. 2008 2010 2014) with strong theoretical support are widely used owing to their excellent per- formance. Considering the complexity in images, these methods divide the image into overlapping patches and aim to jointly train two over-complete dictionaries for LR/HR patches. There are usu- ally three steps in these methods' framework. First, overlapping patches are extracted from input image. Then to reconstruct the HR patch, the sparse representation of LR patch can be applied to the HR dictionary with the assumption that LR/HR patch pair shares similar sparse representation. The final HR image is produced by aggregating the recovered HR patches.
Recently, with the development of Deep Learning (DL), many researchers attempt to combine the advantages of DL and SC for image SR. Dong et al. (Dong et al. (0166) firstly proposed the seminal CNN model for SR termed as SRCNN, which exploits a shallow convolutional neural network to learn nonlinear LR-HR mapping in end-to-end manner and dramatically overshadows conven- tional methods ( (Yang et al.) 2010: Timofte et al., 2014). However, sparse prior is ignored large extent in SRCNN for it adopts generic architecture without considering the domain expertise. To address this issue, Wang et al. (Wang et al. 2015) implemented a Sparse Coding based Network (SCN) for image SR, by combining the merits of sparse coding and deep learning, which fully ploits the approximation sparse coding learned from the LISTA (Gregor & LeCun 2010) based nub-network.
It's worth to note that most of SC based methods utilize the sparse prior locally (Papyan et al. 2017b), i.e., coping with overlapping image patches. Thus the consistency of pixels in overlapped

---

Under review as a conference paper at ICLR 2020
Pre-Upsam Pia--upsamping Models
Post- Saa-auppapinngg Models
32.8
32.0
31.2
30.4
29.6
Set5 x4
Urban100
28.0
27.2
26.4
25.6
24.8
24.0
SRCNN PAMI16
RED30 NIPS16
VDSR CL CYPR16
DRCN CVPRI6
DRRN CVPR17
ManNet ICCV17
CRN Onrs
SeDense ICCV17
EDSR CYPRW17
MDSR CYPRW17 CARN ECCVIS
MSRN ECCV
D-DBYY CVPRIS
RDN CVPRI8
CRN Ours
CRNet-B: Ours
Figure 1: PSNRs of recent state-of-the-arts for scale factor x4 on Set5 (Bevilacqua et al. 2012) and Urban 100 (Huang et al. 2015). Red names represent our proposed models.
patches has been ignored (Gu et al. 2015 Papyan et al. 2017b To address this issue, CSC is proposed to serve sparse prior as a global prior Zeiler et al. 2010 Papyan et al.] 20176 2018) and it furnishes way to fill the local-global gap by working directly the entire image by convolution
operation. Consequently, CSC has attained much attention from researchers (Ziiler et al., 2010
Bristow al. 2013 Heide al. 2015 Gu al. 2015 Sreter & Giryes) 2018 CaarciaCCardona & Wohlberg 20) However, very few studies focus on the validation of CSC for image SR (Gu al. 2015), resulting in no work been reported that CSC based image SR can achieve state-of-the art performance. Can CSC based image SR show highly competitive results with recent state-of the
art methods (Dong et al. 2016 Mao et al. 2016, Kim et al. 2016a b. Tai al., 201 b Tong al.)
2017 Lim et al. 2017 Li et al. 2018 Zhang et al. 2018)?
? To answer this question, the following
issues need to be considered:
Framework Issue. Compared with SC based image SR methods (Yang et al. 2008 2010). the lack of a unified framework has hindered progress towards improving the performance of CSC based image SR.
Optimization Issue The previous CSC based image SR method (Gu al. 2015) contains several steps and they are optimized independently. Hundreds of iterations required to solve the CSC problem in each step.
Memory Issue. To solve the CSC problem, ADMM (Boyd et al. 2011 is commonly employed (Bristow et al. 2013) Wohlberg, 2014, Heide et al., 2015) Wohlberg, 2016 Faaria-Cardona & Wohlberg 2018a) Sa), where the whole training needs to be loaded in memory. As consequence, is not applicable to improve the performance by enlarging the training set.
Multi-Scale Issue. Training single model for multiple scales is difficult for the previous CSC based image SR method (Gu et al. 2015).
Based on these considerations, in this paper, attempt answer the aforementioned question. Specifically, we exploit the advantages of CSC and the powerful learning ability of deep learning to address image SR problem. Moreover, massive theoretical foundations for CSC (Papyan al. 20176] 2018: Garcia-Cardona & Wohlberg 2018b) make our proposed architectures interpretable and also enable to theoretically analyze our SR performance. In the rest of this paper, we first troduce CISTA, which can be naturally implemented using CNN architectures for solving the CSC problem. Then we develop framework for CSC based image SR, which can address the Frame- work Issue. Subsequently, CRNet-A (CSC and Residual learning based Network) and CRNet-B in- spired by this framework are proposed for image SR. They are classified as pre- and post-upsampling the former takes Interpolated LR (ILR) images as input
models (Wang et al. 2019) respectively, while the latter processes LR images directly. By adopting CNN architectures, Optimization Issue and Memory Issue would be mitigated to some extent. For Multi-Scale Issue, with the help of the recently introduced scale augmentation (Kim et al.) 2016a]]b or speciiic multi-path learning (Lim et al. 2017 Wang et al., 2019) strategies, both of our models are capable of handling multi-
2

---

Under review as a conference paper at ICLR 2020
scale SR problem effectively, and achieve favorable performance against state-of-the-ards as shown in Fig. The main contributions of this paper include:
We introduce CISTA, which can be naturally implemented using CNN architectures for solving the CSC problem.
A novel framework for CSC based image SR is developed. Two models, CRNet-A and
CRNet-B, B, inspired by this framework are proposed for image SR.
The differences between our proposed models and several SR models with recursive learn- ing strategy, e.g., DRRN (Tai et al., 2017a), SCN (Wang et al. 2013), DRCN (Kim et al., 2016b), are discussed.
RELATED WORK
Sparse coding has been widely used in variety of applications (Zhang et al., 2015). As for SISR, Yang et al. Yang et al., 2008) proposed a representative Sparse coding based Super- Resolution (ScSR) method. In the training stage, ScSR attempts to learn the LR/HR overcomplete dictionary pair Di/D, jointly by given a group of LR/HR training patch pairs Ty/K;, In the test stage, the HR patch 2t is reconstructed from its LR version æ by assuming they share the same sparse code. Specifically, the optimal sparse code is obainted by minimizing the following sparsity-inducing 4 norm regularized objective function
arg min a0 - Di=112 + A||=||1>
(1)
then the HR patch is obtained by 2n DAz². . Finally, the HR image can be estimated by ag- gregating all the reconstructed HR patches. Inspired by ScSR, many SC based methods have been proposed with various constraints sparse code dictionary Yang al. (2012) Wang al. 2012).
Traditional SC based SR algorithms usually process images in a patch based manner to reduce the burden of modeling and computation, resulting in the inconsistency problem (Papyan et al. 20176). As special case of SC, CSC is inherently suitable Zeiler et al. 2010) and proposed to avoid the inconsistency problem by representing the whole image directly. Specifically, an image E xn., can be represented as the summation of m feature maps z, E R", convolved with the corresponding filters fi Rsxs y >i fi Zz, where @ is the convolution operation.
Gu et al. (Gu et al. 2015) proposed the CSC-SR method and revealed the potential of CSC for image SR. In (Gu al. 20115). CSC-SR requires solve the following CSC based optimization problem in both the training and testing phase:
M - 2--AS
min f.z
y
fi
1
1=1
which is solved by alternatively optimizing the and I subproblems (Wohlberg 2014) The sub- problem is a standard CSC problem. Hundreds iterations are required to solve the CSC problem and the aforementioned Optimization Issue and Memory Issue cannot be completely avoided. In- spired by the success of deep learning based sparse coding (Gregor & LeCun 2010), we exploit the natural connection between CSC and CNN to solve the CSC problem efficiently.
3 CISTA FOR SOLVING CSC PROBLEM
CSC can be considered as special case of conventional SC, due to the fact that convolution opera- tion be replaced with matrix multiplication, so the objective function of CSC be formulated as:
m
min
y
Σ F2Z
2
(3)
y, 2i are in vectorized form and F, is sparse convolution matrix with the following attributes:
Fiz, f: 8
Ff 2; fli (1ipp(((lipr((()) = #lip(ff) 8
(4)
3

---

Under review as a conference paper at ICLR 2020
CISTA Block
19
R
Rell
lg
recursions
W
Rel.1
ReLl
ReLU
Figure 2: The architecture of the pre-upsampling model CRNet-A. The proposed CISTA block with K' recursions is surrounded by the dashed box and unfolded version is shown the bottom. S is shared across every recursion.
where fliplr(.) and f flipud(; are following the notations of Zeiler et al. Zeiler et al. 2010 representing that array flipped in left/right or up/down direction.
Iterative Soft Thresholding Algorithm (ISTA) (Daubechies et al. 2004) can be utilized to solve
the R iteration:
2k+11 ho
+
F y Fzz))
L
where L is the Lipschitz constant, F [F, FmJ and Fz Σm F,z, Using the relation in (4 replace the matrix multiplication with convolution operator, can reformulate (5) as:
Zk-1 = ho
Izs + f
2x)
(6)
(lip(())) flip:).........f................ flip(fm)) and I is the identity matrix. Note that identity matrix I is also a sparse convolution matrix, so according to (4). there existing filter satisfies:
where =
fm]. flip(f)
===
so 66 becomes:
2++1 = ho (W 3 S8
where W W= f and S - fflip(f) f. Even though 3 is for a single image with channel, the extension to multiple channels (for both image and filters) and multiple images is mathematically straightforward. Thus for y E Rdxxx xn= representing images size ny x ne with channels, (8) still true with W mxxxxx and S mmxmxsxe
As for hgg Papyyn et al. 2017a reveals two important facts: (1) the expressiveness of the sparsity inspired model is not affected even by restricting the coefficients be nonnegative; (2) the ReLU Nair & Hinton 2010) activation function and the soft nonnegative thresholding operator are equal, that
h (ax max(a a 0,0) ReLU (a 0).
We set = 0 for simplicity. So the final form of is:
I ReLU (W 2 5 xz).
(10)
One that (10) is convolutional form of (5) name CISTA. It provides the solution of (3) with theoretical guarantees (Daubechies et al. 2004). Furthermore, this convolutional form can be implemented employing CNN architectures. So W and S in (T0) would be trainable.

---

Under review as a conference paper at ICLR 2020
4 PROPOSED METHOD
In this work, exploiting the basic CISTA block, we implement the SR using CNN techniques. With more recursions used in CISTA, the network becomes deeper and tends to be bothered by the gradi- ent vanishing/exploding problems. Residual learning (Kim et al. 201 b Tai et al.) 2017a) such useful tool that not only mitigates these difficulties, but helps network converge faster (A.1)
4.1 CRNET-A MODEL FOR PRE-UPSAMPLING
As shown in Fig. 2 CRNet-A takes the ILR image I, with c channels as input, and predicts the output HR image I Two convolution layers, Fb R consisting of no filters of spatial size c x 8 x and Fi E R x Xaxe containing no filters of spatial size ny x x s are utilized for hierarchical features extraction from ILR image:
y ReLU (F @ (eL 2 I,)).
(11)
The ILR features are then fed into a CISTA block to learn the convolutional sparse codes. As stated in (10), two convolutional layers Wi E 2
and S E R2 mm m are needed:
2k+1 = ReL & + S z+),
(12)
where 2o is initialized to (W, @ y). The convolutional sparse codes learned after K recursions with S shared across every recursion. When the convolutional sparse codes & are obtained, it is then passed through convolution layer Wh E Te x recover the HR feature maps. The last convolution layer H E Rexng Xaxa is used as HR filters:
(13) Note that we pad zeros before all convolution operations to keep all the feature maps to have the same size, which is a common strategy used in variety of methods (Kim al. 22T Tai al. 2017ab. So the residual image R has the same size as the input ILR image Ly, and the final HR image I would be reconstructed by:
R= H @ ReLU(W,, & z).
I. Iy R.
(14)
4.2 CRNET-B MODEL FOR POST-UPSAMPLING
We extend CRNet-A its post-upsampling version to further mine its potential. Notice that most post-upsampling models (Ledig et al. 2017: Tong et al. 2017: Li et al. 2018 ZZhng et al., 2018) need train and store many scale-dependent models for various scales without fully using the inter- scale correlation, presented in MDSR (Lim et al. 2017) with minor modifications to address this issue. The complete model is shown in A.2 The main branch is our CRNet-A module. The pre-processing modules are used for reducing the variance from input images of different scales and only one residual unit with 3 x 3 kernels is used in each of the pre- pre-processing module. At the end of CRNet-B, upsampling modules are used for multi-scale reconstruction.
we adopt the scale-specific multi-path learning strategy (Wang al. 2019)
EXPERIMENTAL RESULTS
5.1
SETTINGS
Datasets and Metrics Being fair to (Kim et al. 2016a; Tai et al. 201 Zhang et al., 2018)
291 (Yang et al., 2010 Martin al. 2001) images are used for CRNet-A. while 800 (Timofte al.,
2017) images for C.Net-B. During testing, Set5 (Bevilacqua et al. 2012), Sei14 (Zeyde et al. 2010). B100 (Martin et al., 200T). 1rban100 (Huang et al. | 2015) and Manga109 099 (Matsui et al. 2017] are employed. Both PSNR and SSIM (Wang al. 2004) on Y channel of transformed YCbCr space are calculated for evaluation.
Parameter Settings We set no = 128, ma 256 for CRNet-A (i.e. every convolution layer contains 128 filters while W; and have 256 filters), while ny = 64, my = 1024 for CRNet-B. We choose K 25 in both of our models. We implement models using the PyTorch (Paszke al. 2017) framework with NVIDIA Titan Xp. More training details and parameter study could be found A.4 and A.5
5

---

Under review as a conference paper at ICLR 2020
Table 2: Average PSNR/SSIMs of Pre-upsampling models for scale factor x2, x3 and x4 on datasets Set5, Set14, BSD100 and Urban100. Red: the best; blue: the second best.
Dataser
Scale
Bicubic
Kim
VDSR
2016a
Kim of
DRCN
2016h
Thi et 2017a
DRRN
Thi al 2017h
MomNet
CRNer-A
333 95..669999999
37 5330, 958T
37. www.63009955
37. 37.74/0, 9591
37. 78/0 787099799 9597
37. 21..7990.900 9600
Set5
x3
50.39/0, 8682
33 66/0. 9213
33. 055.8200.222
34.0 34.03/0. 9244
34.090 9248
34. 1/0 9254
28.42/0.8 1044
31.35/0.8838
31.53/0.8652
31.68/0.8888
31,74/0 8893
31,8200.880 8907
x2
300.24/0.800 8G88
33.03/D. 91124
33.04/0.9118
33.23/0.9115
33.28/0.91
33.3,300911 9100
Se14
29.77/0.8314
29.76/0.8311
29. 29.96/0.8334
30.000.8300
29.99M
x4
000.0000.7025
28. 01/0. 7674
28. 7270.7670
28.21/0. 28/2
26.26/0. 26/0.77777
26, 288.2/00774 7741
x2
29..6/0084431
31 .900.996
31. 8660089942
32 32.05/0.8973
322 08/0, 5378
32. 022.0900.880 8985
B100
27.21/0.7388
28. 82/0. 7976
28. 09.95500.0 8000
25, (00.966000000
998.9900.0011
x4
25.96/0.6675
27/02/0011 7251
27. 22/30772333
27.36/0. 7284
27.40/0. 7281
27.4460 7302
26.88/0.8400
30.76/0 9:1-0
900.66009918
31.23/0 91188
81.33/0099150
31.36A0.9207
Urbon100
24.46/0.7348
27.14/0.8279
27. 37.16/0.8277
27.53/0.8375
27.560..837 8336
27.6-300. 84003
24.14/06377
25.18007024
25.14/0.7010
20.44/0. 7G38
25.300076630
25.09A017680
5.2 COMPARISON WITH CSC-SR
We first compare our proposed models with the existing CSC based image SR method, i.e., CSC- SR (Gu al.) 2015). Since CSC-SR utilizes LR images input image, it can be considered most-upsamplin method, thus CRNet-B is used for comparison. Tab. presents that our CRNet-B clearly outperforms CSC-SR by large margin.
Table 1: Left Table: Average PSNR/SSIMs of CSC-SR (Gu et al. 2015) and CRNet-B for scale factor x2, x3 and x4 on Set5, S and B100. Red: the best; blue: the second best. Right Table: Ablation Study of LRL,, PA and CC.
Scale
Bizshic
CSC-R
CRNer-R
LKL.
PA
CC
PSNR
33.66 33.269 0.9299
36.62/ (0959999
38.1310.9610
Set5
3
30.39 0.8682
32.65 7099998
34.7990.9296
X
33.53
x4
28.42/0.8011 0.8101
30.36 0.8607
32.57 0.8899
x
V
X
33.52
x 2
52.30/0.9070
34.09/0.9219
34,05
Sc:14
x 3
27.55 7,4.742
29.14/0.8300
30.58 0.8:66
V
V
x
33,58
27.30/0.7499
28.79/0.7867
34.09
2
29 (0.56//0 0.8431
31.27 8876
32.32 0.9004
33,97
B100
27.21 /00.385
28.31 /.77857
29,26/0.800
34.15
x4
25.96/0.6675
26.82 007.00
27.73/0.7414
5.3
COMPARISON WITH STATE OF THE ARTS
We now compare the proposed models with other state-of titte-of-thh-arts the-arts in recent years. We compare CRNet-A with pre-upsampling models (i.e., VDSR (Kim et al.) 2016a)). DRCN (Kim et al. 20166), DRRN Tai et al. 2017a). MemNet (Tai et al. (017b)) while CRNer-B with post-upsampling ar- chitectures (i.e., SRDenseNet (Tong al.) 2017), MSRN (Li al. 2018). D-DBPN (Haris al.)
2018 EDSR (Lim et al. 2017). RDN Zhang et al. 2018).. Similar to (Lim et al. 2017 Zhang al. 2018). self-ensemble strategy (Lim al. 2017) also adopted further improve the perfor- mance of CRNet-B, and we denote the sell-ensembled version as CRNet-B+.
Tab. and Tab. 3 show the quantitative comparisons on the benchmark testing sets. Both of our models achieve superior performance against the ofteeoofthheartt which indicates the effectiveness of our models. Qualitative results are provided in Fig. (3] Our methods tend to produce shaper edges and more correct textures, while other images may be blurred or distorted. More visual comparisons are available in A.6
Fig. 4(a) and 4(b) shows the performance versus the number of parameters, our CRNet-B and CRNet-B+ achieve better results with fewer parameters than EDSR (Lim et al. 2017) and RDN Zhang et al.) 2018). It's worth noting that EDSR/MDSR and RDN are far deeper than CRNet- (e.g., 169 vs. 36), but CRNet-B quite wider (W, and S have 1, 024 filters). As reported in (Lm et al. 2017), when increasing the number of filters to certain level, e.g., 256, the training procedure of EDSR (for x2) without residual scaling (Szegedy et al. 2018 Lim et al. 2017) is nu- merically unstable, shown in Fig. [(c) However, CRNet-B is relieved from the residual scaling

---

Under review as a conference paper at ICLR 2020
Ground Truth (PSNR, SSIM)
Bicubic
DRCN
DRRN (29.55. 0.9084)
CRNeí-A (29.96. 0.9112)
EDSR (30.60. 0.9171)
RDN (30.80. 0.9180)
CRNeI-B
(25.48.
(29.29. 0.9026)
(30.90,
0.8259)
0.9206)
CRNet-A
CRNet-B
(PSNR, SSIM)
(20.45.
(21.28.
(21.49.
(21.58
(22.71, 0.7905)
(22.43, 0.7831)
(23.07.
0.6556)
0.7238)
0.7391)
0.7423)
0.7994)
Figure 3: SR results of "img016" and "img059" from Urban with scale factor x4. Red indicates the best performance.
Table 3: Average PSNR/SSIMs of Post-upsampling models for scale factor x2, x3 and x4 on datasets Set5, Set14, BSD100, Urban 100 and Manga109, Red: the best; blue: the second best.
D-DBPN Haris 2018
EDSR
RDN
CRNet-B+ (ours)
Datnset
Scale
La al 2018
Lim et 2017
Zhang ct 2018
(ours)
x2
38. 09A1.9605
38. 38.090D99600
38.1 1/0. 9601
38. 29.24/999644
38. 38.13/0.946
38. 20..2500..55 96644
Set5
34.65/0. 92282
34.71/U. 9/291
34.75/0.91200
34.8000. wWA..993.3
x4
32. 022.07/0.880 8903
32 4784770.896
32.46/0. 8968
02.4770.8990
32.57/0.899
32.7 9008
11..7400.9190
33.85/0.91990
33.92/0.
34.01/0.921.2
992...92219
34.15/0.9
Secid
x3
30. 3...383399
-f-
30.52/0. 8462
90.37/0.8468
30.58/0. 00.56800-846
30.6770.844
6060.776
82/0.7800
28.800..
24.81/0.7877
28.7970.7867
7804
21.23/00000 90133
22.2770-900
901
32.38/0.00 02.38/0.90020 9020
8100
x3
29. 1943.8041
+/-
29.25/0. 2000
29, 8098
29. 221.2//0..000 8091
29.32/0.810 320
37.6241.7227
27.72/0.7400
27.72/0.
37.77407744
27.80/. 7-4300
x2
32. 20102/00000
22.2220..336 9326
32.55/0.9324
32.93/0. 9351
32.89/0-9358 9353
32. 322.930099355
33. 14901440..00 1.400993700 93370
100
8554
28.800. 8053
24.8000.8652
29. 099.0.8007777777
26. 049.088076
26. 7944
26.64/0. 8033
26.61/0.8022
26 69..6900.844
26.9090.800 8089
x2
38. 62/0 62/0.9668
9868
38 88.089909775
39. 10/0, 9773
39. 1870 9780
39 079.077..778
39. 29..200022 9784
Manga109 109
9476
9-196
x4
30.17/0.0034
30.91/D.9137
31.02/0.9148
31.0000.910
31.16/0.9154
81.55/0099188
trick.
The training loss of CRNet-B is depicted in Fig. 4(d) it converges fast at the begining, then keeps decreasing and finally fluctuates at certain range.
5.4 ABLATION STUDY
The proposed networks combine the merits of CSC and Residual learning. Particularly, CSC prob- lem is solved in learnable manner, namely CISTA, where three techniques are naturally inferred, ie. Local Residual Learning (LRL), Pre-Activation (PA) and Consistency Constraint (CC). Ablation studies implemented address the superiority of the proposed networks with the el- ementary blocks of CISTA by combining LRL, PA and CC. shown in Tab. that the network simultaneously containing LRI., PA and CC gives the best performance.

---

Under review as a conference paper at ICLR 2020
301
E350.43
12177
2718
V.78.50 193
244h
350
(a) Pre-upsampling (b) Post-upsampling
(c) EDSR
(d) CRNet-B
Figure 4: (a) and (b): PSNR of recent state-of-the-arts versus the number of parameters for scale factor x4 Set5. The number of layers marked in the parentheses; (c) and (d): Training loss of EDSR without residual scaling and CRNet-B.
DISCUSSIONS
We discuss the differences between proposed models and several recent CNN models for SR with recursive learning strategy, i.c., DRRN (Tai et al. (017a)), SCN (Wang et al. 2015) and DRCN (Kim al. 2016b). Due the fact that CRNet-B is extension of CRNet-A, I.e., the main part of CRNet-B has the same structure as CRNet-A, so we use CRNet-A here for comparison. The simplified structures of these models are shown in A.3 where the digits the left of the recursion line represent the number of recursions.
Difference to DRRN. The main part of DRRN Tai al. 2017ab is the recursive block structure, where several residual units with BN layers are stacked. On the other hand, guided by (10). CRNet- contains BN layers. Coinciding with EDSR/MDSR (Lim et al. 2017). by normalizing features, BN layers get rid of range flexibility from networks. Furthermore, BN consumes much amount of GPU memory and increases computational complexity. Experimental results on benchmark datasets under common used assessments demonstrate the superiority of CRNet-A.
Difference SCN.
There are two main differences between CRNet-A and SCN (Wang al. 2015):
CISTA block and residual learning. Specifically, CRNet-A takes consistency constraint into consid- eration with the help of CISTA block, while SCN uses linear layers and ignores the information from the consistency prior. On the other hand, CRNet-A adopts residual learning, which is powerful tool for training deeper networks. CRNet-A (30 layers) is much deeper than SCN (5 layers). As indicated in (Kim et al., 2016a), deeper network has larger receptive fileds, information in an image would be utilized to infer nigh-frequency details. In A.I more recursions, e.g., 48, can be used to achieve better performance.
contextual
we show that
Difference to DRCN. CRNet-A differs with DRCN (Kim et al. 20166) in two aspects: recursive block and training techniques. In the recursive block, both local residual learning (Tai et al., 2017a) and pre-activation (He et al.) 2016 Tai et al. 2017a) are utilized in CRNet-A, which are demon- strated be effective (Tai et 70) As for training techniques, DRCN not easy train, so recursive-spperviionn is introduced to facilitate the network to converge. Moreover, an ensemble strategy (the final output the weighted average of all intermediate predictions) is used further improve the performance. CRNet-A is relieved from these techniques and can be easily trained with more recursions.
CONCLUSIONS
In this work, propose two effective CSC based image SR models, i.e., CRNet-A and CRNet-B, for pre-fpott-ussapphe SR, respectively. By combining the merits of CSC and CNN, we achieve superior performance against recent -tate-of-the-arts. Furthermore, our framework and CISTA block are expected to be applicable in various CSC based tasks, though in this paper we focus on CSC based image SR.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In ECCV, 2018.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Aloeri-Mooel. Low- Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. BMVC, pp. 135.1-135. 10, 2012.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning, 3(1):1-122, 2011.
Hilton Bristow, Anders Eriksson, and Simon Lucey. Fast Convolutional Sparse Coding. In CVPR, 2013.
I Daubechies, M Defrise, and De Mol. An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Commuiccations on Pure and Applied Mathematics, 57(11): 1413-1457, 2004.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295-307, 2016.
Cristina Garcia-Cardona Cardona and Brendt Wohlberg. Convolutional Dictionary Learning: A Comparative Review and New Algorithms. IEEE Transactions on Computational Imaging, 3(3) 80
Cristina Garcia-Cardona and Brendt Wohlberg. Convolutional dictionary learning: comparative review and new algorithms. IEEE Transactions on Computational Imaging. 4(3):366-381, 2018b.
Karol Gregor and Yann LeCun. Learning Fast Approximations of Sparse Coding, In ICML, 2010.
Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangchu Feng, and Lei Zhang. Convolutional Sparse Coding for Image Super-Resolution. ICCV, 2015.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back pook-proecttion networks for super-resolution. In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers Surpass- ing Human-Level Performance on ImageNet Classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. Fast and flexible convolutional sparse coding. In CVPR, 2015.
Jia-Bin Huang, Abhishck Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In CVPR, 2015.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016a.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-Recursive Convolutional Network for Image Super-Resolution. In CVPR, 2016b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- Realistic Single Image Super- Resolutnn Using Generative Adversarial Network. In CVPR, 2017.
Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. Multi-scale residual network for image super-resolution. In ECCV, 2018.

---

Under review as a conference paper at ICLR 2020
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In CVPR Workshops, 2017.
Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS, 2016.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented
natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal 109 dataset. Multimedia Tools and Applications, 2017.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via con- volutional sparse coding. The Journal of Machine Learning Research, 18(1):2887-2338, 2017a. 2017a.
Vardan Papyan, Jeremias Sulam, and Michael Elad. Working locally thinking globally: Theoretical guarantees for convolutional sparse coding. IEEE Transactions on Signal Processing, 65(21): 5687-5701, 2017b.
Vardan Papyan, Yaniv Romano, Jeremias Sulam, and Michael Elad. Theoretical foundations of deep learning via sparse representations: A multilayer sparse model and its connection to convolutional neural networks. IEEE Signal Processing Magazine, 35(4)722-89, 2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detito, Zeming Lin, et al. Automatic differentiation in pytorch. In NIPS- W, 2017.
Hillel Sreter and Raja Giryes. Learned convolutional sparse coding. In ICASSP, 2018.
Christian Szegedy. Sergey Ioffe, and Vincent Vanhoucke. Inception-y4, inception-resnet and the impact of residual connections on learning. l 2018
Ying Tai, Jian Yang, and Xiaoming Liu. Image Super-Resolution via Deep Recursive Residual Network. In CVPR, 2017a.
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In ICCV, 2017b.
Radu Timofte, Vincent De Smet, and Lue Van Gool. A+: Adjusted anchored neighborhood regres- sion for fast super-resolution. In ACCV, 2014.
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Yang, Lei Zhang, et al. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPR Workshops, 2017.
Radu Timofte, Shuhang Gu, Jiqing Wu, and Lue Van Gool. NTIRE 2018 challenge on single image super-resolution: methods and results. In CVPR Workshops, 2018.
Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connec- tions. In ICCV, 2017.
Shenlong Wang, Lei Zhang, Yan Liang, and Quan Pan. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In CVPR, pp. 2216-2223, 2012.
Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas Huang. Deep networks for image super-resolution with sparse prior. In ICCV, 2015.
Zhihao Wang. Jian Chen, and Steven C.H. Hoi. Deep learning for image super-resolution: survey. arXiv: arXiv:/902.00 06068, 2019.
10

---

Under review as a conference paper at ICLR 2020
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, al. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.
Brendt Wohlberg. Efficient convolutional sparse coding. In ICASSP, 2014.
Brendt Wohlberg. Boundary handling for convolutional sparse representations. In
Chih-Yuse Yang, Chao Ma, and Ming-Hsuan Yang. Single-image super-resolution: A benchmark. In ECCV, 2014.
Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. Image super-resolution as sparse repre- sentation of raw image patches. In CVPR, 2008.
Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE TIP, 19(11):281-2877) 2010.
Jianchao Yang. Zhaowen Wang, Zhe Lin, Scott Cohen, and Thomas Huang. Coupled dictionary training for image super-resolution. IEEE TIP, 21(8):3467-3478 2012.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In CVPR, 2010.
Roman Zeyde, Michael Blad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Sprine.. 2010.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018.
Zheng Zhang, Yong Xu, Jian Yang, Xuelong Li, and David Zhang. A Survey Sparse Representa- tion Algorithms and Applications. IEEE Access, 3:490-530, 2015.

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1
RESIDUAL VS. RON-RESIDUAL
32
28
PSNR 480
26
24
22
22
18
Not-Reaida
Residral
20
Non Resitull
Redidual
16
10
30
Epochs
Epuccs
(a) K
(b)
Figure 5: Performance curve for residuaa//non-residue networks with different recursions. The tests are conducted on Set5 for scale factor x3.
A.2 CRNET-B
Reskkin Emit
Block
Con
F0
Rel 1
Rel. 1
Wy
Cony
Restint
ReLU
ReLU
W.
Rel
X4
Pre-processing Modules
Modules
ReLU
Cony
CanTy Shaffl
CNNY
Shuffle
COmY
Shartlle
Figure 6: The architecture of the post-upsampling model CRNet-B.
A.3 GRAPHICAL COMPARISON TO OTHER RECURRENT CNNs IN SR
Theokoki
Liwor
(a) DRRN
(b) SCN
(c) DRCN
(d) CRNet
Figure 7: Simplified network structures of (a) DRRN Tai et al. 2017a). (b) SCN WWag et al.
2015) (c) DRCN (Kim al. 2016b). (d) our model CRNet.
A.4 IMPLEMENTATION DETAILS
CRNet-A To enlarge the training data augmentation, which includes flipping (horizontally and vertically), rotating (90, 180, and 270 degrees), scaling (0.7, 0.5 and 0.4), is performed on each image of 29/ dataset, where 91 of these images from Yang al. (Yang et al., 2010) with the
12

---

Under review as a conference paper at ICLR 2020
addition of 200 images from Berkeley Segmentation Dataset (Martin al. 2001p. Scale augmen- tation is exploited by combining images of different scales (x2, x3 and x4) into one training set. Furthermore, all training images are partitioned into 33 x 33 patches with the stride of 33, provid- ing a total of 1, 929, 728 ILR-HR training pairs. Every convolution layer in CRNet-A contains 128 filters (ng- 128) of size 3 3 while Wi and S have 256 filters (mo 256). We follow the same strategy as He et al. (He et al. 2015) for weight initialization where all weights are drawn from a normal distribution with zero mean and variance 2/Tout where Nout is the number of output units. The network is optimized using SGD with mini-batch size of 128, momentum parameter of 0.9 and weight decay of 10 The learning rate is initially set to 0.1 and then decreased by factor of 10 every 10 epochs. We train total of 35 epochs as no further decrease of the loss can be observed. For maximal convergence speed, we utilize the adjustable gradient clipping strategy stated (Kim) et al. 2016a), with gradients clipped to [-0,0] 0). where 0 = 0.4 is the gradient clipping parameter.
CRNet-B We use the 800 training images of the DIV2K (Timofte et al.j 2017) dataset to train CRNet-B, and all the images are pre-processed by substracting the mean RGB value of the DIV2K dataset. Data augmentation includes random horizontal flips and rotations. Every weight layer in CRNet-B has 64 filters (no = 64) with the size of 3 x 3 except Wi and S have 1, 024 filters (my = 1, 024). CRNet-B is updated using Adam (Kingma & Ba 2014) with 31 = 0.9, 32 = 0.999, and 10-5, The mini-batch size is 16. We use RGB LR patches of size 54 x 54 as inputs. For each epoch, there are 10s iterations. The initial learning rate is 10-1 and halved every 200 epochs. We train CRNet-B for 800 epochs. Unlike CRNet-A, CRNet-B is trained using L1 loss for better convergence speed.
Recursion We choose K- 25 in both of our models. We implement our models using the PyTorch Paszke et al. (017) framework with NVIDIA Titan Xp. It takes approximately 4.5 days to train CRNet-A, and 15 days to train CRNet-B.
A.5 PARAMETER STUDY
The key parameters in both of our models are the number of filters (No, my) and recursions K.
Number of Filters We set ni = 128, my = 256, K = 25 for CRNet-A as stated in Section A.4 In Fig. 8(a) CRNet-A with different number of filters tested (DRCN (Kim et al., (016b) is used for reference). We find that even no is decreased from 128 to 64, the performance is not affected greatly. On the other hand, if we decrease mi from 256 to 128, the performance would suffer an obvious drop, but still better than DRCN (Kim et al. 20166). Based on these observations, we set the parameters of CRNet-B by making mo larger and no smaller for the trade off between model size and performance. Specifically, we use 70g = 64, mo 1024, K = 25 for CRNet-B. As shown in Fig. 8(b) the performance of CRNet-B can be significantly boosted with larger my (MDSR (Lim) et al. 2017) and MSRN (Li et al. 2018) are used for reference). Even with small mo. i.e., 256, al. 2018) with fewer parameters (2.0M vs. 6.1M).
CRNet-B still outperforms MSRN
Number of Recursions We also have trained and tested CRNet-A with 15, 20, 25, 48 recursions, so the depth of the these models are 20, 25, 30, 53 respectively. The results are presented in Fig. 9aa It's clear that CRNet-A with 20 layers still outperforms DRCN with the same depth and increasing K can promote the final performance. The results of using different recursions in CRNei-B are shown in Fig. 9b) which demonstrate that more recursions facilitate the performance improved.
A.6 MORE QUALITATIVE RESULTS
In Fig. 15 we provide additional visual results on benchmark datasets to clearly show the superiority of our proposed models. The pre-upsampling models (SRCNN (Dong al. 2016). VDSR (Kim et al. 201 DRCN (Kim et al., 2016b). DRRN Tai et al. 2017a). MemNet ( (Tai al. 2017b)) and the post- upsammpsnn models (MSRN (Li et 2018). EDSR/MDSR (Lim al., RDN (Zhang et al. 2018)) are used for comparisons. Red indicates the best performance.
2017
13

---

Under review as a conference paper at ICLR 2020
360
33.9
315
33.7
33.5
128, my 236
200
150
306
256
Pa Po- 3054
Fg 128, my 128
Pg D6- 540
64,
365
54.56 255
DR.CN
160
MISI
MARK
33.0
20
30
35
203
160
Epocts
(a) CRNet-A
(b) CRNet-B
Figure 8: PSNR of proposed models versus different number of filters on Set5 with scale factor x3.
34.5
33.9
33.2
32.5
20
750
32.0
10
280
Epecls
tpech
(a) CRNet-A
(b) CRNet-B
Figure 9: PSNR of proposed models versus different number of recursions on Set5 with scale factor x3.
Ground Truth (PSNR, SSIM)
Bicubic (21.11, 0.6810)
SRCNN (22.17. 0.7668)
VDSR (22.42, 0.7955)
DRCN (22.68. 0.8016)
DRRN (22.62,
CRNet-A (23.07,
0.8077)
0.8241)
MemNet (22.10, 0.8022)
MSRN (23.86, 0.8467)
EDSR (24.08, 0.8595)
MDSR (21.31, 0.8637)
RDN (21.13. 0.863-1
CRNet-B (24.40, 0.8640)
CRNet-B+ (24.88,
0.8736)
Figure 10: SR results of "img004" from Urban100 with scale factor x4.
14

---

Under review as a conference paper at ICLR 2020
Ground Truth (PSNR, SSIM)
Bicubic (25.47. 0.6149)
SRCNN (26.58, 0.6453)
VDSR (27.45, 0.6892)
DRCN (27.29, 0.6882)
DRRN (28.05, 0.7057)
CRNet-A (28.00, 0.7058)
MemNet (28.17, 0.7101)
MSRN (28.55. 0.7206)
EDSR (28.84, 0.7289)
MDSR (28.85.
RDN (28.68.
CRNet-B (29.01. 0.7330)
CRNet-B+ (29.10,
0.7282)
0.7254)
0.7345)
Figure 11: SR results of "img026" from Urban100 with scale factor x4.
Ground Truth (PSNR, SSIM)
Bicubic (28.48, 0.8120)
SRCNN
VDSR (31.01, 0.8683)
DRCN (30.78, 0.8638)
DRRN (31.08. 0.8699)
CRNet-A (31.30. 0.8761)
(30.23, 0.8502)
MemNet (31.13, 0.8694)
MSRN (31.57. 0.8791)
EDSR (31.95, 0.8859)
MDSR (31.85, 0.8849)
RDN (31.86,
CRNei-B (32.16, 0.8882)
CRNeI-B+ (32.24,
0.8834)
0.8900)
Figure 12: SR results of "img028" from Urban100 with scale factor x4.
15

---

Under review as a conference paper at ICLR 2020
Ground Truth (PSNR, SSIM)
Bicubic (19.87, 0.6467)
SRCNN (20.61, 0.7205)
VDSR (20.75, 0.7476)
DRCN (20.62, 0.7437)
DRRN (20.89, 0.7636)
CRNet-A (20.81, 0.7650)
MemNet
MSRN
EDSR (22.50. 0.8480)
MDSR (22.36,
RDN
CRNet-B B (22.72.
CRNct-B+ B++ (23.04, 0.8563)
(21.02.
(21.60. 0.8118)
(22.31, 0.8406)
0.7692)
0.8392)
0.8464)
Figure 13: SR results of "img062" from Urban100 with scale factor X 4.
PaiverPoint 2002
How
PriverPoint 2002
Aourt
PanverPaint 2002
PunverPaint 2002
Loor
PawerrPiitt 2002
How
PavverPuint 2002
PanverPoint 2000
Make Makr yake yourlake yaue your yourlake your
monninai
masniaat manninai mosninah monninai manninai manninat
Ground Truth (PSNR, SSIM)
Bicubic (26.95, 0.9481)
SRCNN
VDSR
DRCN
DRRN
CRNet-A
(31.65. 0.9809)
(32.81, 0.9871)
(32.39.
(34.00,
(34.24,
0.9869)
0.9904)
0.9910)
PaiverPoint Eonrythiee
PaverPPiitt 2002
PaverPPiitt 2002
PuyverPaint 2002
PuvvePPoint 2002
PavverPaint
PowverPoint
2002
Everything
Evermmen
2002
2002
Make yourl lake youreake Maue your yourlake your
mesnined manninad moaninei monnined monnined monninal monningl
MemNet
MSRN
EDSR
MDSR
RDN
CRNet-B (36.76.
CRNet-B+
(34.46,
(35.12,
(36.20,
(35.14.
(35.89.
(36.84,
0.9885)
0.9920)
0.9933)
0.9925)
0.9933)
0.9935)
0.9937)
Figure 14: SR results of "ppt3" from Set with scale factor x2.
16

---

Under review as a conference paper at ICLR 2020
Ground Truth (PSNR, SSIM)
Bicubic (28.50. 0.8288)
SRCNN
VDSR (29.54. 0.8654)
DRCN (30.29. 0.8656)
DRRN (29.70. 0.8673)
CRNet-A (30.54, 0.8704)
(29.40. 0.8565)
MemNet (30.25. 0.8701)
MSRN (30.95, 0.8778)
EDSR (31.18, 0.8797)
MDSR (31.12, 0.8791)
RDN (30.34.
CRNet-B (31.14, 0.8801)
CRNet-B+ (31.23, 0.8809)
0.8771)
Figure 15: SR results of "8023" from B100 with scale factor x4.
Ground Truth (PSNR, SSIM)
Bicubic 134.13. 0.6923)
SRCNN (25.00, 0.7450)
VDSR (25.66. 0.7769)
DRCN (25.69, 0.7776)
DRRN (26.10. 0.7938)
CRNet-A (26.17, 0.7956)
MemNet
MSRN (26.12) 12, 0.8006)
EDSR (26.61, 0.8193)
MDSR (26.53,
RDN (26 64. 0.8191)
CRNet-B (26.71. 0.8173)
CRNet-B+ (26.82, 0.8213)
(26.20, 0. 795988)
0.8114)
Figure 16: SR results "86000" from B100 with scale factor x4.
17


Reviews:
[{"review_id": "ryxv0ry1qr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a single image super-resolution method. The discussion refers to sparse coding as motivation and how such coding can be achieved within a neural network using activation functions, and then it slides into iterative soft thresholding and ends up with the observation that ReLU and the soft nonnegative thresholding operator are equal (paper should have pointed that they are equal with slight difference around the bias term). This argument is not new, this conclusion has already been known and analyzed extensively in (Papyan et al., 2017a).\n\nThe proposed network is not much different from what several other SR methods used before (see NTIRE 2019). It is basically composed of a collection of single layer residual units that share the same parameters. As expected, there is also a skip connection from the input to the last layer. The only slight variation is that the activation function ReLU for the residual layers are arranged before the convolutional layers. \n\nTraining details are missing. \n\nThe paper does not provide any comparisons with the top-ranking methods in the NTIRE 2019 single image super-resolution challenge leaderboard. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ryxYnhNCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates the problem of super-resolution using\nconvolutional sparse coding (CSC). The work is motivated\nby a reduction in optimization time stating that the\nstate-of-the-art CSC solvers are slow and require to\nput the entire image in memory. This is not true,\nthere exist online approaches such as https://arxiv.org/abs/1706.09563\nor distributed methods https://arxiv.org/abs/1901.09235.\n\nThe use of an unfolding algorithm similar to LISTA is\nnot clearly motivated and explained, especially concerning\nthe training procedure when working in the SR context.\n\nFinally the latex formatting suffers from many issues and typos.\nBe careful with difference between \\citet and \\citep for citations.\n\nFonts in figure 4 are too small\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJxXb346Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work exploits the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and state-of-the-art performance is achieved on popular benchmarks.\n\n[Strengths]\n- This paper is well-written and easy to follow.\n\n[Weaknesses]\n- My main concern on this work is its novelty seems to be limited. In Introduction, the authors raised 4 kinds of issues, including framework/optimization/memory/multi-scale. However, optimization and memory issues are mitigated by the CNN architectures; multi-scale issue is addressed with the help of work from Kim et al.. It seems the proposed questions are mostly addressed by the previous methods.\n\n- It seems CISTA may be the main novelty of this work. However, no comparasions with previous ISTA methods (e.g., [R1-3]) are conducted and discussed. \n\n- From Tabs. 2&3, the improvement is very limited. Besides, time complexity needs to be compared with the competitors, including flops or inference times.\n\n[R1] A Fast Proximal Method for Convolutional Sparse Coding, IJCNN 2013.\n[R2] Convolutional Neural Networks Analyzed via Convolutional Sparse Coding, JMLR 2017.\n[R3] On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks, TPAMI 2018.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers agreed that the paper should not be accepted. No rebuttal was offered, thus the paper is rejected.
Decision: Reject