CRNet: Image Super-Resolution Using A Convolutional Sparse Coding  Inspired Network

1 INTRODUCTION . Single Image Super-Resolution ( SISR ) , which aims to restore a visually pleasing High-Resolution ( HR ) image from its Low-Resolution ( LR ) version , is still a challenging task within computer vision research community ( Timofte et al. , 2017 ; 2018 ) . Since multiple solutions exist for the mapping from LR to HR space , SISR is highly ill-posed . To regularize the solution of SISR , various priors of natural images have been exploited , especially the current leading learning-based methods ( Wang et al. , 2015 ; Dong et al. , 2016 ; Mao et al. , 2016 ; Kim et al. , 2016a ; b ; Tai et al. , 2017a ; b ; Lim et al. , 2017 ; Ahn et al. , 2018 ; Haris et al. , 2018 ; Li et al. , 2018 ; Zhang et al. , 2018 ) are proposed to directly learn the non-linear LR-HR mapping . By modeling the sparse prior in natural images , the Sparse Coding ( SC ) based methods for SR ( Yang et al. , 2008 ; 2010 ; 2014 ) with strong theoretical support are widely used owing to their excellent performance . Considering the complexity in images , these methods divide the image into overlapping patches and aim to jointly train two over-complete dictionaries for LR/HR patches . There are usually three steps in these methods ’ framework . First , overlapping patches are extracted from input image . Then to reconstruct the HR patch , the sparse representation of LR patch can be applied to the HR dictionary with the assumption that LR/HR patch pair shares similar sparse representation . The final HR image is produced by aggregating the recovered HR patches . Recently , with the development of Deep Learning ( DL ) , many researchers attempt to combine the advantages of DL and SC for image SR. Dong et al . ( Dong et al. , 2016 ) firstly proposed the seminal CNN model for SR termed as SRCNN , which exploits a shallow convolutional neural network to learn a nonlinear LR-HR mapping in an end-to-end manner and dramatically overshadows conventional methods ( Yang et al. , 2010 ; Timofte et al. , 2014 ) . However , sparse prior is ignored to a large extent in SRCNN for it adopts a generic architecture without considering the domain expertise . To address this issue , Wang et al . ( Wang et al. , 2015 ) implemented a Sparse Coding based Network ( SCN ) for image SR , by combining the merits of sparse coding and deep learning , which fully exploits the approximation of sparse coding learned from the LISTA ( Gregor & LeCun , 2010 ) based sub-network . It ’ s worth to note that most of SC based methods utilize the sparse prior locally ( Papyan et al. , 2017b ) , i.e. , coping with overlapping image patches . Thus the consistency of pixels in overlapped patches has been ignored ( Gu et al. , 2015 ; Papyan et al. , 2017b ) . To address this issue , CSC is proposed to serve sparse prior as a global prior ( Zeiler et al. , 2010 ; Papyan et al. , 2017b ; 2018 ) and it furnishes a way to fill the local-global gap by working directly on the entire image by convolution operation . Consequently , CSC has attained much attention from researchers ( Zeiler et al. , 2010 ; Bristow et al. , 2013 ; Heide et al. , 2015 ; Gu et al. , 2015 ; Sreter & Giryes , 2018 ; Garcia-Cardona & Wohlberg , 2018a ) . However , very few studies focus on the validation of CSC for image SR ( Gu et al. , 2015 ) , resulting in no work been reported that CSC based image SR can achieve state-of-theart performance . Can CSC based image SR show highly competitive results with recent state-of-theart methods ( Dong et al. , 2016 ; Mao et al. , 2016 ; Kim et al. , 2016a ; b ; Tai et al. , 2017a ; b ; Tong et al. , 2017 ; Lim et al. , 2017 ; Li et al. , 2018 ; Zhang et al. , 2018 ) ? To answer this question , the following issues need to be considered : Framework Issue . Compared with SC based image SR methods ( Yang et al. , 2008 ; 2010 ) , the lack of a unified framework has hindered progress towards improving the performance of CSC based image SR. Optimization Issue . The previous CSC based image SR method ( Gu et al. , 2015 ) contains several steps and they are optimized independently . Hundreds of iterations are required to solve the CSC problem in each step . Memory Issue . To solve the CSC problem , ADMM ( Boyd et al. , 2011 ) is commonly employed ( Bristow et al. , 2013 ; Wohlberg , 2014 ; Heide et al. , 2015 ; Wohlberg , 2016 ; Garcia-Cardona & Wohlberg , 2018a ) , where the whole training set needs to be loaded in memory . As a consequence , it is not applicable to improve the performance by enlarging the training set . Multi-Scale Issue . Training a single model for multiple scales is difficult for the previous CSC based image SR method ( Gu et al. , 2015 ) . Based on these considerations , in this paper , we attempt to answer the aforementioned question . Specifically , we exploit the advantages of CSC and the powerful learning ability of deep learning to address image SR problem . Moreover , massive theoretical foundations for CSC ( Papyan et al. , 2017b ; 2018 ; Garcia-Cardona & Wohlberg , 2018b ) make our proposed architectures interpretable and also enable to theoretically analyze our SR performance . In the rest of this paper , we first introduce CISTA , which can be naturally implemented using CNN architectures for solving the CSC problem . Then we develop a framework for CSC based image SR , which can address the Framework Issue . Subsequently , CRNet-A ( CSC and Residual learning based Network ) and CRNet-B inspired by this framework are proposed for image SR . They are classified as pre- and post-upsampling models ( Wang et al. , 2019 ) respectively , as the former takes Interpolated LR ( ILR ) images as input while the latter processes LR images directly . By adopting CNN architectures , Optimization Issue and Memory Issue would be mitigated to some extent . For Multi-Scale Issue , with the help of the recently introduced scale augmentation ( Kim et al. , 2016a ; b ) or scale-specific multi-path learning ( Lim et al. , 2017 ; Wang et al. , 2019 ) strategies , both of our models are capable of handling multi- scale SR problem effectively , and achieve favorable performance against state-of-the-arts , as shown in Fig . 1 . The main contributions of this paper include : • We introduce CISTA , which can be naturally implemented using CNN architectures for solving the CSC problem . • A novel framework for CSC based image SR is developed . Two models , CRNet-A and CRNet-B , inspired by this framework are proposed for image SR. • The differences between our proposed models and several SR models with recursive learning strategy , e.g. , DRRN ( Tai et al. , 2017a ) , SCN ( Wang et al. , 2015 ) , DRCN ( Kim et al. , 2016b ) , are discussed . 2 RELATED WORK . Sparse coding has been widely used in a variety of applications ( Zhang et al. , 2015 ) . As for SISR , Yang et al . ( Yang et al. , 2008 ) proposed a representative Sparse coding based Super-Resolution ( ScSR ) method . In the training stage , ScSR attempts to learn the LR/HR overcomplete dictionary pair Dl/Dh jointly by given a group of LR/HR training patch pairs xl/xh . In the test stage , the HR patch xh is reconstructed from its LR version xl by assuming they share the same sparse code . Specifically , the optimal sparse code is obainted by minimizing the following sparsity-inducing ` 1- norm regularized objective function z∗ = argmin z ‖xl −Dlz‖22 + λ‖z‖1 , ( 1 ) then the HR patch is obtained by xh = Dhz∗ . Finally , the HR image can be estimated by aggregating all the reconstructed HR patches . Inspired by ScSR , many SC based methods have been proposed with various constraints on sparse code or dictionary Yang et al . ( 2012 ) ; Wang et al . ( 2012 ) . Traditional SC based SR algorithms usually process images in a patch based manner to reduce the burden of modeling and computation , resulting in the inconsistency problem ( Papyan et al. , 2017b ) . As a special case of SC , CSC is inherently suitable ( Zeiler et al. , 2010 ) and proposed to avoid the inconsistency problem by representing the whole image directly . Specifically , an image y ∈ Rnr×nc can be represented as the summation of m feature maps zi ∈ Rnr×nc convolved with the corresponding filters fi ∈ Rs×s : y = ∑m i=1 fi ⊗ zi , where ⊗ is the convolution operation . Gu et al . ( Gu et al. , 2015 ) proposed the CSC-SR method and revealed the potential of CSC for image SR . In ( Gu et al. , 2015 ) , CSC-SR requires to solve the following CSC based optimization problem in both the training and testing phase : min f , z 1 2 ∥∥∥∥∥y − m∑ i=1 fi ⊗ zi ∥∥∥∥∥ 2 2 + λ m∑ i=1 ‖zi‖1 . ( 2 ) which is solved by alternatively optimizing the z and f subproblems ( Wohlberg , 2014 ) . The z subproblem is a standard CSC problem . Hundreds of iterations are required to solve the CSC problem and the aforementioned Optimization Issue and Memory Issue can not be completely avoided . Inspired by the success of deep learning based sparse coding ( Gregor & LeCun , 2010 ) , we exploit the natural connection between CSC and CNN to solve the CSC problem efficiently . 3 CISTA FOR SOLVING CSC PROBLEM . CSC can be considered as a special case of conventional SC , due to the fact that convolution operation can be replaced with matrix multiplication , so the objective function of CSC can be formulated as : min z ∥∥∥∥∥y − m∑ i=1 Fizi ∥∥∥∥∥ 2 2 + λ m∑ i=1 ‖zi‖1 . ( 3 ) y , zi are in vectorized form and Fi is a sparse convolution matrix with the following attributes : Fizi ≡ fi ⊗ zi F Ti zi ≡ flipud ( fliplr ( fi ) ) ⊗ zi ≡ flip ( fi ) ⊗ zi ( 4 ) where fliplr ( · ) and flipud ( · ) are following the notations of Zeiler et al . ( Zeiler et al. , 2010 ) , representing that array is flipped in left/right or up/down direction . Iterative Soft Thresholding Algorithm ( ISTA ) ( Daubechies et al. , 2004 ) can be utilized to solve ( 3 ) , at the kth iteration : zk+1 = hθ ( zk + 1 L F T ( y − Fzk ) ) ( 5 ) where L is the Lipschitz constant , F = [ F1 , F2 , . . . , Fm ] and Fz = ∑m i=1 Fizi . Using the relation in ( 4 ) to replace the matrix multiplication with convolution operator , we can reformulate ( 5 ) as : zk+1 = hθ ( Izk + 1 L flip ( f ) ⊗ ( y − f ⊗ zk ) ) ( 6 ) where f = [ f1 , f2 , . . . , fm ] , flip ( f ) = [ flip ( f1 ) , flip ( f2 ) , . . . , flip ( fm ) ] and I is the identity matrix . Note that identity matrix I is also a sparse convolution matrix , so according to ( 4 ) , there existing a filter n satisfies : Iz = n⊗ z , ( 7 ) so ( 6 ) becomes : zk+1 = hθ ( W ⊗ y + S ⊗ zk ) , ( 8 ) whereW = 1Lflip ( f ) and S = n− 1 Lflip ( f ) ⊗ f . Even though ( 3 ) is for a single image with one channel , the extension to multiple channels ( for both image and filters ) and multiple images is mathematically straightforward . Thus for y ∈ Rb×c×nr×nc representing b images of size nr × nc with c channels , ( 8 ) is still true withW ∈ Rm×c×s×s and S ∈ Rm×m×s×s . As for hθ , ( Papyan et al. , 2017a ) reveals two important facts : ( 1 ) the expressiveness of the sparsity inspired model is not affected even by restricting the coefficients to be nonnegative ; ( 2 ) the ReLU ( Nair & Hinton , 2010 ) activation function and the soft nonnegative thresholding operator are equal , that is : h+θ ( α ) = max ( α− θ , 0 ) = ReLU ( α− θ ) . ( 9 ) We set θ = 0 for simplicity . So the final form of ( 8 ) is : zk+1 = ReLU ( W ⊗ y + S ⊗ zk ) . ( 10 ) One can see that ( 10 ) is a convolutional form of ( 5 ) , so we name it as CISTA . It provides the solution of ( 3 ) with theoretical guarantees ( Daubechies et al. , 2004 ) . Furthermore , this convolutional form can be implemented employing CNN architectures . SoW and S in ( 10 ) would be trainable .

Summary: This paper presents a single image super-resolution method. The discussion refers to sparse coding as motivation and how such coding can be achieved within a neural network using activation functions, and then it slides into iterative soft thresholding and ends up with the observation that ReLU and the soft nonnegative thresholding operator are equal (paper should have pointed that they are equal with slight difference around the bias term). This argument is not new, this conclusion has already been known and analyzed extensively in (Papyan et al., 2017a).