A Deep Recurrent Neural Network via Unfolding Reweighted l1-l1 Minimization

1 INTRODUCTION . The problem of reconstructing sequential signals from low-dimensional measurements across time is of great importance for a number of applications such as time-series data analysis , future-frame prediction , and compressive video sensing . Specifically , we consider the problem of reconstructing a sequence of signals st ∈ Rn0 , t = 1 , 2 , . . . , T , from low-dimensional measurements xt = Ast , where A ∈ Rn×n0 ( n n0 ) is a sensing matrix . We assume that st has a sparse representation ht ∈ Rh in a dictionary D ∈ Rn0×h , that is , st = Dht . At each time step t , the signal st can be independently reconstructed using the measurements xt by solving ( Donoho , 2006 ) : min ht { 1 2 ‖xt −ADht‖22 + λ‖ht‖1 } , ( 1 ) where ‖ · ‖p is the ` p-norm and λ is a regularization parameter . The iterative shrinkage-thresholding algorithm ( ISTA ) ( Daubechies et al. , 2004 ) solves ( 1 ) by iterating over h ( l ) t = φλ c ( h ( l−1 ) t − 1 cD TAT ( ADh ( l−1 ) t − xt ) ) , where l is the iteration counter , φγ ( u ) = sign ( u ) [ 0 , |u| − γ ] + is the soft-thresholding operator , γ = λc , and c is an upper bound on the Lipschitz constant of the gradient of 12‖xt −ADht‖ 2 2 . Under the assumption that sequential signal instances are correlated , we consider the following sequential signal reconstruction problem : min ht { 1 2 ‖xt −ADht‖22 + λ1‖ht‖1 + λ2R ( ht , ht−1 ) } , ( 2 ) where λ1 , λ2 > 0 are regularization parameters and R ( ht , ht−1 ) is an added regularization term that expresses the similarity of the representations ht and ht−1 of two consecutive signals . Wisdom et al . ( 2017 ) proposed an RNN design ( coined Sista-RNN ) by unfolding the sequential version of ISTA . That study assumed that two consecutive signals are close in the ` 2-norm sense , formally , R ( ht , ht−1 ) = 1 2‖Dht − FDht−1‖ 2 2 , where F ∈ Rn0×n0 is a correlation matrix between st and st−1 . More recently , the study by Le et al . ( 2019 ) designed the ` 1- ` 1-RNN , which stems from unfolding an algorithm that solves the ` 1- ` 1 minimization problem ( Mota et al. , 2017 ; 2016 ) . This is a version of Problem ( 2 ) with R ( ht , ht−1 ) = ‖ht −Ght−1‖1 , where G ∈ Rh×h is an affine transformation that promotes the correlation between ht and ht−1 . Both studies ( Wisdom et al. , 2017 ; Le et al. , 2019 ) have shown that carefully-designed deep RNN models outperform the generic RNN model and ISTA ( Daubechies et al. , 2004 ) in the task of sequential frame reconstruction . Deep neural networks ( DNN ) have achieved state-of-the-art performance in solving ( 1 ) for individual signals , both in terms of accuracy and inference speed ( Mousavi et al. , 2015 ) . However , these models are often criticized for their lack of interpretability and theoretical guarantees ( Lucas et al. , 2018 ) . Motivated by this , several studies focus on designing DNNs that incorporate domain knowledge , namely , signal priors . These include deep unfolding methods which design neural networks to learn approximations of iterative optimization algorithms . Examples of this approach are LISTA ( Gregor & LeCun , 2010 ) and its variants , including ADMM-Net ( Sun et al. , 2016 ) , AMP ( Borgerding et al. , 2017 ) , and an unfolded version of the iterative hard thresholding algorithm ( Xin et al. , 2016 ) . LISTA ( Gregor & LeCun , 2010 ) unrolls the iterations of ISTA into a feed-forward neural network with weights , where each layer implements an iteration : h ( l ) t = φγ ( l ) ( W ( l ) h ( l−1 ) t + U ( l ) xt ) , with W ( l ) = I − 1cD TATAD , U ( l ) = 1cD TAT , and γ ( l ) being learned from data . It has been shown ( Gregor & LeCun , 2010 ; Sprechmann et al. , 2015 ) that a d-layer LISTA network with trainable parameters Θ = { W ( l ) , U ( l ) , γ ( l ) } dl=1 achieves the same performance as the original ISTA but with much fewer iterations ( i.e. , number of layers ) . Recent studies ( Chen et al. , 2018 ; Liu et al. , 2019 ) have found that exploiting dependencies between W ( l ) and U ( l ) leads to reducing the number of trainable parameters while retaining the performance of LISTA . These works provided theoretical insights to the convergence conditions of LISTA . However , the problem of designing deep unfolding methods for dealing with sequential signals is significantly less explored . In this work , we will consider a deep RNN for solving Problem ( 2 ) that outputs a sequence , ŝ1 , . . . , ŝT from an input measurement sequence , x1 , . . . , xT , as following : ht = φγ ( Wht−1 + Uxt ) , ŝt = Dht . ( 3 ) It has been shown that reweighted algorithms—such as the reweighted ` 1 minimization method by Candès et al . ( 2008 ) and the reweighted ` 1- ` 1 minimization by Luong et al . ( 2018 ) —outperform their non-reweighted counterparts . Driven by this observation , this paper proposes a novel deep RNN architecture by unfolding a reweighted- ` 1- ` 1 minimization algorithm . Due to the reweighting , our network has higher expressivity than existing RNN models leading to better data representations , especially when depth increases . This is in line with recent studies ( He et al. , 2016 ; Cortes et al . ; Huang et al. , 2017 ) , which have shown that better performance can be achieved by highly overparameterized networks , i.e. , networks with far more parameters than the number of training samples . While the most recent studies ( related over-parameterized DNNs ) consider fully-connected networks applied on classification problems ( Neyshabur et al. , 2019 ) , our approach focuses on deep-unfolding architectures and opts to understand how the networks learn a low-complexity representation for sequential signal reconstruction , which is a regression problem across time . Furthermore , while there have been efforts to build deep RNNs ( Pascanu et al. , 2014 ; Li et al. , 2018 ; Luo et al. , 2017 ; Wisdom et al. , 2017 ) , examining the generalization property of such deep RNN models on unseen sequential data still remains elusive . In this work , we derive the generalization error bound of the proposed design and further compare it with existing RNN bounds ( Zhang et al. , 2018 ; Kusupati et al. , 2018 ) . Contributions . The contributions of this work are as follows : • We propose a principled deep RNN model for sequential signal reconstruction by unfolding a reweighted ` 1- ` 1 minimization method . Our reweighted-RNN model employs different soft-thresholding functions that are adaptively learned per hidden unit . Furthermore , the proposed model is over-parameterized , has high expressivity and can be efficiently stacked . • We derive the generalization error bound of the proposed model ( and deep RNNs ) by measuring Rademacher complexity and show that the over-parameterization of our RNN ensures good generalization . To best of our knowledge , this is the first generalization error bound for deep RNNs ; moreover , our bound is tighter than existing bounds derived for shallow RNNs ( Zhang et al. , 2018 ; Kusupati et al. , 2018 ) . • We provide experiments in the task of reconstructing video sequences from low-dimensional measurements . We show significant gains when using our model compared to several state-of-the-art RNNs ( including unfolding architectures ) , especially when the depth of RNNs increases . 2 A DEEP RNN VIA UNFOLDING REWEIGHTED- ` 1- ` 1 MINIMIZATION In this section , we describe a reweighted ` 1- ` 1 minimization problem for sequential signal reconstruction and propose an iterative algorithm based on the proximal method . We then design a deep RNN architecture by unfolding this algorithm . The proposed reweighted ` 1- ` 1 minimization . We introduce the following problem : min ht { 1 2 ‖xt −ADZht‖22 + λ1‖g ◦ Zht‖1 + λ2‖g ◦ ( Zht −Ght−1 ) ‖1 } , ( 4 ) where “ ◦ ” denotes element-wise multiplication , g ∈ Rh is a vector of positive weights , Z ∈ Rh×h is a reweighting matrix , and G ∈ Rh×h is an affine transformation that promotes the correlation between ht−1 and ht . Intuitively , Z is adopted to transform ht to Zht ∈ Rh , producing a reweighted version of it . Thereafter , g aims to reweight each transformed component of Zht and Zht −Ght−1 in the ` 1-norm regularization terms . Because of applying reweighting ( Candès et al. , 2008 ) , the solution of Problem ( 4 ) is a more accurate sparse representation compared to the solution of the ` 1- ` 1 minimization problem in Le et al . ( 2019 ) ( where Z = I and g = I ) . Furthermore , the use of the reweighting matrix Z to transform ht to Zht differentiates Problem ( 4 ) from the reweighted ` 1- ` 1 minimization problem in Luong et al . ( 2018 ) where Z = I . The objective function in ( 4 ) consists of the differentiable fidelity term f ( Zht ) = 12‖xt−ADZht‖ 2 2 and the non-smooth term g ( Zht ) = λ1‖g ◦ Zht‖1 + λ2‖g ◦ ( Zht −Ght−1 ) ‖1 . We use a proximal gradient method ( Beck & Teboulle , 2009 ) to solve ( 4 ) : At iteration l , we first update h ( l−1 ) t —after being multiplied by Zl—with a gradient descent step on the fidelity term as u = Zlh ( l−1 ) t − 1 cZl∇f ( h ( l−1 ) t ) , where ∇f ( h ( l−1 ) t ) = D TAT ( ADh ( l−1 ) t − xt ) . Then , h ( l ) t is updated as h ( l ) t = Φλ1 c gl , λ2 c gl , Ght−1 ( Zlh ( l−1 ) t − 1 c Zl∇f ( h ( l−1 ) t ) ) , ( 5 ) where the proximal operator Φλ1 c gl , λ2 c gl , Ght−1 ( u ) is defined as Φλ1 c gl , λ2 c gl , ~ ( u ) = arg min v∈Rh { 1 c g ( v ) + 1 2 ||v − u||22 } , ( 6 ) with ~ = Ght−1 . Since the minimization problem is separable , we can minimize ( 6 ) independently for each of the elements gl , ~ , u of the corresponding gl , ~ , u vectors . After solving ( 6 ) , we obtain Φλ1 c gl , λ2 c gl , ~ ( u ) [ for solving ( 6 ) , we refer to Proposition B.1 in Appendix B ] . For ~ ≥ 0 : Φλ1 c gl , λ2 c gl , ~≥0 ( u ) = u− λ1c gl − λ2 c gl , ~ + λ1 c gl + λ2 c gl < u < ∞ ~ , ~ + λ1c gl − λ2 c gl ≤ u ≤ ~ + λ1 c gl + λ2 c gl u− λ1c gl + λ2 c gl , λ1 c gl − λ2 c gl < u < ~ + λ1 c gl − λ2 c gl 0 , −λ1c gl − λ2 c gl ≤ u ≤ λ1 c gl − λ2 c gl u+ λ1c gl + λ2 c gl , −∞ < u < − λ1 c gl − λ2 c gl , ( 7 ) and for ~ < 0 : Φλ1 c gl , λ2 c gl , ~ < 0 ( u ) = u− λ1c gl − λ2 c gl , λ1 c gl + λ2 c gl < u < ∞ 0 , −λ1c gl + λ2 c gl ≤ u ≤ λ1 c gl + λ2 c gl u+ λ1c gl − λ2 c gl , ~− λ1 c gl + λ2 c gl < u < − λ1 c gl + λ2 c gl ~ , ~− λ1c gl − λ2 c gl ≤ u ≤ ~− λ1 c gl + λ2 c gl u− λ1c gl + λ2 c gl , −∞ < u < ~− λ1 c gl − λ2 c gl ( 8 ) Under review as a conference paper at ICLR 2020 ℏ − − 1 2 ℏ − + 1 2 ( ) , , ℏ 1 2 ℏ + 1 2 − + 1 2 0 ℏ + + 1 2 − − 1 2 ( ) , , ℏ 1 2 ℏ + − 1 2 ℏ − 1 2 0 Algorithm 1 : The proposed algorithm for sequential signal reconstruction . 1 Input : Measurements x1 , . . . , xT , measurement matrix A , dictionary D , affine transform G , initial h ( d ) 0 ≡ h0 , reweighting matrices Z1 , . . . , Zd and vectors g1 , . . . , gd , c , λ1 , λ2 . 2 Output : Sequence of sparse codes h1 , . . . , hT . 3 for t = 1 , . . . , T do 4 h ( 0 ) t = Gh ( d ) t−1 5 for l = 1 to d do 6 u = [ Zl − 1cZlD TATAD ] h ( l−1 ) t + 1cZlD TATxt 7 h ( l ) t = Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 ( u ) 8 end 9 end 10 return h ( d ) 1 , . . . , h ( d ) T Fig . 1 depicts the proximal operators for ~ ≥ 0 and ~ < 0 . Observe that different values of gl lead to different shapes of the proximal functions Φλ1 c gl , λ2 c gl , ~ ( u ) for each element u of u . Our iterative algorithm is given in Algorithm 1 . We reconstruct a sequence h1 , . . . , hT from a sequence of measurements x1 , . . . , xT . For each time step t , Step 6 applies a gradient descent update for f ( Zht−1 ) and Step 7 applies the proximal operator Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 element-wise to the result . Let us compare the proposed method against the algorithm in Le et al . ( 2019 ) —which resulted in the ` 1- ` 1-RNN—that solves the ` 1- ` 1 minimization in Mota et al . ( 2016 ) ( where Zl = I and gl = I ) . In that algorithm , the update terms in Step 6 , namely I− 1cD TATAD and 1cD TAT , and the proximal operator in Step 7 are the same for all iterations of l. In contrast , Algorithm 1 uses a different Zl matrix per iteration to reparameterize the update terms ( Step 6 ) and , through updating gl , it applies a different proximal operator to each element u ( in Step 7 ) per iteration l. The proposed reweighted-RNN . We now describe the proposed architecture for sequential signal recovery , designed by unrolling the steps of Algorithm 1 across the iterations l = 1 , . . . , d ( yielding the hidden layers ) and time steps t = 1 , . . . , T . Specifically , the l-th hidden layer is given by h ( l ) t = Φλ1c g1 , λ2c g1 , Gh ( d ) t−1 ( W1h ( d ) t−1 + U1xt ) , if l = 1 , Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 ( Wlh ( l−1 ) t + Ulxt ) , if l > 1 , ( 9 ) and the reconstructed signal at time step t is given by ŝt = Dh ( d ) t ; where Ul , Wl , V are defined as Ul = 1 c ZlD TAT , ∀l , ( 10 ) W1 = Z1G− 1 c Z1DTATADG , ( 11 ) Wl = Zl − 1 c ZlD TATAD , l > 1 . ( 12 ) The activation function is the proximal operator Φλ1 c gl , λ2 c gl , ~ ( u ) with learnable parameters λ1 , λ2 , c , gl ( see Fig . 1 for the shapes of the activation functions ) . Under review as a conference paper at ICLR 2020 11/12/2019 Copy of Copy of Copy of deepRNN.drawio 3/4 𝐡 ( 𝑑 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐖2 𝐠1 𝐆 ... ... ... 𝐬𝑡 𝐀 𝐱𝑡 𝐔1 𝐔2 𝐔𝑑 𝐬 ̂ 𝑡 𝐆 𝐆 𝐖3 𝐖𝑑 𝐃 𝐠2 𝐠𝑑 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐖1 𝐖2 𝐆 ... ... ... 𝐬𝑡 𝐀 𝐱𝑡 𝐔1 𝐔1 𝐔1 𝐬 ̂ 𝑡 𝐆 𝐆 𝐖2 𝐖2 𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 1 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐕1 𝐖2 ... ... 𝐱𝑡 𝐔1 𝐖3 𝐖𝑑 𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡 𝐬𝑡 𝐬 ̂ 𝑡 𝐀 𝐡 ( 2 ) 𝑡−1 𝐕2 𝐡 ( 𝑑 ) 𝑡−1 𝐕𝑑 𝐖1 𝐡0 𝐡 ( 1 ) 1 𝐡 ( 2 ) 1 𝐖2 𝐠1 𝐆 ... ... ... 𝐬1 𝐀 𝐱1 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐃 𝐠2 𝐠𝑑 𝐬 ̂ 1 𝐡 ( 𝑑 ) 1 𝐡 ( 1 ) 2 𝐡 ( 2 ) 2 𝐖2 𝐠1 𝐆 ... ... ... 𝐱2 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐠2 𝐠𝑑 𝐬2 𝐬 ̂ 2𝐡 ( 𝑑 ) 2 𝐡 ( 𝑑 ) 1 𝐡 ( 𝑑 ) 𝑇−1 𝐡 ( 1 ) 𝑇 𝐡 ( 2 ) 𝑇 𝐖2 𝐠1 𝐆 ... ... ... 𝐱𝑇 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐠2 𝐠𝑑 𝐬𝑇 𝐬 ̂ 𝑇𝐡 ( 𝑑 ) 𝑇 ... ... ... ... . ... 𝐀 𝐀 𝐃 𝐃 𝐖1 𝐖1 𝐖1 ( a ) The proposed reweighted-RNN 5/14/2019 deepRNN.drawio 1/2 x1s1 A U1 W2 h1 ( 2 ) g2 h1 ( 1 ) U2 Wd h1 ( d ) Ud D s1 W1 h0 G x2s2 A U1 W2 h2 ( 2 ) h2 ( 1 ) U2 Wd h2 ( d ) Ud D s2 W1 G W1 G xTA U1 W2 hT ( 2 ) hT ( 1 ) U2 Wd hT ( d ) Ud D sT W1 G ... sT ... ... ...... ... ... . ... ... ... . ... ... ... ^ ^ ^ g1 gd g2g1 gd g2g1 gd h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 g1 ˤ , , G ̌ 1 c ̌ 2 c h ( d ) t−1 G ... ... ... st A xt U1 U2 Ud s ̂ t G G W3 Wd D g2 gd h ( d ) t h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 G ... ... ... st A xt U1 U1 U1 s ̂ t G G W2 W2 D h ( d ) t ( b ) ` 1- ` 1-RNN . 5/16/2019 deepRNN.drawio 1/2 x1s1 A U1 W2 h1 ( 2 ) g2 h1 ( 1 ) U2 Wd h1 ( d ) Ud D s1 W1 h0 G x2s2 A U1 W2 h2 ( 2 ) h2 ( 1 ) U2 Wd h2 ( d ) Ud D s2 W1 G W1 G xTA U1 W2 hT ( 2 ) hT ( 1 ) U2 Wd hT ( ) Ud D sT W1 G ... sT ... ... ...... ... ... . ... ... ... . ... ... ... ^ ^ ^ g1 gd g2g1 gd g2g1 gd h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 g1 G ... ... ... st A xt U1 U2 Ud s ̂ t G G W3 Wd D g2 gd h ( d ) t h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 G ... ... ... st A xt U1 U1 U1 s ̂ t G G W2 W2 D h ( d ) t h ( 1 ) t−1 h ( 1 ) t h ( 2 ) t V1 W2 ... ... xt U1 W3 Wd D h ( d ) t h ( d ) t st s ̂ t A h ( 2 ) t−1 V2 h ( d ) t−1 Vd ( c ) Stacked RNN . Figure 2 : The proposed ( a ) reweighted-RNN vs. ( b ) ` 1- ` 1-RNN and ( c ) Stacked RNN with d layers . Fig . 2 ( a ) depicts the architecture of the proposed reweighted-RNN . Input vectors st , t = 1 , . . . , T are compressed by a linear measurement layer A , resulting in compressive measurements xt . The reconstructed vectors ŝt , t = 1 , . . . , T , are obtained by multiplying linearly the hidden representation h ( d ) t with the dictionary D. We train our network in an end-to-end fashion . During training , we minimize the loss function L ( Θ ) = E s1 , ··· , sT [ ∑T t=1 ‖st − ŝt‖22 ] using stochastic gradient descent on mini- batches , where the trainable parameters are Θ = { A , D , G , h0 , Z1 , . . . , Zd , g1 , . . . , gd , c , λ1 , λ2 } . We now compare the proposed reweighted-RNN [ Fig . 2 ( a ) ] against the recent ` 1- ` 1-RNN ( Le et al. , 2019 ) [ Fig . 2 ( b ) ] . The l-th hidden layer in ` 1- ` 1-RNN is given by h ( l ) t = Φλ1c , λ2c , Gh ( d ) t−1 ( W1h ( d ) t−1 + U1xt ) , if l = 1 , Φλ1 c , λ2 c , Gh ( d ) t−1 ( W2h ( l−1 ) t + U1xt ) , if l > 1 . ( 13 ) The proposed model has the following advantages over ` 1- ` 1-RNN . Firstly , ` 1- ` 1-RNN uses the proximal operator Φλ1 c , λ2 c , ~ ( u ) as activation function , whose learnable parameters λ1 , λ2 are fixed across the network . Conversely , the corresponding parameters λ1c gl and λ2 c gl [ see ( 7 ) , ( 8 ) , and Fig . 1 ] in our proximal operator , Φλ1 c gl , λ2 c gl , ~ ( u ) , are learned for each hidden layer due to the reweighting vector gl ; hence , the proposed model has a different activation function for each unit per layer . The second difference comes from the set of parameters { Wl , Ul } in ( 13 ) and ( 9 ) . The ` 1- ` 1-RNN model uses the same { W2 , U1 } for the second and higher layers . In contrast , our reweighted-RNN has different sets of { Wl , Ul } per hidden layer due to the reweighting matrix Zl . These two aspects [ which are schematically highlighted in blue fonts in Fig . 2 ( a ) ] can lead to an increase in the learning capability of the proposed reweighted-RNN , especially when the depth of the model increases . In comparison to a generic stacked RNN ( Pascanu et al. , 2014 ) [ Fig . 2 ( c ) ] , reweighted-RNN promotes the inherent data structure , that is , each vector st has a sparse representation ht and consecutive ht ’ s are correlated . This design characteristic of the reweighted-RNN leads to residual connections which reduce the risk of vanishing gradients during training [ the same idea has been shown in several works ( He et al. , 2016 ; Huang et al. , 2017 ) in deep neural network literature ] . Furthermore , in ( 10 ) and ( 12 ) , we see a weight coupling of Wl and Ul ( due to the shared components of A , D and Z ) . This coupling satisfies the necessary condition of the convergence in Chen et al . ( 2018 ) ( Theorem 1 ) . Using Theorem 2 in Chen et al . ( 2018 ) , it can be shown that reweighted-RNN , in theory , needs a smaller number of iterations ( i.e. , d in Algorithm 1 ) to reach convergence , compared to ISTA ( Daubechies et al. , 2004 ) and FISTA ( Beck & Teboulle , 2009 ) .

Summary: This paper proposes a novel method to solve the sequential signal reconstruction problem. The method is based on the deep unfolding methods and incorporates the reweighting mechanism. Additionally, they derive the generalization error bound and show how their over-parameterized reweighting RNNs ensure good generalization. Lastly, the experiments on the task of video sequence reconstruction suggest the superior performance of the proposed method.