A Deep Recurrent Neural Network via Unfolding Reweighted l1-l1 Minimization

1 INTRODUCTION . The problem of reconstructing sequential signals from low-dimensional measurements across time is of great importance for a number of applications such as time-series data analysis , future-frame prediction , and compressive video sensing . Specifically , we consider the problem of reconstructing a sequence of signals st ∈ Rn0 , t = 1 , 2 , . . . , T , from low-dimensional measurements xt = Ast , where A ∈ Rn×n0 ( n n0 ) is a sensing matrix . We assume that st has a sparse representation ht ∈ Rh in a dictionary D ∈ Rn0×h , that is , st = Dht . At each time step t , the signal st can be independently reconstructed using the measurements xt by solving ( Donoho , 2006 ) : min ht { 1 2 ‖xt −ADht‖22 + λ‖ht‖1 } , ( 1 ) where ‖ · ‖p is the ` p-norm and λ is a regularization parameter . The iterative shrinkage-thresholding algorithm ( ISTA ) ( Daubechies et al. , 2004 ) solves ( 1 ) by iterating over h ( l ) t = φλ c ( h ( l−1 ) t − 1 cD TAT ( ADh ( l−1 ) t − xt ) ) , where l is the iteration counter , φγ ( u ) = sign ( u ) [ 0 , |u| − γ ] + is the soft-thresholding operator , γ = λc , and c is an upper bound on the Lipschitz constant of the gradient of 12‖xt −ADht‖ 2 2 . Under the assumption that sequential signal instances are correlated , we consider the following sequential signal reconstruction problem : min ht { 1 2 ‖xt −ADht‖22 + λ1‖ht‖1 + λ2R ( ht , ht−1 ) } , ( 2 ) where λ1 , λ2 > 0 are regularization parameters and R ( ht , ht−1 ) is an added regularization term that expresses the similarity of the representations ht and ht−1 of two consecutive signals . Wisdom et al . ( 2017 ) proposed an RNN design ( coined Sista-RNN ) by unfolding the sequential version of ISTA . That study assumed that two consecutive signals are close in the ` 2-norm sense , formally , R ( ht , ht−1 ) = 1 2‖Dht − FDht−1‖ 2 2 , where F ∈ Rn0×n0 is a correlation matrix between st and st−1 . More recently , the study by Le et al . ( 2019 ) designed the ` 1- ` 1-RNN , which stems from unfolding an algorithm that solves the ` 1- ` 1 minimization problem ( Mota et al. , 2017 ; 2016 ) . This is a version of Problem ( 2 ) with R ( ht , ht−1 ) = ‖ht −Ght−1‖1 , where G ∈ Rh×h is an affine transformation that promotes the correlation between ht and ht−1 . Both studies ( Wisdom et al. , 2017 ; Le et al. , 2019 ) have shown that carefully-designed deep RNN models outperform the generic RNN model and ISTA ( Daubechies et al. , 2004 ) in the task of sequential frame reconstruction . Deep neural networks ( DNN ) have achieved state-of-the-art performance in solving ( 1 ) for individual signals , both in terms of accuracy and inference speed ( Mousavi et al. , 2015 ) . However , these models are often criticized for their lack of interpretability and theoretical guarantees ( Lucas et al. , 2018 ) . Motivated by this , several studies focus on designing DNNs that incorporate domain knowledge , namely , signal priors . These include deep unfolding methods which design neural networks to learn approximations of iterative optimization algorithms . Examples of this approach are LISTA ( Gregor & LeCun , 2010 ) and its variants , including ADMM-Net ( Sun et al. , 2016 ) , AMP ( Borgerding et al. , 2017 ) , and an unfolded version of the iterative hard thresholding algorithm ( Xin et al. , 2016 ) . LISTA ( Gregor & LeCun , 2010 ) unrolls the iterations of ISTA into a feed-forward neural network with weights , where each layer implements an iteration : h ( l ) t = φγ ( l ) ( W ( l ) h ( l−1 ) t + U ( l ) xt ) , with W ( l ) = I − 1cD TATAD , U ( l ) = 1cD TAT , and γ ( l ) being learned from data . It has been shown ( Gregor & LeCun , 2010 ; Sprechmann et al. , 2015 ) that a d-layer LISTA network with trainable parameters Θ = { W ( l ) , U ( l ) , γ ( l ) } dl=1 achieves the same performance as the original ISTA but with much fewer iterations ( i.e. , number of layers ) . Recent studies ( Chen et al. , 2018 ; Liu et al. , 2019 ) have found that exploiting dependencies between W ( l ) and U ( l ) leads to reducing the number of trainable parameters while retaining the performance of LISTA . These works provided theoretical insights to the convergence conditions of LISTA . However , the problem of designing deep unfolding methods for dealing with sequential signals is significantly less explored . In this work , we will consider a deep RNN for solving Problem ( 2 ) that outputs a sequence , ŝ1 , . . . , ŝT from an input measurement sequence , x1 , . . . , xT , as following : ht = φγ ( Wht−1 + Uxt ) , ŝt = Dht . ( 3 ) It has been shown that reweighted algorithms—such as the reweighted ` 1 minimization method by Candès et al . ( 2008 ) and the reweighted ` 1- ` 1 minimization by Luong et al . ( 2018 ) —outperform their non-reweighted counterparts . Driven by this observation , this paper proposes a novel deep RNN architecture by unfolding a reweighted- ` 1- ` 1 minimization algorithm . Due to the reweighting , our network has higher expressivity than existing RNN models leading to better data representations , especially when depth increases . This is in line with recent studies ( He et al. , 2016 ; Cortes et al . ; Huang et al. , 2017 ) , which have shown that better performance can be achieved by highly overparameterized networks , i.e. , networks with far more parameters than the number of training samples . While the most recent studies ( related over-parameterized DNNs ) consider fully-connected networks applied on classification problems ( Neyshabur et al. , 2019 ) , our approach focuses on deep-unfolding architectures and opts to understand how the networks learn a low-complexity representation for sequential signal reconstruction , which is a regression problem across time . Furthermore , while there have been efforts to build deep RNNs ( Pascanu et al. , 2014 ; Li et al. , 2018 ; Luo et al. , 2017 ; Wisdom et al. , 2017 ) , examining the generalization property of such deep RNN models on unseen sequential data still remains elusive . In this work , we derive the generalization error bound of the proposed design and further compare it with existing RNN bounds ( Zhang et al. , 2018 ; Kusupati et al. , 2018 ) . Contributions . The contributions of this work are as follows : • We propose a principled deep RNN model for sequential signal reconstruction by unfolding a reweighted ` 1- ` 1 minimization method . Our reweighted-RNN model employs different soft-thresholding functions that are adaptively learned per hidden unit . Furthermore , the proposed model is over-parameterized , has high expressivity and can be efficiently stacked . • We derive the generalization error bound of the proposed model ( and deep RNNs ) by measuring Rademacher complexity and show that the over-parameterization of our RNN ensures good generalization . To best of our knowledge , this is the first generalization error bound for deep RNNs ; moreover , our bound is tighter than existing bounds derived for shallow RNNs ( Zhang et al. , 2018 ; Kusupati et al. , 2018 ) . • We provide experiments in the task of reconstructing video sequences from low-dimensional measurements . We show significant gains when using our model compared to several state-of-the-art RNNs ( including unfolding architectures ) , especially when the depth of RNNs increases . 2 A DEEP RNN VIA UNFOLDING REWEIGHTED- ` 1- ` 1 MINIMIZATION In this section , we describe a reweighted ` 1- ` 1 minimization problem for sequential signal reconstruction and propose an iterative algorithm based on the proximal method . We then design a deep RNN architecture by unfolding this algorithm . The proposed reweighted ` 1- ` 1 minimization . We introduce the following problem : min ht { 1 2 ‖xt −ADZht‖22 + λ1‖g ◦ Zht‖1 + λ2‖g ◦ ( Zht −Ght−1 ) ‖1 } , ( 4 ) where “ ◦ ” denotes element-wise multiplication , g ∈ Rh is a vector of positive weights , Z ∈ Rh×h is a reweighting matrix , and G ∈ Rh×h is an affine transformation that promotes the correlation between ht−1 and ht . Intuitively , Z is adopted to transform ht to Zht ∈ Rh , producing a reweighted version of it . Thereafter , g aims to reweight each transformed component of Zht and Zht −Ght−1 in the ` 1-norm regularization terms . Because of applying reweighting ( Candès et al. , 2008 ) , the solution of Problem ( 4 ) is a more accurate sparse representation compared to the solution of the ` 1- ` 1 minimization problem in Le et al . ( 2019 ) ( where Z = I and g = I ) . Furthermore , the use of the reweighting matrix Z to transform ht to Zht differentiates Problem ( 4 ) from the reweighted ` 1- ` 1 minimization problem in Luong et al . ( 2018 ) where Z = I . The objective function in ( 4 ) consists of the differentiable fidelity term f ( Zht ) = 12‖xt−ADZht‖ 2 2 and the non-smooth term g ( Zht ) = λ1‖g ◦ Zht‖1 + λ2‖g ◦ ( Zht −Ght−1 ) ‖1 . We use a proximal gradient method ( Beck & Teboulle , 2009 ) to solve ( 4 ) : At iteration l , we first update h ( l−1 ) t —after being multiplied by Zl—with a gradient descent step on the fidelity term as u = Zlh ( l−1 ) t − 1 cZl∇f ( h ( l−1 ) t ) , where ∇f ( h ( l−1 ) t ) = D TAT ( ADh ( l−1 ) t − xt ) . Then , h ( l ) t is updated as h ( l ) t = Φλ1 c gl , λ2 c gl , Ght−1 ( Zlh ( l−1 ) t − 1 c Zl∇f ( h ( l−1 ) t ) ) , ( 5 ) where the proximal operator Φλ1 c gl , λ2 c gl , Ght−1 ( u ) is defined as Φλ1 c gl , λ2 c gl , ~ ( u ) = arg min v∈Rh { 1 c g ( v ) + 1 2 ||v − u||22 } , ( 6 ) with ~ = Ght−1 . Since the minimization problem is separable , we can minimize ( 6 ) independently for each of the elements gl , ~ , u of the corresponding gl , ~ , u vectors . After solving ( 6 ) , we obtain Φλ1 c gl , λ2 c gl , ~ ( u ) [ for solving ( 6 ) , we refer to Proposition B.1 in Appendix B ] . For ~ ≥ 0 : Φλ1 c gl , λ2 c gl , ~≥0 ( u ) = u− λ1c gl − λ2 c gl , ~ + λ1 c gl + λ2 c gl < u < ∞ ~ , ~ + λ1c gl − λ2 c gl ≤ u ≤ ~ + λ1 c gl + λ2 c gl u− λ1c gl + λ2 c gl , λ1 c gl − λ2 c gl < u < ~ + λ1 c gl − λ2 c gl 0 , −λ1c gl − λ2 c gl ≤ u ≤ λ1 c gl − λ2 c gl u+ λ1c gl + λ2 c gl , −∞ < u < − λ1 c gl − λ2 c gl , ( 7 ) and for ~ < 0 : Φλ1 c gl , λ2 c gl , ~ < 0 ( u ) = u− λ1c gl − λ2 c gl , λ1 c gl + λ2 c gl < u < ∞ 0 , −λ1c gl + λ2 c gl ≤ u ≤ λ1 c gl + λ2 c gl u+ λ1c gl − λ2 c gl , ~− λ1 c gl + λ2 c gl < u < − λ1 c gl + λ2 c gl ~ , ~− λ1c gl − λ2 c gl ≤ u ≤ ~− λ1 c gl + λ2 c gl u− λ1c gl + λ2 c gl , −∞ < u < ~− λ1 c gl − λ2 c gl ( 8 ) Under review as a conference paper at ICLR 2020 ℏ − − 1 2 ℏ − + 1 2 ( ) , , ℏ 1 2 ℏ + 1 2 − + 1 2 0 ℏ + + 1 2 − − 1 2 ( ) , , ℏ 1 2 ℏ + − 1 2 ℏ − 1 2 0 Algorithm 1 : The proposed algorithm for sequential signal reconstruction . 1 Input : Measurements x1 , . . . , xT , measurement matrix A , dictionary D , affine transform G , initial h ( d ) 0 ≡ h0 , reweighting matrices Z1 , . . . , Zd and vectors g1 , . . . , gd , c , λ1 , λ2 . 2 Output : Sequence of sparse codes h1 , . . . , hT . 3 for t = 1 , . . . , T do 4 h ( 0 ) t = Gh ( d ) t−1 5 for l = 1 to d do 6 u = [ Zl − 1cZlD TATAD ] h ( l−1 ) t + 1cZlD TATxt 7 h ( l ) t = Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 ( u ) 8 end 9 end 10 return h ( d ) 1 , . . . , h ( d ) T Fig . 1 depicts the proximal operators for ~ ≥ 0 and ~ < 0 . Observe that different values of gl lead to different shapes of the proximal functions Φλ1 c gl , λ2 c gl , ~ ( u ) for each element u of u . Our iterative algorithm is given in Algorithm 1 . We reconstruct a sequence h1 , . . . , hT from a sequence of measurements x1 , . . . , xT . For each time step t , Step 6 applies a gradient descent update for f ( Zht−1 ) and Step 7 applies the proximal operator Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 element-wise to the result . Let us compare the proposed method against the algorithm in Le et al . ( 2019 ) —which resulted in the ` 1- ` 1-RNN—that solves the ` 1- ` 1 minimization in Mota et al . ( 2016 ) ( where Zl = I and gl = I ) . In that algorithm , the update terms in Step 6 , namely I− 1cD TATAD and 1cD TAT , and the proximal operator in Step 7 are the same for all iterations of l. In contrast , Algorithm 1 uses a different Zl matrix per iteration to reparameterize the update terms ( Step 6 ) and , through updating gl , it applies a different proximal operator to each element u ( in Step 7 ) per iteration l. The proposed reweighted-RNN . We now describe the proposed architecture for sequential signal recovery , designed by unrolling the steps of Algorithm 1 across the iterations l = 1 , . . . , d ( yielding the hidden layers ) and time steps t = 1 , . . . , T . Specifically , the l-th hidden layer is given by h ( l ) t = Φλ1c g1 , λ2c g1 , Gh ( d ) t−1 ( W1h ( d ) t−1 + U1xt ) , if l = 1 , Φλ1 c gl , λ2 c gl , Gh ( d ) t−1 ( Wlh ( l−1 ) t + Ulxt ) , if l > 1 , ( 9 ) and the reconstructed signal at time step t is given by ŝt = Dh ( d ) t ; where Ul , Wl , V are defined as Ul = 1 c ZlD TAT , ∀l , ( 10 ) W1 = Z1G− 1 c Z1DTATADG , ( 11 ) Wl = Zl − 1 c ZlD TATAD , l > 1 . ( 12 ) The activation function is the proximal operator Φλ1 c gl , λ2 c gl , ~ ( u ) with learnable parameters λ1 , λ2 , c , gl ( see Fig . 1 for the shapes of the activation functions ) . Under review as a conference paper at ICLR 2020 11/12/2019 Copy of Copy of Copy of deepRNN.drawio 3/4 𝐡 ( 𝑑 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐖2 𝐠1 𝐆 ... ... ... 𝐬𝑡 𝐀 𝐱𝑡 𝐔1 𝐔2 𝐔𝑑 𝐬 ̂ 𝑡 𝐆 𝐆 𝐖3 𝐖𝑑 𝐃 𝐠2 𝐠𝑑 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐖1 𝐖2 𝐆 ... ... ... 𝐬𝑡 𝐀 𝐱𝑡 𝐔1 𝐔1 𝐔1 𝐬 ̂ 𝑡 𝐆 𝐆 𝐖2 𝐖2 𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 1 ) 𝑡−1 𝐡 ( 1 ) 𝑡 𝐡 ( 2 ) 𝑡 𝐕1 𝐖2 ... ... 𝐱𝑡 𝐔1 𝐖3 𝐖𝑑 𝐃 𝐡 ( 𝑑 ) 𝑡 𝐡 ( 𝑑 ) 𝑡 𝐬𝑡 𝐬 ̂ 𝑡 𝐀 𝐡 ( 2 ) 𝑡−1 𝐕2 𝐡 ( 𝑑 ) 𝑡−1 𝐕𝑑 𝐖1 𝐡0 𝐡 ( 1 ) 1 𝐡 ( 2 ) 1 𝐖2 𝐠1 𝐆 ... ... ... 𝐬1 𝐀 𝐱1 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐃 𝐠2 𝐠𝑑 𝐬 ̂ 1 𝐡 ( 𝑑 ) 1 𝐡 ( 1 ) 2 𝐡 ( 2 ) 2 𝐖2 𝐠1 𝐆 ... ... ... 𝐱2 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐠2 𝐠𝑑 𝐬2 𝐬 ̂ 2𝐡 ( 𝑑 ) 2 𝐡 ( 𝑑 ) 1 𝐡 ( 𝑑 ) 𝑇−1 𝐡 ( 1 ) 𝑇 𝐡 ( 2 ) 𝑇 𝐖2 𝐠1 𝐆 ... ... ... 𝐱𝑇 𝐔1 𝐔2 𝐔𝑑 𝐆 𝐆 𝐖3 𝐖𝑑 𝐠2 𝐠𝑑 𝐬𝑇 𝐬 ̂ 𝑇𝐡 ( 𝑑 ) 𝑇 ... ... ... ... . ... 𝐀 𝐀 𝐃 𝐃 𝐖1 𝐖1 𝐖1 ( a ) The proposed reweighted-RNN 5/14/2019 deepRNN.drawio 1/2 x1s1 A U1 W2 h1 ( 2 ) g2 h1 ( 1 ) U2 Wd h1 ( d ) Ud D s1 W1 h0 G x2s2 A U1 W2 h2 ( 2 ) h2 ( 1 ) U2 Wd h2 ( d ) Ud D s2 W1 G W1 G xTA U1 W2 hT ( 2 ) hT ( 1 ) U2 Wd hT ( d ) Ud D sT W1 G ... sT ... ... ...... ... ... . ... ... ... . ... ... ... ^ ^ ^ g1 gd g2g1 gd g2g1 gd h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 g1 ˤ , , G ̌ 1 c ̌ 2 c h ( d ) t−1 G ... ... ... st A xt U1 U2 Ud s ̂ t G G W3 Wd D g2 gd h ( d ) t h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 G ... ... ... st A xt U1 U1 U1 s ̂ t G G W2 W2 D h ( d ) t ( b ) ` 1- ` 1-RNN . 5/16/2019 deepRNN.drawio 1/2 x1s1 A U1 W2 h1 ( 2 ) g2 h1 ( 1 ) U2 Wd h1 ( d ) Ud D s1 W1 h0 G x2s2 A U1 W2 h2 ( 2 ) h2 ( 1 ) U2 Wd h2 ( d ) Ud D s2 W1 G W1 G xTA U1 W2 hT ( 2 ) hT ( 1 ) U2 Wd hT ( ) Ud D sT W1 G ... sT ... ... ...... ... ... . ... ... ... . ... ... ... ^ ^ ^ g1 gd g2g1 gd g2g1 gd h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 g1 G ... ... ... st A xt U1 U2 Ud s ̂ t G G W3 Wd D g2 gd h ( d ) t h ( d ) t−1 h ( 1 ) t h ( 2 ) t W1 W2 G ... ... ... st A xt U1 U1 U1 s ̂ t G G W2 W2 D h ( d ) t h ( 1 ) t−1 h ( 1 ) t h ( 2 ) t V1 W2 ... ... xt U1 W3 Wd D h ( d ) t h ( d ) t st s ̂ t A h ( 2 ) t−1 V2 h ( d ) t−1 Vd ( c ) Stacked RNN . Figure 2 : The proposed ( a ) reweighted-RNN vs. ( b ) ` 1- ` 1-RNN and ( c ) Stacked RNN with d layers . Fig . 2 ( a ) depicts the architecture of the proposed reweighted-RNN . Input vectors st , t = 1 , . . . , T are compressed by a linear measurement layer A , resulting in compressive measurements xt . The reconstructed vectors ŝt , t = 1 , . . . , T , are obtained by multiplying linearly the hidden representation h ( d ) t with the dictionary D. We train our network in an end-to-end fashion . During training , we minimize the loss function L ( Θ ) = E s1 , ··· , sT [ ∑T t=1 ‖st − ŝt‖22 ] using stochastic gradient descent on mini- batches , where the trainable parameters are Θ = { A , D , G , h0 , Z1 , . . . , Zd , g1 , . . . , gd , c , λ1 , λ2 } . We now compare the proposed reweighted-RNN [ Fig . 2 ( a ) ] against the recent ` 1- ` 1-RNN ( Le et al. , 2019 ) [ Fig . 2 ( b ) ] . The l-th hidden layer in ` 1- ` 1-RNN is given by h ( l ) t = Φλ1c , λ2c , Gh ( d ) t−1 ( W1h ( d ) t−1 + U1xt ) , if l = 1 , Φλ1 c , λ2 c , Gh ( d ) t−1 ( W2h ( l−1 ) t + U1xt ) , if l > 1 . ( 13 ) The proposed model has the following advantages over ` 1- ` 1-RNN . Firstly , ` 1- ` 1-RNN uses the proximal operator Φλ1 c , λ2 c , ~ ( u ) as activation function , whose learnable parameters λ1 , λ2 are fixed across the network . Conversely , the corresponding parameters λ1c gl and λ2 c gl [ see ( 7 ) , ( 8 ) , and Fig . 1 ] in our proximal operator , Φλ1 c gl , λ2 c gl , ~ ( u ) , are learned for each hidden layer due to the reweighting vector gl ; hence , the proposed model has a different activation function for each unit per layer . The second difference comes from the set of parameters { Wl , Ul } in ( 13 ) and ( 9 ) . The ` 1- ` 1-RNN model uses the same { W2 , U1 } for the second and higher layers . In contrast , our reweighted-RNN has different sets of { Wl , Ul } per hidden layer due to the reweighting matrix Zl . These two aspects [ which are schematically highlighted in blue fonts in Fig . 2 ( a ) ] can lead to an increase in the learning capability of the proposed reweighted-RNN , especially when the depth of the model increases . In comparison to a generic stacked RNN ( Pascanu et al. , 2014 ) [ Fig . 2 ( c ) ] , reweighted-RNN promotes the inherent data structure , that is , each vector st has a sparse representation ht and consecutive ht ’ s are correlated . This design characteristic of the reweighted-RNN leads to residual connections which reduce the risk of vanishing gradients during training [ the same idea has been shown in several works ( He et al. , 2016 ; Huang et al. , 2017 ) in deep neural network literature ] . Furthermore , in ( 10 ) and ( 12 ) , we see a weight coupling of Wl and Ul ( due to the shared components of A , D and Z ) . This coupling satisfies the necessary condition of the convergence in Chen et al . ( 2018 ) ( Theorem 1 ) . Using Theorem 2 in Chen et al . ( 2018 ) , it can be shown that reweighted-RNN , in theory , needs a smaller number of iterations ( i.e. , d in Algorithm 1 ) to reach convergence , compared to ISTA ( Daubechies et al. , 2004 ) and FISTA ( Beck & Teboulle , 2009 ) .

Summary: This paper proposes a new reweighted-RNN by unfolding a reweighted L1-L1 minimization problem. It develops an iterative algorithm to solve the reweighted L1-L1 minimization problem, where the soft-thresholding functions can be adaptively learned. This paper provides the generalization error bound for deep RNNs and shows that the proposed reweighted-RNN has a lower generalization error bound. In addition, the paper shows that the proposed algorithm can be applied to video-frame reconstruction and achieves favorable results against state-of-the-art methods. The paper is well organized, and the motivation is clear.