Target Propagation via Regularized Inversion

1 INTRODUCTION . Target propagation algorithms can be seen as perturbative learning alternatives to the gradient backpropagation algorithm , where virtual targets are propagated backward instead of gradients ( Le Cun , 1986 ; Le Cun et al. , 1989 ; Rohwer , 1990 ; Mirowski & LeCun , 2009 ; Bengio , 2014 ; Goodfellow et al. , 2016 ) . A high-level summary is presented in Fig . 1 : while gradient back-propagation considers storing intermediate gradients in a forward pass , target propagation algorithms proceed by computing and storing approximate inverses . The approximate inverses are then passed on backward along the graph of computations to finally yield a weight update for stochastic learning . Target propagation aims to take advantage of the availability of approximate inverses to compute better descent directions for the objective at hand . Bengio et al . ( 2013 ) ; Bengio ( 2020 ) argued that the approach could be relevant for problems involving multiple compositions such as the training of Recurrent Neural Networks ( RNNs ) , which generally suffer from the phenomenon of exploding or vanishing gradients ( Hochreiter , 1998 ; Bengio et al. , 1994 ; ? ) . Recently , empirical results indeed showed the potential advantages of target propagation over classical gradient back-propagation for training RNNs on several tasks ( Manchev & Spratling , 2020 ) . However , these recent investigations remain built on multiple approximations , which hinder the analysis of the core idea of TP , i.e. , using layer inverses . On the theoretical side , difference target propagation , a modern variant of target propagation , was related to an approximate Gauss-Newton method , suggesting interesting venues to explain the benefits of target propagation ( Bengio , 2020 ; Meulemans et al. , 2020 ) . Previous works have considered approximating inverses by adding multiple reverse layers ( Manchev & Spratling , 2020 ; Meulemans et al. , 2020 ; Bengio , 2020 ) . However , it is unclear whether such reverse layers actually learn layer inverses during the training process . Even if they were , the additional cost of computational complexity of learning approximate inverses should be carefully accounted for . In this work , we propose a simple target propagation approach , revisiting the original insights of Le Cun et al . ( 1989 ) on the critical importance of the good conditioning of layer inverses . We define regularized inverses through a variational formulation and we obtain approximate inverses via these regularized inverses . In this spirit , we can also interpret the difference target propagation formula ( Lee et al. , 2015 ) as a finite difference approximation of a linearized regularized inverse . We propose a smoother formula that can directly be integrated into a differentiable programming framework . G adien Back-P o aga ion Ta ge P o aga ion We detail the computational complexity of the proposed target propagation and compare it to the one of gradient back-propagation , showing that the additional cost of computing inverses can be effectively amortized for very long sequences . Following the benchmark of Manchev & Spratling ( 2020 ) , we observe that the proposed target propagation can perform better than classical gradientbased methods on several tasks involving RNNs . Related work . Many variations of back-propagation algorithms have been explored ; see Werbos ( 1994 ) ; Goodfellow et al . ( 2016 ) for an extensive bibliography . Closer to target propagation , penalized formulations of the training problem have been considered to decouple the optimization of the weights in a distributed way or using an ADMM approach ( Carreira-Perpinan & Wang , 2014 ; Taylor et al. , 2016 ; Gotmare et al. , 2018 ) . Rather than modifying the backward operations in the layers , one can also modify the weight updates for deep forward networks by using a regularized inverse ( Frerix et al. , 2018 ) . Wiseman et al . ( 2017 ) recast target propagation as an ADMM-like algorithm for language modeling and reported disappointing experimental results . Recently , in a careful experimental benchmark evaluation , Manchev & Spratling ( 2020 ) explored further target propagation to train RNNs , mapping a sequence to a single final output , in an attempt to understand the benefits of target propagation to capture long-range dependencies , and obtained promising experimental results . Another line of research has considered synthetic gradients that approximate gradients using an additional layer instead of using back-propagated gradients ( Jaderberg et al. , 2017 ; Czarnecki et al. , 2017 ) to speed up the training of deep neural networks . Recently , Ahmad et al . ( 2020 ) ; Dalm et al . ( 2021 ) considered using analytical inverses to implement target propagation and blend it with what they called a gradient-adjusted incremental formula . Yet , an additional orthogonality penalty is critical for their approach to work . Recently , Meulemans et al . ( 2020 ) considered using as many reverse layers as forwarding operations . We focus here on the optimization gains of using target propagation that can not be obtained by adding a prohibitive number of reverse layers . Finally , we do not discuss the biological plausibility of TP since we are unable to comment on this . We refer the interested reader to , e.g. , ( Bengio , 2020 ) . Notations . For f : Rp ⇥ Rq ! Rd , we denote @ xf ( x , y ) = @ f j ( x , y ) / @ xi i , j 2 Rd⇥p . 2 TARGET PROPAGATION WITH LINEARIZED REGULARIZED INVERSES . While target propagation was initially developed for multi-layer neural networks , we focus on its implementation for recurrent neural networks , as we shall follow the benchmark of Manchev & Spratling ( 2020 ) in the experiments . Recurrent Neural Networks ( RNNs ) are also a canonical family of neural networks in which interesting phenomena arise in back-propagation algorithms . Problem setting . A simple RNN parameterized by ✓ = ( Whh , Wxh , bh , Why , by ) maps a sequence of inputs x1 : ⌧ = ( x1 , . . . , x⌧ ) to an output ŷ = g✓ ( x1 : ⌧ ) by computing hidden states ht 2 Rp corresponding to the inputs xt . Formally , the output ŷ and the hidden states ht are computed as an output operation following transition operations defined as ŷ = c✓ ( h⌧ ) : = s ( Whyh⌧ + by ) , ht = f✓ , t ( ht 1 ) : = a ( Wxhxt +Whhht 1 + bh ) for t 2 { 1 , . . . , ⌧ } , where s is , e.g. , the soft-max function for classification tasks , a is a non-linear operation such as the hyperbolic tangent function , and the initial hidden state is generally fixed as h0 = 0 . Given samples of sequence-output pairs ( x1 : ⌧ , y ) , the RNN is trained to minimize the error ` ( y , g✓ ( x1 : ⌧ ) ) of predicting ŷ = g✓ ( x1 : ⌧ ) instead of y . As one considers longer sequences , RNNs face the challenge of exploding/vanishing gradients @ g✓ ( x1 : ⌧ ) / @ ht ( Bengio & Frasconi , 1995 ) ; see Appendix A for more discussion . We acknowledge that specific parameterization-based strategies have been proposed to address this issue of exploding/vanishing gradients , such as orthonormal parameterizations of the weights ( Arjovsky et al. , 2016 ; Helfrich et al. , 2018 ; Lezcano-Casado & Martınez-Rubio , 2019 ) . The focus here is to simplify and understand target propagation as a backpropagation-type algorithm using RNNs as a workbench . Indeed , training RNNs is an optimization problem involving multiple compositions for which approximate inverses can easily be available . The framework could also be potentially applied to , e.g. , time-series or control models ( Roulet et al. , 2019 ) . Given the parameters Whh , Wxh , bh of the transition operations , we can get approximate inverses of f✓ , t ( ht 1 ) for all t 2 { 1 , . . . , ⌧ } , that yield optimization surrogates that can be better performing than the ones corresponding to regular gradients . We present below a simple version of target propagation based on regularized inverses and inverse linearizations . Back-propagating targets . The idea of target propagation is to compute virtual targets vt for each layer t = ⌧ , . . . , 1 such that if the layers were able to match their corresponding target at time t , i.e. , f✓ , t ( ht 1 ) ⇡ vt , the objective would decrease . The final target v⌧ is computed as a gradient step on the loss w.r.t . h⌧ . The targets are then back-propagated using an approximate inverse1 f 1✓ , t of f✓ , t at each time step . Formally , consider an RNN that computed ⌧ states h1 , . . . , h⌧ from a sequence x1 , . . . , x⌧ with associated output y . For a given stepsize h > 0 , we propose to back-propagate targets by computing v⌧ = h⌧ h @ h ` ( y , c✓ ( h⌧ ) ) , ( 1 ) vt 1 = ht 1 + @ hf 1 ✓ , t ( ht ) > ( vt ht ) , for t 2 { ⌧ , . . . , 1 } . ( 2 ) The update rule ( 2 ) blends two ideas : i ) regularized inversion ; ii ) linear approximation . We shall describe below that our update ( 2 ) allows us to interpret the “ magic formula ” of difference target propagation in Eq . 15 of Lee et al . ( 2015 ) as 0th-order finite difference approximation , while ours is a 1st-order linear approximation . We shall also show that ( 2 ) puts in practice an insight from Bengio ( 2020 ) suggesting to use the inverse of the gradients in the spirit of a Gauss-Newton method . Once all targets are computed , the parameters of the transition operations are updated such that the outputs of f✓ , t at each time step move closer to the given target . Formally , the update consists of a gradient step with stepsize ✓ on the squared error between the targets and the current outputs , i.e. , for ✓h 2 { Whh , Wxh , bh } , ✓ next h = ✓h ✓ ⌧X t=1 @ ✓hkf✓ , t ( ht 1 ) vtk22/2 . ( 3 ) As for the parameters ✓y = ( Why , by ) of the output operation , they are updated by a simple gradient step on the loss with a stepsize ✓ . 2.1 REGULARIZED INVERSION . To explore further the original idea of Le Cun et al . ( 1989 ) , we consider using the variational definition of the inverse , f 1 ✓ , t ( vt ) = argmin vt 12Rp kf✓ , t ( vt 1 ) vtk22 = argmin vt 12Rp ka ( Wxhxt +Whhvt 1 + bh ) vtk22 . ( 4 ) As long as vt belongs to the image f✓ , t ( Rp ) of f✓ , t , this definition recovers exactly the inverse of vt by f✓ , t . More generally , if vt 62 f✓ , t ( Rp ) , Eq . ( 4 ) computes the best approximation of the 1In the following , to ease the presentation , we abuse notations and denote approximate inverses by f 1✓ , t . inverse in the sense of the Euclidean projection . When one considers an activation function a and ✓h = ( Whh , Wxh , bh ) , the solution of ( 4 ) can easily be computed . Formally , for the sigmoid , the hyperbolic tangent or the ReLU , their inverse can be obtained analytically for any vt 2 a ( Rp ) . So for vt 2 a ( Rp ) and Whh full rank , we get f 1 ✓ , t ( vt ) = ( W > hhWhh ) 1 W > hh ( a 1 ( vt ) Wxhxt bh ) . If vt 62 a ( Rp ) , the minimizer of ( 4 ) is obtained by first projecting vt onto a ( Rp ) , before inverting the linear operation . To account for non-invertible matrices Whh , we also add a regularization in the computation of the inverse . Overall we consider approximating the inverse of the layer by a regularized inverse of the form f 1 ✓ , t ( vt ) = ( W > hhWhh + r I ) 1 W > hh ( a 1 ( ⇡ ( vt ) ) Wxhxt bh ) , with r > 0 and ⇡ a projection onto a ( Rp ) . Regularized inversion vs. parameterized inversion . Bengio ( 2014 ) ; Manchev & Spratling ( 2020 ) parameterize the inverse as a reverse layer such that f 1 ✓ , t ( vt ) = ✓0 , t ( vt ) : = a ( Wxhxt + V vt + c ) , and learn the parameters ✓0 = ( V , c ) for this reverse layer to approximate the inverse of the forward computations . The parameterized layer needs to be learned to get a good approximation which involves numerically solving an optimization problem for each layer . These optimization problems come with a computational cost that can be better controlled by using regularized inversions presented earlier . However , the approach based on parameterized inverses may lack theoretical grounding , as pointed out by Bengio ( 2020 ) , as we do not know how close the learned inverse is to the actual inverse throughout the training process . In contrast , the regularized inversion ( 4 ) is less ad hoc and clearly defined and , as we shall show in the experiments , leads to competitive performance on real datasets . In any case , the analytic formulation of the inverse gives simple insights on an approach with parameterized inverses . Namely , the analytical formula suggests parameterizing the reverse layer s.t . ( i ) the reverse activation is defined as the inverse of the activation and not any activation , ( ii ) the layer uses a non-linear operation followed by a linear one instead of the usual scheme , i.e. , a linear operation followed by a non-linear one .

Summary: The authors study a variant of target propagation in which targets are computed by solving a sequence of minimization problems. Instead of resorting to iterative methods the authors propose to use an analytical solution. The algorithm is investigated as a recurrent neural network learning algorithm in a number of experiments. A relationship between the proposed descent direction and the loss function gradient is developed.