Target Propagation via Regularized Inversion for Recurrent Neural Networks
Authors: Vincent Roulet
Zaid Harchaoui
Venue: Accepted by TMLR ()

Abstract: Target Propagation (TP) algorithms compute targets instead of gradients along neural networks and propagate them backward in a way that is similar to yet different than gradient back-propagation (BP). The idea initially appeared as a perturbative alternative to BP that may improve gradient evaluation accuracy when training multi-layer neural networks (LeCun, 1985) and has gained popularity as a biologically plausible counterpart of BP. However, there have been many variations of TP, and a simple version of TP still remains worthwhile. Revisiting the insights of LeCun (1985) and Lee et al (2015), we present a simple version of TP based on regularized inversions of layers of recurrent neural networks. The proposed TP algorithm is easily implementable in a differentiable programming framework. We illustrate the algorithm with recurrent neural networks on long sequences in various sequence modeling problems and delineate the regimes in which the computational complexity of TP can be attractive compared to BP.

Published in Transactions on Machine Learning Research (01/2023)
Target Propagation via Regularized Inversion for Recurrent Neural Networks
Vincent Roulet*
vroulet edu
Deportment of Statistics University of Washington
Zaid Harchaoui
waid@ow.. edu
Department of Statistics University of Washington
Reviewed on OpenReview: https: // openreview. net/ formm? id= Q5vdEJyh8
Abstract
Target Propagation (TP) algorithms compute targets instead of gradients along neural net- works and propagate them backward in a way that similar to yel different than gradient back-propagation (BP). The idea initially appeared as perturbative alternative to BP that may improve gradient evaluation accuracy when training multi-layer neural networks (Lo- Cun, 1985) and has gained popularity there have been many variations of TP, and a simple version of TP still remains worthwhile. Revisiting the insights of LeCun (1985) and Lee et al. (2015), we present a simple version of TP based on regularized inversions of layers of recurrent noural networks. The proposed TP algorithm easily implementable in a differentiable programming framework. We illus- trate the algorithm with recurrent neural networks on long sequences in various sequence modeling problems and delineate the regimes in which the computational complexity of TP can be attractive compared to BP.
biologically plausible counterpart of BP. However,
1 Introduction
Target Propagation (TP) algorithms can be seen as perturbative learning alternatives to the gradient back- propagation algorithm, where virtual targets are propagated backward instead of gradients (LeCun, 1985; Rohwer, 1989; Mirowski & LeCun, 2009; Bengio, 2014; Lee et al., 2015; Meulemans et al., 2020; Manchev & Spratling, 2020). A high-level summary is presented in Fig. 1: while gradient back-propagation con- siders storing intermediate gradients in forward pass, TP algorithms proceed by computing and storing approximate inverses. The approximate inverses are then used to pass targets backward along the graph of computations to finally yield a weight update for stochastic learning.
TP aims to take advantage of the availability of approximate inverses compute better descent dircctions for the objective at hand. Bengio et al. (2013) and Bengio (2020) argued that the approach could be relevant for problems involving multiple compositions such the training of Recurrent Neural Networks (RNNs), which generally suffer from the phenomenon of exploding vanishing gradients (Hochreiter, 1998; Bengio et al., 1994; Schmidhuber, 1992). This perspective was studied theoretically for Difference Target Propagation (DTP), modern variant of TP, that was related to an approximate Gauss-Newion method, suggesting interesting venues to explain the benefits of TP (Bengio, 2020; Meulemans et al., 2020; 2021; Ernoult et al., 2022; Fairbank et al., 2022).
Another motivation for TP comes from neuroscience to propose biologically plausible learning processes in the brain and overcome the biological implausibility of gradient Back- Paok-Propngttion (BP) (Ernoult et al.,
* Now Google.

---

Published in Transactions on Machine Learning Research (01/2023)
Gradient Back-Propagaan Propagatipa
Target Propagation
Forward
Ø
f
f
Pass
0f())
0
Backward Pass
@f(x)
0 (y) T
Fig. 1: Our implementation of target propagation uses the linearization of gradient inverses instead of gradients in a backward pass akin to gradient back-propagation.
2022). The biological plausibility of TP required its actual implementation to use several approximations such as learning inverses with reverse layers (Lee et al., 2015; Manchev & Spratling, 2020). However, is unclear whether such reverse layers learn layer inverses during the training process, which hinders the study of the original idea of using approximate inverses.
To promote the original idea of TP and guide its implementation from both a biological and an optimization viewpoint, we develop here a simplified implementation of TP using analytical formulations. The proposed approach formalizes several heuristics used in previous implementations to highlight the key components that can make TP successful to, c.g., learning from long sequenees. In our implementation of TP for re- current neural networks, we consider approximating inverses via their variational formulation with analytic expressions. The analytic expression of our approximate inverse sheds light on the relevance of a regular- ization term which can be interpreted as inverting perturbed layers, an insight studied earlier by Lee et al. (2015) and Manchev & Spratling (2020) when learning reversed layers. In addition, by using analytical ex- pressions, we can directly lest. recent. interpretations of TP as an approximate Gause-Newion method whose approximation comes from the learned reverse layers (Bengio, 2020; Meulemans et al., 2020; Ernoult et al., 2022).
In this spirit, we also use the interpretation of the DTP formula (Lee et al., 2015) finite difference approx- imation of a linearized regularized inverse to propose a smoother formula that can directly be integrated into differentiable programming framework. We detail the computational complexity of the proposed implemen- tation of TP and compare it to the one of BP, showing that the additional cost of computing inverses can be effectively amortized for very long sequences. Following the benchmark of Manchev & Spratling (2020), we observe that the proposed implementation of TP can perform better than classical gradient- based methods on several tasks involving RNNs. The code is made publicly available at https: / /gi titt . con/vroulet/tpri.
Related work. The use of the gradient Back- BackkPProaggatinn (BP) algorithm to train deep networks was popu- larized by Rumelhart et al. (1986), though the algorithm had been used in several other fields before (Werbos, 1974; 1994; Griewank & Walther, 2008; Goodfellow et al., 2016). At the same time, alternatives to BP were proposed by, e.g., (LeCun, 1985, Section 2.2), by using "ideal states" propagated through the transpose of the weights and initialized at the desired outputs to define local learning objectives at each layer of the network. This idea was further explored in two different ways: either by redefining the training problem through the optimization of ideal states called "targets" as initiated by Rohwer (1989) or by redefining the backward dynamics of BP back pack-propagate the desired outputs through approximate inverses as developed by, e.g., Bengio (2014) and Lee et al. (2015).
Optimization in target space. The training of deep networks can be seen an optimization problem over both the weights of the layers and the intermediate states computed by the network, the two different sets of variables being constrained by the dynamies of the network. For a given set of weights, the intermediate states are entirely determined by the input of the network; that is the viewpoint adopted by BP by using forward-backward passes on the network (Rumelhart et al., 1986). Rohwer (1989) proposed to consider the intermediate states as the main variables of the network, which he called "targets", by computing for fixed set of targets the optimal weights such that the dynamics are approximately satisfied while the input and
2

---

Published in Transactions on Machine Learning Research (01/2023)
output of the network are fixed at the values given by the training set. The overall training objective depends only on targets that can be optimized by gradient descent on the error of reconstruction of the dynamics.
Such an approach has been re-proposed under different names by Atiya & Parlos (2000) and Castillo et al. (2006) and later cast as penalized formulation of the dynamical constraints by Carreira-Perpinan & Wang (2014) who exploited the resulting unconstrained formulation to decouple the training of each layer by alternate optimization. Taylor et al. (2016) and Gotmare et al. (2018) considered tackling the dynamical constraints by using Lagrangian formulation of the problem and using an Alternative Direction Method of Multipliers (ADMM). In this spirit, Wiseman et al. (2017) used an ADMM-like algorithm for language modeling and reported disappointing experimental results.
On the other hand, Fairbank et al. (2022) revisited the original approach of Rohwer (1989) by considering a reconstruction error of the dynamics only based on its linear part and reported successful experimental results on several architectures. The main difficulty of the optimization in target space is that the total nunber of variables is a priori equal to the number of samples times the width and depth of the network. This difficulty was overcome by Fairbank et al. (2022) by considering optimizing targets defined on subset of the training set, then propagating the information given by new samples by the chain rule. Taylor et al. (2016) overcame this issue by simplifying the optimization of the targets as one coordinate descent pass on the Lagrangian formulation which then resembles a back-propagation algorithm. In our work, we consider back poopaaaopggg targets to update weights rather than considering directly the targets as variables.
Back- propaganpggg targets or gradient surrogates. The original idea of LeCun (1985) to use back- propagaton mechanisms that could back-propagate ideal states through the network was revisited few decades later by Bengio (2014) and Lee ct al. (2015). Although directly using approximate inverses of the layers to back-propagate targets appeared unsuccessful, slight variant called Difference Target Propagation (DTP), which stabilizes the inverses by means of a finite difference scheme explained in Sec. 2.2, matched roughly the performance of BP. We formalize this variant by using directly the Jacobian of the approximate inverse to test the relevance of the approach from an optimization viewpoint. DTP has been recently interpreted theoretically as an approximate Gauss-Newton method (Bengio, 2020; Meulemans et. al., 2020; 2021), a viewpoint that we discuss in detail in Appendix C. On a practical side, DTP was successfully implemented for recurrent neural networks by Manchev & Spratling (2020). We follow the experimental benchmark done by Manchev & Spratling (2020) while formalizing some of the mechanisms of DTP in our work. Recently, Ahmad et al. (2020) and Dalm et al. (2021) considered using analytical inverses to implement TP and blend it with what they called a gradient-adjusted incremental formula. Yet, an additional orthogonality penalty is critical for their approach to work, while our approach can dispense of such additional penalty.
The back-propagattin of gradients has also been bypassed by approximating the sensitivity of the output of the network to small perturbations by Le Cun al. (1988) in order to update weights according to these sensitivities. Alternatively, gradient surrogates were approximated with "synthetic gradients" by Jaderberg et al. (2017) and Czarnecki et al. (2017) to decouple the backward pass of feed-forward deep networks and speed up the training process. Roulet & Harchaoui (2022) considered computing gradients of the Morean envelope through the structure of the network and cast several previous works as instances of approximate computations of the gradients of the Moreau envelope. Rather than modifying the backward operations in the layers, one can also modify the weight updates for deep forward networks by using a regularized inverse as successfully implemented by Frerix et al. (2018). Recently, Amid et al. (2022) generalized the approach of Frerix et al. (2018) by considering alternative reconstruction losses, that is, alternative objectives for the weights to minimize for given targets by using matching losses on pre-activations or post-activations targets defined by a gradient step on these quantities.
Biological plausibility. A recurrent motivation for the development of TP algorithms has been the biological implausibility of BP (Crick, 1989). We summarize here some of the arguments against the biological plau- sibility of BP and refer the interested reader to, e.g., Bengio (2020) and Manchev & Spratling (2020) for detailed discussion. First BP requires each neuron to emit two distinct signals (forward and backward) which has not been observed neurophysiologically (Roclfsema & Ooyen, 2005). Another issuc is that BP solves the credit assignment problem (Minsky, 1961; Hinton et al., 1984) by using weight transport, imply- ing a symmetric backward connectivity pattern, which is thought impossible in the brain (Lillicrap et al.,
3

---

Published in Transactions on Machine Learning Research (01/2023)
2016). On the other hand, DTP may circumvent some of these issues such weight transport by using different dynamics forward and backward. The credit assignment problem and other biological constraints to model learning schemes motivated several variations of TP or other learning schemes such as direct feedback alignment (Nokland, 2016; Bartunov et al., 2018; Crafton et al., 2019; Lansdell et al., 2019; Akrout et al., 2019; Kunin et al., 2020; Ernoult. et al., 2022). As mentioned earlier, our objective here is to complement the biological viewpoint on TP by taking an optimization viewpoint to understand relevance training performance and use these findings to serve as guidelines for biologically plausible implementations of TP.
Notation. For f R x RR-R the partial derivative of f w.r.t. x on (a, E RP x R denotod 0zf(z,v) (0 (I, (0)/) 1,3 E Bpxd
2 Target Propagation with Linearized Regularized Inverses
While TP was initially developed for multi-layer neural networks, we focus on its implementation for Re- current Neural Networks (RNNs), as we shall follow the benchmark of Manchev & Spratling (2020) in the experiments. RNNs are also a canonical family of neural networks in which interesting phenomena arise in back-propagation algorithms.
Problem setting. A simple RNN parameterized by 0 = (Wir, W Wh bp. Why by) maps a sequence of inputs Fir = (#1...., F.y) to an output & Fs (= by computing hidden states he E Rdd corresponding to the inputs x, R Formally, the output á and the hidden states hy are computed as an output operation following transition operations
ÿ = go (hr) = s(Wnglt + by);
h = fo, = a(Wah + Whahe-1 + by) for E (1,...,77)
where is, e.g,, the soft-max function for classification lasks, a is a non-linear operation such as the hyperbolic tangent function, and the initial hidden state is generally fixed as ho = O. Given samples of sequence-outpuu pairs y), the RNN trained to minimize the error E(y, Fa of predicting (=Fo( instcad of y.
As one considers longer sequences, RNNs face the challenge of saaloding/vanishiinn gradients (F) (Bengio & Frasconi, 1995); see Appendix A for more discussion. We acknowledge that specific parameterization-beeeee strategies have been proposed to address this issue of exploding/vaniie
gradients,
such as orthonormal parameterizations of the weights (Arjovsky et al., 2016; Helfrich et al., 2018; Lezcano- Casado & Martmez-Rubio, 2019). The focus here is to simplify and understand TP as a back-propagation- type algorithm using R.NNs as a workbench. Indeed, training RNNs is an optimization problem involving multiple compositions for which approximate inverses can casily be available. The framework could also be applied to, e.g., time-series or control models (Roulet et al., 2019).
Given the parameters WO., Wahs by of the transition operations, we can get approximate inverses of (he-()) for all L E (1, that yield optimization surrogates that can be better performing than the ones cor- responding to regular gradients. We present below a simple version of TP based on regularized analytic inverses and inverse linearizations.
Baok-ppopaggting targets. The idea of TP is to compute virtual targets is for each layer t = ....., such that if the layers were able to match their corresponding target at time i, i.e. fe.(hhr-)) RS th the objective would decrease. The final target e, is computed as a gradient step on the loss w.it. h,, The targets are back-propagated using an approximate inverse fó, of fo,t at each time step. detailed in Sec. 2.1.
Formally, consider an RNN that computed t states hi,..., hr from sequence oI, with associated output y. For a given stepsize nh O, we propose to back propaaro targets by computing
Ur = h 7, (,.
(1)
0- = h + anfd) (vr - -h),, for t E {o....,
(2)
4

---

Published in Transactions on Machine Learning Research (01/2023)
The update rule (2) blends two ideas: i) regularized analytic inversions to approximate the inverse fø, by fole: ii) linear approximation, i.c., using the Jacobian of the approximate inverse instead of the difference larget propagation formula proposed by Lee et al. (2015, Eq. 15). We shall also show that the update (2) puts in practice an insight from Bengio (2020) suggesting to use the inverse of the gradients in the spirit of a Gauss-Newton method. Once all targets are computed, the parameters of the transition operations are updated such that the outputs of fa,: at each time step move closer the given target.
Formally, the update consists of gradient step with stepsize to the squared error between the targets and the current outputs, L.e, for On E {Whn, Wah, bn):
0gext = 0g - 70 Σ Sen lff.,((hs-)) - - 1212/2.
(3)
As for the parameters 0y = (Why, by), they are updated by a simple gradient step on the loss with the stepsize "/0.
2.1 Regularized Analytic Inversion
To explore further the ideas of LeCun (1985) and Lee al. (2015), we consider the variational definition of the inverse,
(w)) = argmin (v)_) - 0e vel12 = argmin ||a(Wat Wah Wha + bb) - 04 112- o IER4A
(4)
ot IRR4
As long as vt belongs to the image (I2d)) fe.s, this definition recovers exactly the inverse of or by fa,t- More generally, U & fo,((R),, Eq. (4) computes the best approzimation of the inverse the sense of the Euclidean projection. For an injective activation function a and 0, = (War, Wah, by), the solution of (4) can easily be computed. Formally, for the sigmoid, the hyperbolic tangent, or the ReLU, their inverse can be obtained analytically for any E a(Rd)). So for E a(Rd)) and WAA full rank, get
(w)) (W ha WAn) -n (T ((<))) W.n bb..
If oe & a(Rd),, the minimizer of (4) is obtained by first projecting vt onto a(Rd),, before inverting the linear operation. To account for non-invertible matrices WaA, we also add a regularization in the computation of the inverse. Overall consider approximating the inverse of the layer by regularized inverse of the form
(vx) (W,, Wh. + +r WW (a- (T(x)) WIa bb);
with a regularization parameter > 0 and projection π onto (IRd) detailed in Appendix B.
Analytic inversion vs. parameterized inversion. Bengio (2014) and Manchev & Spratling (2020) parameterize the inverse as a reverse layer such that
(v.) (v.) = a(Wan Vv, + cc,
and learn the parameters 0' = (V.c) for this reverse layer to approximate the inverse of the forward com- putations. Learning good approximations comes with a computational cost that can be better controlled by using analytic inversions presented above. The approach based on parameterized inverses may lack the- oretical grounding, as pointed out by Bengio (2020), we do not know how close the learned inverse is to the actual inverse throughout the training process. In contrast, the analytic inversion (4) is less ad hoc, it enables us to test directly the main idea of TP and it leads to competitive performance on real datasets.
The analytic formulation the inverse gives simple insights into an approach with parameterized inverses. Namely, the analytic formula suggests parameterizing the reverse layer such that (i) the reverse activation is defined as the inverse of the activation, (ii) the layer uses a non-linear operation followed by linear one instead of linear operation followed by non-linear one used by, e.g., Manchey & Spratling (2020).
5

---

Published in Transactions on Machine Learning Research (01/2023)
Regularized inversion noise injection. A potential issue with learning parameterized inverses done by Manchev & Spratling (2020) and Lee et al. (2015) is that the inverse depends a priori on the current set of parameters of the transition function, which, in turn, changes along the training process. Manchev & Spratling (2020) and Loe et al. (2015) considered then stabilizing the learning process of the parameterized inverse by letting minimize perturbed versions of the layers, an idea reminiscent of the early work of Cun et al. (1988).
Formally, the parameterized inverses are learned by performing gradient steps over 8' Ei 1e (ff..(/ff. Et-1)) - (h-- + E1-1)ll2 for ét-1 (O, o² 1) and ho...., given by forward pass on the RNN for given sample and fixed parameters 0. Such strategy led to performance gains in the overall training (Manchev & Spratling, 2020, Section 3.1.2, Figure! 7).
In our formulation the regularization can seen as a counterpart of the noise injection. Namely, consider the inverse of averaged perturbed layers, i., the minimizer Ut-1 of Ez~V(0, n)lff+=..((<-11>- - vell2- - As shown in Appendix C, the regularized inverse fe. minimizes upper-bound of this objective, namely, for o² r/dn and U2 E a(Rd)),
(v)) argmin E2 nIl ER-
a (vx)
where (0 1) ((ex==(((_=20011)))))))))) -a continuous activation and the approximation error of using the upper-bound is quantified in Appendix C.
L.Eg~~((000 1) (00=,, vell for a L.-Lipschitz-
The regularization can then be interpreted as a hyper-parameter that ensures that the inverses computed for the current paramelers are also approximalely valid for the updated parameters as long as |0 - -0, l1 < o² = r/dn, which hints that the regularization r may be chosen such that r is larger than constant times 79.
2.2 Linearized Inversion
Earlier instances of TP used directly approximate inverses of the network layers such that the TP update formula would read 1 (v.) in (2). Yet, we unaware of successful implementation of TP using directly the inverses. To circumvent this issue, Lee et al. (2015) proposed the Difference Target Propogation (DTP) formula
Ut P--1-1-11
- - (1x)
If the inverses were exact, the DTP formula would reduce to 2t-1 = foyl (v)). Lee et al. (2015) introduced the DTP formula to mitigate the approximation error of the inverses by parameterized layers. The DTP formula was recently interpreted as an approximate Gauss-Newtrr method (Meulemans et al., 2020; Bengio, 2020) by observing that can be approximated by the linearization of the inverse as
(v)( - (he) = @nfba(be) (v. - h1) + O(Iee -hel2).
Our implementation puts in practice this insight. We show in Appendix D that the linearization we pro- pose (2) leads to slightly better training curves than the finite-difference approximation. Moreover, our linearized formula enables a simple implementation TP in a differentiable programming framework and simplifies its comparison with gradient back propppropagttion from a computational viewpoint.
3 Gradient
versus Target Propagation
Graph of computations. Gradient prop-pronnnntion and target propagation both compute an update direction for the objective hand. The difference lies in the oracles computed and stored in the forward pass, while the graph of computations remains the same. To clarify this view, reformulate TP in terms

---

Published in Transactions on Machine Learning Research (01/2023)
Zt
y
hg
f
g
E y)
h1
h
h
hr
y
0rft
0,
Gr
8g
0t/09
A1
X
A
8
8,
8
0.
Ugg
00
Fig. 2: The graph of computations of target propagation is the same the of gradient back- pack-propagation except that f needs to be computed and Jacobian of the inverses, fr, T are used instead of gradients 0 in the transition operations.
of displacements A Ut h4 such that Eq. (1), (2) and (3) read
x = Elv.
d = -inSb,e de. for E fr,...,11 1},
So,, Jo,
Ohext On + 71110n.
TP amounts then computing an update direction uo, for the parameters 0 with a graph of computations, illustrated in Fig. 2, analogous to that of BP illustrated in Appendix A. The difference lies in the use of the Jacobian of the inverse
So (he) T instead of (o
The implementation of TP with the formula (2) can be done in differentiable programming framework, where, rather than computing the gradient of the layer, one evaluates the inverse and keep the Jacobian of the inverse. With the precise graph of computation of TP and BP, we can compare their computational complexity explicitly and bound the difference the directions they output.
Arithmetic complexity. Clearly, the space complexities of BP and our implementation of TP are the same since the Jacobians of the inverse and the original gradients have the same size. In terms of time complexity, TP appears at first glance to introduce an important overhead since it requires the computation of some inverses. However, a close inspection of the formula of the regularized inverse reveals that a matrix inversion needs to be computed only once for all time steps. Therefore the cost of the inversion may be amortized if the length of the sequence is particularly long.
Formally, the time complexity of the forward- backward pass of gradient back- propagation is essentially driven by matrix-vector products, i.c.,
TBP= Σ
T(Jo.) +T (0n.o,.) + T Se.e) T(ohSee(hb-i))) T(Og, (ho
Forward
Backward
2 Tddddh + +
where d, the dimension of the inputs ze. d, is the dimension of the hidden states h, for a function f we denote by T(f) the time complexity evaluate of and we consider, e.g., (h the linear function x Do (h

---

Published in Transactions on Machine Learning Research (01/2023)
On the other hand, the time complexity of target propagation is
TIP Σ T(feN++ (T fo.e) +T(Onfoe)
(t) (h.(h
+ P(ffee)
Forward
Backward
where P(fó.r) is the cost of encoding the inverse, which, in our case, amounts the cost of encoding Co : - (W - 2 such that our regularized inverse can be computed as (v,) = ca(a (v:)- Wan Encoding Co comes at the cost of inverting matrix of size d,. Therefore, the time- complexxyy of TP can be estimated as
TTP dp + (da,d, + d2,2) rdh² ~ TBP if da.
So for long sequences whose length is larger than the dimension of the hidden states, the cost of TP with regularized inverses is approximately the same the cost of BP. If a parameterized inverse was used rather than a regularized inverse, the cost of encoding the inverse would correspond to the cost of updating the reverse layers by, e.g., a stochastic gradient descent. This update has a cost similar to BP. However, it is unclear whether these updates get us close to the actual inverses.
Bounding the difference between target propagation and gradient back-propagation. As the computational graphs of BP and TP are the same, we can bound the difference between the oracles returned by both methods. First, note that the updates of the parameters of the output functions are the same since, in TP, gradient steps of the loss are used to update these parameters. The difference between TP and BP lies in the updates with respect to the parameters of the transition operations.
For BP, the update direction computed by chain rule as
Ohr On. Ely, (h()))) Th,
oo, l (y, (o( = -o.,, So.
where the term 0h,/Oh, decomposes along the time steps ah, 0h,, IT==44a - fo, The direction computed by TP has the same structure, namely it can be decomposed for V 1
Ohr on, C(v. go (hz)),
u0 Σ So So.
ah,
where Ôh-/ôh. IT So. (ha) We can then bound the difference between the directions given by BP or TP as formally stated in the following lemma.
Lemma 3.1. The difference between the oracle returned by gradient back- touk--roppgation oo E (y. Fo (21:)) and the oracle returned by larget propagation uoa can be bounded
|10b, e (y. Fo( uo. S Σ ca callob.
((y, (o sup 110h, fo, Ohe.b..(he) I1.
where at -E-baal 1-- with supt I.... 11,
sup. (1ene "1l
For regularized inverses, we have, denoling 2t Wankr While + bb,
(0....fe.(hb-1) Sn.f T [W.JT (IVa(z)) ( (VVa(2))VV((z)--1 +|++||I-- I|I-IIIIWTTTT WhA + +11)=
For the two oracles to be close, need the preactivation 2, = W 25, Winh,-1 + by to lie in the region of the activation function that is close to being linear s.t. Va(z;) R 1. We also need (W hh Whr + +r1) to be close to the identity which can be the case if, c.g., r = 0 and the weight matrices W were orthonormal. By initializing the weight matrices as orthonormal matrices, the differences between the two oracles can be close in the first iterations. However, in the long term, target propagation appears to give potentially better oracles, as shown in the experiments below. From thcoretical standpoint, the above lemma can be usod obtain convergence guarantees to stationary points of TP up to the approximation error of BP by TP shown in Appendix C.
8

---

Published in Transactions on Machine Learning Research (01/2023)
TP
BP
Train Loss
1000
4000
100
2000
50
0
2
2
Iterations
10
Iterations
x10'
Iterations
x 10
100
100
100-
75
75
75
Accuracy
50
50
50
23
25
25
0-
2
0
Iterations
x10
Iterations
x10
Iterations
x x10*
Fig. 3:
Temporal order problem T 60, Temporal order problem T 120, Adding problem T == 30.
Testing the interpretation of TP as a Gauss-Newton method. Recently target propagation has been interpreted as an approximate Gauss-Newton (GN) method, by considering that the DTP formula approximates the lincarization the inverse, which itself is a priori equal to the inverse of the gradi- ents (Bengio, 2020; Meulemans et al., 2020; 2021). Namely, provided that f (()())))()))))) h4-1 such that (hh
we have
On, So. (Om- (oxhhh))))
By composing the inverses of the gradients, we get an update similar to the one of GN. Namely, recall that if invertible functions fi...., .....,., fu were composed to solve a least square problem of the form ol GN update would take the form (1++1) iteratively as 20 = (8),, Ft+1 = (x,). In other words, GN and TP share the idea of composing the inverse of gradients. However, numerous differences remain as detailed in Appendix gradient of the layer with respect to its parameters is not inverted in the usual implementation of TP and that the approximation error of GN by TP may grow exponentially with the length of the network. Note that even if TP was approximating GN, it is unclear whether GN updates adapted to stochastic problems.
fi (xx)- ... -2.-1 (xn- (a,, (xx, where 2, is defined
such as the fact that the
In any case, by using analytical formula for the inverse, can test this interprelation by using non- regularized inverses, which would amount to directly using the inverses of the gradients as in GN method. If the success of TP could be explained by its interpretation as a GN method, we should observe efficient training curves when no regularization added.
4 Experiments
In the following, we compare our simple TP approach, which we shall refer to as TP, to gradient Back- Propagation referred to as BP. We follow the experimental benchmark of Manchev Spratling (2020) which we add results for RNNs CIFAR and for GRU networks FashionMNIST. Additional experimental, details the initialization and the parem-p selection be found in Appendix D.
Data. We consider two synthetic datascts generated to present training difficulties for RNNs and several real datasets consisting of scanning images pixel by pixel classify them (Hochreiter & Schmidhuber, 1997; Le et al., 2015; Manchev & Spratling, 2020).
9

---

Published in Transactions on Machine Learning Research (01/2023)
BP
TP
b000
8000
Train Lees
7200
8000
600
6000
6000
4000
1000
7100
7000
4000
2000
2000
7000
0000
0
Iterations
10
Iterations
x 10
Iterations
x1
Iterations
10P
Iterations
x 10
1000
100
40
75
75
20
30
20
Accuracy
20
10
10
25
25
10
2
Iterations
x X10
Iterations
x 10
Iterations
x100
Iterations
x10*
Iterations
x10
Fig. 4: Image classification pixel by pixel and word prediction. From left to right: MNIST, MNIST with permuted images, CIFAR10, FashionMNIST with GRU, Penn Treebank with RNN.
Temporal order problem. A sequence of length T is generated using set of randomly chosen sym- bols {a, b, c, d). Two additional symbols X and Y are added at positions ti [T/10, 2T/10] and la E [4T/10, 57/10). The network must predict the correct order of appearance of X and Y out of four possible choices [XX,XX, XY,XX,YXXXXX }Y).
Adding problem. The input consists of two sequences: one is made of randomly chosen numbers from [o, 1]. and the other one binary sequence full of zeros except at positions 4 E [1, T/10) and l2 E [T/10, T/2]. The sccond position acts as marker for the time steps 1 and f2. The goal of the network is output the mean of the two random numbers of the first sequence (X,, + Xx2)/2.
Image classification pizel by pizel. The inputs are images of (i) grayscale handwritten digits given in the database MNIST (LeCun & Cortes, 1998), (ii) colored objects from the database CIFAR10 (Krizhevsky, 2009) or (iii) grayscale images of clothes from the database FashionMNIST (Xiao et al., 2017). The images are scanned pixel by pixel and channel by channel for CIFAR1O, and fed to a sequential network such as a simple RNN or a GRU network (Cho al., 2014). The inputs are then sequences of x 28 = 784 pixels for MNIST or FashionMNIST and 32 32 3 3072 pixcls for CIFAR with a very long-range dependency problem. We also consider permuting the images of MNIST by fixed permutation before feeding them into the network, which gives potentially longer dependencies in the data.
Word prediction. Given sequence of words from 10000 sentences of the Penn Treebank dataset (Marcinkiewicz, 1994), the task is to predict the t + 1th word among a vocabulary of 10 000 words. We consider sequences of length 64 and a fixed embedding of size 1024 of the words.
Model. In both synthetic settings, we consider randomly generated mini-batches of size 20, a simple RNN with hidden states of dimension 100, and hyperbolic tangent activation. For the temporal order problem, the output function uses a soft-max function on top of a lincar operation, and the loss is the cross-entropy. For the adding problem, the output function is linear, the loss is the mean-squared error, and a sample is considered to be accurately predicted if the mean squared error is less than 0.04 as done by (Manchev & Spratling, 2020).
For the classification of images with sequential networks, we consider mini-batches of size 16 and a cross- entropy loss. For MNIST and CIFAR, we consider a simple RNN with hidden states of dimension 100, hyperbolic tangent activation, and a softmax output. For FashionMNIST, we consider GRU network and adapted our implementation of target propagation to that case while using hidden states of dimension 100 and a softmax output, Appendix B for the detailed implementation of TP for GRU networks.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Region parameters with convergence
TP DTP-RI DTP-PI
100
10
512
75
TP
Stepsize "i
Accuracy
10-1
Width
128
50
64
25
10-3
32
BP
0
4
10
10'
392
784
Iterations
x10
Regularization
Length
Fig. 5: Comp. of TP implem.
Fig. 6: Conv. w.r.t. stepsize & reg.
Fig. 7: Perf. vs width & length.
For the prediction of words, we consider an RNN that oulputs a prediction al each time step. Our imple- mentation of TP, in this case, consists in replacing the usual gradient back-propagation rule used in the computational scheme of gradient back-propagation by the Jacobian of the regularized inverse as explained in Sec. 3 and detailed in Appendix B. We use a simple RNN architecture with hyperbolic langent activalions for the transitions hidden states of size 256, an output function consisting of a oof- layer on top of a linear layer, and entroyy loss.
Performance comparisons. In Fig. 3, we observe that TP performs better than BP on the temporal ordering problem: it is able to reach 100% accuracy in fewer iterations than BP for sequences of length 60 and, for sequences of length 120, it still able to reach 100% accuracy fewer than 40 000 iterations while BP is not. On the other hand, for the adding problem, TP performs less well than BP. The contrast in performance between the two synthetic tasks was also observed by (Manchev & Spratling, 2020) using difference larget propagation with parameterized inverses. The main difference between these tasks is the nature of the outputs, which are binary for the temporal problem and continuous for the adding problem.
In Fig. 4, we observe that TP generally performs better or as well as BP for image classification tasks. For the MNIST dataset, it reaches around 74% accuracy after 104 iterations. This phenomenon is also observed with permuted images, where the optimization appears smoother, and TP obtains around 86% accuracy after 4 10s iterations and is still slightly faster than BP. On the CIFAR dataset, no algorithms appear to reach significant accuracy, though TP is still faster. On the FashionMNIST dataset, where GRU network is used, our implementation of TP performs on par with BP, which shows that our approach can be generalized to more complex networks than a simple RNN. For the word prediction task, our implementation of TP is still able to optimize well the objective in terms of training loss, outperforming BP in that regard, and performing slightly worse in terms of test accuracy.
Comparison of different implementations of TP. We evaluate the impact of using regularized in- verses as opposed to parameterized inverses and linearized propagation as opposed to finite-difference based propagation. The variant of target propagation with parameterized inverse and finite-difference propaga- tion corresponds the approach of Lee al. (2015) recently implemented by Manchev & Spratling (2020) and referred to in Fig. 5 as DTP-PI. The variant of target propagation with regularized inverse and finite- difference propagation is referred to in Fig. 5 DTP-RL. Recall that approach involves regularized inverses and lincarized propagation, referred to as TP. In Fig. 5, we observe that both TP and DTP-RI outperform DTP-PI, demonstrating the benefits of using regularized inverses. On the other hand, both TP and DTP-RI perform on par overall, with the former being slightly better for the given parameters. The lincarized formula has the advantage to be casily adapted other architectures such as a GRU network or an RNN with intermediate outputs as explained in Appendix B. Additional comparisons of our implementation of TP as opposed to the implementation of TP by Manchey & Spratling (2020) are presented in Appendix D.
baseed
11

---

Published in Transactions on Machine Learning Research (01/2023)
40
Accuracy
16.06
Accuracy
16-04
Accuracy
20
1c.05
20
1e.03
20
1e0
16.04
1s.00
13.03
le 01
1e2
0
2 Iterations
1e.02
0
x10
Iterations
x10P
Iterations
x10
Fig. 8: Sensitivity analyses the hyper-parameeers of TP on MNIST pixel by pixel. From left to right: varying Th for 70=10-1 r=1, varying "i for 7n=10 r=1, varying for m=100 7==10-1
Impact of the regularization term. As mentioned in Sec. 3, by using an analytical formula to compute the inverse of the layers, we can question the interpretation of TPP Gauss-Newton method, which would amount to TP without regularization. To understand the effect of the regularization term, we computed the area under the training loss curve of TP for 400 iterations on a log10 grid of varying step-sizes no and regularizations r for a fixed stepsize ?/-- = 10-3, The results are presented in Fig. 6, where the smaller the area, the brighter the point and the absence of dots in the grid means that the algorithm diverged. Fig. 6 shows that without regularization we were not able to obtain convergence of the algorithm. Simply using the gradients of the inverse as in a Gauss-Newton method may not directly work for RNNs. Additional modifications of the method could be added to make larget propagation closer to Gauss-Newton, such as inverting the layers with respect to their parameters proposed by Bengio (2020). For now, the regularization appears to successfully handle the rationale of target propagation.
Performance of TP and BP in terms of the length of the sequence. In Fig. 7, we compare the performance of BP and TP in terms of accuracy after 400 iterations on the MNIST problem for various widths determined by the size of the hidden states and various lengths determined by the size of the inputs (i.e., we feed the RNN with pixels at a time, which gives a length 784/k). Fig. shows that. TP generally appropriate for long sequences, while BP remains more efficient for short sequences. TP may then be seen interesting alternative for dynamical problems which involve many discretization steps in RNNs and related architectures.
Sensitivity analyses. In Fig, 8, observe that the algorithm is mostly sensitive to the learning rate "o while both the stepsize Thh used for the first target and the regularization r admit larger range of values ensuring fact increase of the accuracy. Note that for the regularization, while most values above r = 1 ensure a similar increase of accuracy, smaller values than 10-1 in the MNIST pixel-by-pixt experiment would not lead to convergent algorithms as explained previously Fig. 6.
Conclusion
We proposed a simple target propagation approach grounded in two important computational components, regularized inversion, and linearized propagation. The proposed approach also sheds light on previous insights and successful rules for target propagation. The code is made publicly available at com/vroulet/tpri.. We have used target propagation within stochastic gradient outer loop to train neural networks for a fair comparison to stochastic gradient descent using gradient backpropagation. Developing adaptive stochastic gradient algorithms in the spirit of Adam that lead to boosts in performance when using target propagation instead of gradient backpropagation is an interesting avenue for future work. Continuous counterparts of target propagation in a neural ODE spirit is also an interesting avenue for future work.
This work was supported by NSF CCF-1740551, NSF 183-11893371, the CIFAR program "Learning in Machines and Brains", and faculty rescarch awards. We thank Nikolay Manchev for all the details he provided on his code. We also thank the reviewers and the action editor of TMLR for their insightful feedback that helped improve the content and the presentation of the paper.
Ackuowledgments..
12

---

Published in Transactions on Machine Learning Research (01/2023)
References
Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorfll org/.
Nasir Ahmad, Marcel A van Gerven, and Luca Ambrogioni. GAIT-prop: A biologically plausible learning rule derived from backpropagation of error. In Advances in Neural Information Processing Systems, volume 33, 2020.
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In Advances in neural information processing systems, volume 32, 2019.
Ehsan Amid, Rohan Anil, and Manfred Warmuth. Locoprop: Enhancing backprop via local loss optimization. International Conference on Artificial Intelligence and Statistics, pp. 9626 9642. PMLR, 2022.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceed- ings of the 33rd Internalional Conference on Machine Learning, 2016.
Amir F Atiya and Alexander G Parlos. New results on recurrent network training: unifying the algorithms and accelerating convergence. IEEE transactions on neural networks, 11(3):697-700 709, 2000,
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-mooivateed deep learning algorithms and architectures. In Advances in neural information processing systems, volume 31, 2018.
Yoshua Bengio. How aulo-encoders could provide credit asignmmnt in deep nelworks via larget propagation. arXiv preprint Xr Xim:1407. 1477. 7906, 2014.
Yoshua Bengio. Deriving differential target propagation from iterating approximate inverses. arXie preprint arXiv:2007.15199, 2020.
Yoshua Bengio and Paolo Frasconi. Diffussin of context and credit information in markovian models. Journal of Artificial Intelligence Research, 3:249-270, 270, 1995.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. JEEE transactions on neural networks, 5(2):157-166, 1994.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXin:1308.3432 2013.
Miguel Carreira-Perpinan and Weiran Wang. Distributed optimization of deeply nested systems. In Pro- ceedings of the 17th International Conference on Artificial Intelligence and Statistics, 2014.
Enrique Castillo, Bertha Huujarro-Berdinar Oscar Fontenla-Romoro, Amparo Alonso- Betanzoss and Yoshua Bengio. A very fast learning method for neural werwsrks based on sensitivity analysis. Journal of Machine Learning Research, 7(7), 2006.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Damitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statis- tical machine translation. Xixiv preprint arXiv:1406. 1078, 2014.
Brian Crafton, Abhinav Parihar, Evan Gebhardt, and Arijit Raychowdhury. Direct feedback alignment with sparse connections for local learning. Frondiers in neuroscience, 13:525, 2019.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Francis Crick. The recent excitement about neural networks. Nature, 337:129-132, 1989.
Wojeioch Marian Czarnocki, Grzegorz Swirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 3/th International Conference on Machine Learning, 2017.
Sander Dalm, Nasir Ahmad, Luca Ambrogioni, and Marcel van Gerven. Scaling up learning with GAIT-prop. arXiv preprint arX.0:2102.11598, 2021.
Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1-2):37-75, 2014.
Maxence Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Trina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. arXiv preprinl Xiv: 13415, 2022.
Michael Fairbank, Spyridon Samothrakis, and Luca Citi. Deep learning in target space. Journal of Machine Learning Research, 2368):1446, 2022.
Thomas Frerix, Thomas Möllenhoff, Michael Moeller, and Danicl Cremers. Proximal backpropagation. In Proceedings of the 6th International Conference on Learning Regresenlations, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. The MIT Press, 2016.
Akhilesh Gotmare, Valentin Thomas, Johanni Brea, and Martin Jaggi. Decoupling backpropagation using constrained optimization methods. In Credit Assignment in Deep Learning and Reinforcement Learning Workshop (ICML 2018 ECA), 2018.
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation, volume 105. Siam, 2008
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled Cayley transform. Proceedings of the 35th International Conference on Machine Learning, 2018.
Geoffrey E Hinton, Terrence J Sejnowski, and David H Ackley. Boltzmann machines: Constraint satisfaction networks that learn. Technical report, Computer Science Department, Carnegie Mellon University, 1984.
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem so- lutions. International Journal of Uncertainty, Fuzziness and Knowledge- Based Systems, 6(02):107-1) 116, 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)):1735-1780, 1997.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Daniel Kunin, Aran Nayebi, Javier Bigastuy-Brenaa Surya Ganguli, Jonathan Bloom, and Daniel Yamins. Two routes to scalable credit assignment without weight symmetry. In Proceedings of the 37th International Conference Machine Learning, 2020.
Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording, Learning to solve the credit assignment problem. In Proceedings of the 7th International Conference Learning Representations, 2019.
Quoe V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. Xix preprint or Xrx 5055500.00994 2015.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Yann Le Cun, Conrad C Galland, and Geoffrey E Hinton. GEMINI: gradient estimation through matrix inversion after noise injection. In Advonces in Newral Information. Processing Systems, volume 1, 1988.
Yann LeCun. Une procédure d'apprentissage pour reseau à seuil asymétrique. Proceedings of Cognitive 85, pp. 599-604, 1985.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. http://yaannlccum.comm /exdb//e////// haabb//int//... 1998.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. Machine Learning and Knowledge Discovery in Databases. Springer, 2015.
Mario Lezcano-Casse and David Murtoez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):1-10, 2016.
Nikolay Manchev and Michael Spratling. Target propagation in recurrent neural networks. Journal of Machine Learning Research, 21(7):1-33, -33, 2020.
Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Using Large Corpora, 273, 1994.
Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F. Grewe A theoretical framework for target propagation. In Advances in Neural Information Processing Systems, volume 33, 2020.
Alexander Meulemans, Matilde Tristany Farinha, Javier Garcia Ordonez, Pau Vilimelis Accituno, João Sacramento, and Benjamin Grewe. Credit assignment in neural networks through deep feedback control. In Advances in Neural Information Processing Systems, volume 34, 2021.
Marvin Minsky, Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30 30, 1961.
Piotr Mirowski and Yann LeCun. Dynamic factor graphs for time series modeling. In Machine Learning and Knowledge Discovery in Databases. Springer, 2009.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, volume 29, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. arXiv preprint arXiw:1211.5063, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, Jamos Bradbury, Gregory Chanan, Trevor Killcen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit. Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- gigh-performance deep learning library. In Advonces in Newral Information Processing Systems, volume 32, 2019.
Pieter R Roelfsema and Arjen van Ooyen. Attention-gated reinforcement learning of internal representations for classification. Neural computation, 17(10):2176-22) 221-2244,, 2005.
Richard Rohwer. The "moving targets' training algorithm. In Advances in Neural Information Processing Systems, volume 1989.
Vincent Roulet and Zaid Harchaoui. Differentiable programming la moreau. In International Conference on Acoustics, Speech and Signal Processing (TCASSP), 2022.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Vincent Roulet, Siddhartha Srinivasa, Dmitriy Drusvyatskiy, and Zaid Harchaoui. Iterative lincarized con- trol: stable algorithms and complexity guarantees. In Proceedings of the 36th International Conference Machine Learning, 2019.
D. E. Rumelhart, G. F. Hinton, and R. J. Williams. Learning Internal Regresentations by Ervor Propagation. MIT Press, Cambridge, MA, USA, 1986.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression.
Newral Computation, (2)::33-242, 1992.
llya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning, 2011.
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Training neural networks without gradients: A scalable ADMM approach. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
P. J. Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thosis, Harvard University, 1974.
Paul Werbos. The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting. Wiley-Intersciencee, 1994.
Sam Wiseman, Sumit Chopra, Marc-Aurclio Ranzato, Arthur Szlam, Ruoyu Sun, Soumith Chintala, and Nicolas Vasilache. Training language models using larget-propagation. arXiv preprint arXiv: wrXxi:1702.000 0021100207777777 770, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms.
2017.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Appendix Outline
The Appendix is organized as follows.
Sec. A recalls how gradient pro-ppropagation works for RNNs.
Sec. B details the implementation target propagation.
Sec. C details (i) the interpretation of the regularization as a counterpart of the noise injoction, (ii)
the differences belween TP and BP, (iii) the interpretation of TP as Gauss-Newton method.
4. Sec. D details the parameters used in experiments and presents additional experiments.
A Gradient Back-Propagation in Recurrent Neural Networks
Given differentiable activation functions the training of recurrent neural networks is amenable to opti- mization by gradient descent. The gradients can be computed by gradient back-propagation implemented in modern differentiable programming software (Rumelhart et al., 1986; Werbos, 1994; Paszke et al., 2019; Abadi et al., 2015). The gradient back paok-prooagation algorithm is illustrated in Fig. 9. Formally, the gradients are computed by the chain rule such that, for a sample (y. x 21.r), 0n = (Whn, Wans bn), and ý = F0( the predicted output,
ae (y, 9) 00,
Oh, ah, ag 08
00, Th, oh, 05
The term /0h, decomposes along the time steps
Oh.
Oh. The s=t-1 0h,-1
II
As grows, the norm of the term 0h,/ may then either increase to infinily (erploding gradienls) exponentially decrease to 0 (vanishing gradients). This phenomenon may prevent the RNN from learning from dopendencics between temporally distant events (Hochreiter, 1998). Several solutions were proposed to tackle this issue, including changing the network architecture (Hochreiter Schmidhuber, 1997). Hessian- free optimization (Sutskever et al., 2011), gradient clipping and regularization (Pascanu al., 2012), or orthonormal parametrizations (Arjovsky
al., 2016; Helfrich et al., 2018; Sascano-Casaaee & Martinez- Rubio, 2019). We consider here propagating targets instead of gradients first presented by LeCun (1985) and recently revisited by Bengio (2014) 4);; Lee et al. (2015).
B Detailed Implementation
B.1 Target Propagation for RNNs with final outputs
As detailed in Sec. 3, target propagation with linearized regularized inverses amounts to move along an update direction computed by forward-backward algorithm akin to gradient propagation. The itcrations of linearized target propagation are then summarized in Algo. 1 and Algo. 2.
In the implementation of the regularized inverses, since the inverses of activation functions such the sigmoid or the tangent hyperbolic numerically unstable, we consider projecting on subset of a(Id))). For the hyperbolic tangent, clip the target to [-1 + E, tangent activation function, the projection is then (T)) = mnii(max((;,.. - + E), 1 c) for Rd4,, To read Algo. 2, we recall our notations for 0 = (Wnn, Wah, bb, Way, by):
for 10-3, Concretely, for hyperbolic
go(h>) s(Wnyhr + by),
(5)
fo, a(WzL.r Wht bn),
(6)
(v.( = (WIT, W.. + -r -iT (a- (a (T(v)) - Won - b.).
(7)
Note that Algo. 2 can also be used for element-wise or linear with respect to the sample of sequence-output, pair.
of sequence-output pairs since all operations are either
17

---

Published in Transactions on Machine Learning Research (01/2023)
Zt
8g
y
0
ho
f
f
f
g
L
((0,0)
h1
h
ht
h
hr
fxf
0x
fuf
ag
A1
Ar
0t/0g
8
f
f
00g l
0ay
Fig. 9:
Gradient back-propagation for RNNs.
Algorithm Stochastic learning with target propagation
Inputs: Initial paramcters ((0) (Whn, Wat, bR, Whys by) of an RNN definod by Eq. (5) and (6), stepsize
na, lotal number of iterations K for k= =...K do
3:
Draw sample or a mini-batch of scquenccs-output pairs (II:T; v)-
4: Compute
ug = (wo.: uo,) = (P(()()-1) x1:T. v),
where TP Algo. 2
Update the parameters as g(k) ((x 70210. end for
B.2 Target Propagation for GRU Networks
B.2.1
Formulation
Starting from ho 0, given an input sequence Ti,..., the GRU network (as implemented in Py- torch (Paszke et al., 2019)), iterates for = 1, I,....T,
fm,(he-1) = oWWi in + Whm h + bm)
(8)
2t fz,(14-1) = (Wi. + Whshe-1 + bz)
(9)
ni = fn, = tanh(a (Wn + din + me O (Whahu-1 + bha))
(10)
h4 = fh.c(h:-1,, n4) = (1 - 21) @ h + 2t O ngg
(11)
where @ is the Hadamard product, a is the sigmoid function. In the following, we will denote simply (0m: 22 0n) the parameters of the network with
8, = (Wim. Whm. bm),
0₂ (Wiz, Whs,
0, = (Win bins W bpn)
The output of the network is, e.g., soft-max operation the hidden state computed at the last step (if applied to an image scanned pixel by pixel for example), See the main paper for the expression of the output in that case.
B.2.2 Modifying the Chain Rule
The underlying idea of our implementation of larget propagation in differentiable programming framework is to mix classical gradients and Jacobians of the inverse of the functions. Denote for a given output loss L
CCompared https://pytorch... httpp///wpaa...o///cccc///////ee
GRU html, we used single variable b..
bim + homs for
18

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 2 Proposed targel propagation algorithı
1: Parameters: π a projection onto a susbet of a(Id),, stepsize Th, regularization y.
2: Inputs: Current parameters 0 = (f), By) with 0₁ = (Wh.: Wan, ba). By = (Why by) of the RNN, sample of sequences-output pairs ().r..9.
Forward Pass:
Compute and store V (W Whh W giving access to (ve) defined in Eq. (7).
Initialize hg = 0.
6: for t= 1=......" do
7:
Compute and store h4 fo. fo,.oo((h-11. Dan, gohh))
On (hef),(hs))
8: end for
9: Compute and store gly.o (hr)),
de, go (h,))))
10: Backward Pass:
11: Define X O, f (I+));
uo, = 0n, ely. (1+))
12: tor== 1 do
13:
Compute de-1 Sn, (lt)T
14: end for
15: Outputs: Update directions for tn, Oy:
ua, -ôo, E(v.
computed on a given mini-batch with the current parameters 0, òc/Ôh, the direction brok-propa by our implementation target propagation until the step h4. The directions for the parameters of the network can be output as
dc
Oh, dc
00
30 the
The main task to define Ôc/ôh-11 given åc/ōh, and appropriate regularized inverses. For that, we start with the chain rule for ah,/Oh,- and we will replace some of the gradients by Jacobians of regularized inverses at some places.
Classical chain rule.
We have
dh. thu-1
= (-
diag(h + diiag(1 xx) + dz
On, diag(z))
0hte1 (-
diag(n)) + Oh,-1
(12)
Ohe-1
dz. (diag(n:) - diag(h:-1)) + Thy-1
On, diag(z)).
diag(1 - 21) + Jh,-1
(13)
Now for Onz/Ohe-n we further decompose the function fn,e as fn. ge O at), with ge(u) Mal(Wini + bin + u) and a, = ((bb-1) Wanhe- + ban- We then have, denoting my O az
One
Ôme diag(a:) + Th Thh-1
da, diag(m)) Vge(p),
(14)
Th,1
with (p) = diag(tam (taagttan''rg (Wan + bin + p)).
Inverses. Now, the variables 2,, my and a; are functions of hy that incorporate a lincar operation and that can be inverted. Namely, we can define the following regularized inverses
ftr,(ve) (WT tm Wmm + -rI)-' hm (o ((x)) Wi - bm))
fl((1)) = (W.. W + + - - (a (oo) Wis - -b)) (vx) (W An Whn + W (T bkn)
19

---

Published in Transactions on Machine Learning Research (01/2023)
We can then do the following substitutions in (q. and (14),
Ome Ôme O a5,, (
Oz
du
dai
dai
(z))
aet (a))
Ōh,-1
δhr-1
0h,-1
δhr-1
Thg_1
to define the quantity back- propagated by target propagation.
Note that by taking the gradient of the inverse we can ignore the biases and the inputs. Namely, we have for example 0 2lmme (m)) diag((o-1))) (m))()) (W hm Whm +rl)-1, hence
(m.) = (Win hm + r r) (rrr)-'W W hm diag((o -Y(m)))
The expression for 0fl, is identical. Since & affine, we have simply (at)) = (Wit Whn + +rr) +rr)-IW. WWT hn
Summary, Combined together, we get, denoting de = àc ōh."
dc = (1) @ de + 0z), ((m - 1 h1-1) o de)
δh,-1
+ (m)))(m)) (a: 2 tamh'' tann'(WWnn + bm u) o 2 @d)
+ +et (a,) (mg @ tanh (Win + bin + uu ② 2 2 ① de)
=(1 (1- de (W Wh= + +rl)- ()a)) (n+ h dz)
+ (W hm Am + 1) Wiim ((o (m O a c tanh' (Win In + bin + u) o 2 d.)
+ (W Whn + 1r1)--WTT, (m, @ tanh' (Win + bin + u) 3 ② de).
This provides rule propagate targets through linearized regularized inverses.
B.3 Target Propagation for RNNs for Word Prediction
For the word prediction task, we consider RNN that outputs a prediction each time-step. Namely, the input-output samples consist in two sequences of words (#irr, where Yt is the word following æ in a sequence of words. The RNN consists in first embedding the input words I; into finite dimensional vector 7, = Dz, with D Rd, where is the vocabulary size and dN is the embedding dimension given that a is represented by a one-hot vector of size v. The RNN treats the embedded input as described in Sec. 2 and outputs prediction at each time-step. Namely, the RNN can be summarized as outputting a sequence of predictions d1.7 from a sequence of inputs Fi:r as
dc = (he) = s(Wnghe + by) L (1....T}
h = So, = a(W.r Dx, + Waahe + by) for LE (I,....T)
where is, e.g., the soft-max function and is a nonlinear operation such the hyperbolic langent function and we fixed hg 0.
To implement TP in this case, we consider making one step of gradient descent on the losses of the prediction, namely, computing
" = h; - mn Elve, go (he)) for t e 71.
Then we consider back prop-proogg the targets
1 h4-1 + (h: (we h4 + ht) for E (T....,
starting from v, = h, such that if only h, was used to predict the output we retrieve the implementation of TP presented previously. The parameters of the network are then updated as presented in Sec. 2.
From a differentiable programming viewpoint, the computational scheme of our implementation TP is the same as the one of gradient back- propaaarpng once considering the displacements u, u, h4 and T
X 0t h4, and replacing Or (he by Onld, (h+) presented in Fig. 10.
20

---

Published in Transactions on Machine Learning Research (01/2023)
y1
Dt
Yt
T
yr
On
h
h
ht
h
h
ho
f
f
f
0y
g
91
l
00/0g
0e
0l/0g
0g
0g
0g
ft
f
fz
3h ft T
0
0
0
0 l
u.g
Fig. 10: Computational scheme of TP implemented for RNNs with intermediate outputs.
21

---

Published in Transactions on Machine Learning Research (01/2023)
C Theoretical Insights
C.1 Regularization and Noise Injection
In Section 2, we mentioned how the rogularization used in the analytic formulation of the inverse layers could be interpreted as a counterpart of the noise injection heuristic used to learn parameterized inverses. We formalize this statement below. Recall that the transition functions take the form
fo.o(he) aWWak Whnh + bn)-
(15)
Perturbed versions of this transition function read
(he) = a((W's + Zan)az + (WAr + ZAN)ha-1 bn + 2h);
for 2 ~ NO0 1) decomposed into independent Gaussian random matrices Zeh Zun and random vector Zhe all of which having independent Gaussian random entries with mean zero and variance o².
Lemma C.1. For vt a(IR" and provided that the activation function is la Lippspppshtz sonechitz-continuuuu, the regularized inversion (v))(v)) (W
h.s. Wnn + +r1)- (v:) W Weh bb) of fe.t defined in (15) min- imizes upper-bound on the variational formulation of the inverse subject to random perturbations, i.e., vell2 for o² = r/dn.
min..., [0-, -==. (1l
Proof. We have for U ((R and a, t--Lipcchitz-contiiuuuss
(11012.(1=-1)) voll Ez.n. |la (( w/ + Zuh)aa (Whr + Zhh ( by. + 2n) vel1
S LaEz.r on (Wzh + Zzh)az + (Wha- Znh) + b4 + 24 - a (ve)l1
The minimizer of the above upper bound is then
6--1 argmin Ez,,. Znn.an Zn..n (W + Zan) + (Whh + Znn)uz-1 by + - (x:)1
ER-h
argmin Ez, Ez.n.Znrrt Zan P IERt)
2 ( + Znn) (TnT)(WW + Zhh)ut-1
"W + T (a- (Wah + ZZn) bb, 2n)
1
argmin ---
-/- 1 (W Ww + E m 12 o
WT, (a -w/)) Wan bn)
-WIJ. Wh. + d. w - (v:) - WZ b,) = (c1),
for (1)) the regularized inversion used in our implementation with r = dg.02.
If the activation function is the identily, the regularized inversion f0, is the exact minimizer of the variat ional formulation of the inverse subject to random perturbations. If the activation function is not the identity, we can still quantify the approximation used by using regularized inverses as shown below.
Lemma C.2. Consider an injective and Lapiscccittz continuous activation function. For E a(Ped)), denote the objective of the variational formulation of the inverse subject to random perturbations as F(v) Ex~Ni0000 1) Dll.feez. (1) - vxll2 and denote t (w)) the approzimate minimizer given by the regularized inversion for r di We have that
F(6) min F(v) < (1 vERth
1- Ma.6
F(6) + Ma,5E6
la
where, for o* E argmin F(u), denoting (v)) = Wan + Whh + bu, we defined for any 0, |la(u) a(w)lla
58 = |111/288 - (100zze((")) a -vx))lE (vx)))1Be-1111//22 11:11/200 dz.
Ma,8 umde (. ".]==la0 (vz)
inf
||u
w|l2
such that ma,s O since is injective and es vanishing for 8 + 400.
22

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. Decompose the transition function fe,t as fa, 00,s, where Φ0,(11) Wa + Whne' by and (0) (Wan + T + (Whn + h4 + by + zn. Denote v² arrgminceR*, F(v). We have
/
1|e11//20 dz 10(00+2,((0")) vellze-11 1=11//202 vel12c-1-118/200 dz / |a((o+2,((")) 0z 1
F(v")=
1211250
/|el1226
11(08++,1(0")
vell2e - /20 dz
1=11//20
1
Ma a (w; |=:1//20 dz
1=11a50
N IV3 IV ma,5 /
de - m, [100+2.(00") - a- (v:)) - -1=111/200
ma, /
1100+2,(00) -a (mx)) 11:1:/200 de ma,
mo, - (D) Ma,8Eá;
where in (i) we used that argmin, E. ( - a -vv()1 and we defined
inf Ma,6 u= 1, 18 w=a (v))
[a(u) aww)lle
11:12200 / 1100+2,(00)) a-
/
(w:)lle 11=1//20 de.
I|u w|l2
Lemma C.2 shows that if the solution of the variational formulation of the inverse subject to random pertur- bations is in region of the activation function where the activation function is nearly the identity, i.e., such that ma,5 1 for, e.g., 8 = 3o such that Eς < 1, then the regularized inversion of the transition function we propose nearly minimizes the objective of the variational formulation of the inverse subject to random perturbations.
C.2 Gradient Back- Jack-Propagation vs Target Propagation
Lemma 3.1. The difference belween the oracle relurned by gradient back- propagaroonnio Oo (y, Fo(x1:r)) and the oracle returned by target propagation ue can be bounded
|10g é (y, Fe(21.)) uo S
sup Σ ca 00, (b00).fe ((. t=.....
ll0 fo,((hr-1) Shefdieli)) I1-
where at E'- 1-- with supt=1.... fo. sup 1 10h T1I l|. For regularized inverses, we have, denoting W Wah + Whhhh-1 + bb,
(0.. Sn. (hh [WTT. (IVa(z)) (IVa(z)-Va(z) |||||+||11100 ||I-(WTT
+
Proof. The first claim is direct application of Lemma C.3 and the second claim follows from the formulation of the regularized inverse, using that Va -(h.) = (aha- = Va(2))
Lemma C.3. Given A An, B ...,, Bn E Rpxp, for any sub-muulippicative malvie norm || ||, and any 1 1stán,
II A II B
<
1=1
1=0
where sup: ||ll,, b supı=I.... [Bll and 8 sup
|| Bill-
23

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. Define for 2 1, 6t
||
A, II Bill. we have
8e -
A
II i=1
A
I 1=1
B;
+ +A. B,)
II Bi S +
86
Σ a'b---i
i=1
1=0
A convergence to stationary point for TP can be derived from classical results approximate gradient descent detailed below (the proof is akin to the results of Devolder et al. (2014)).
Corollary C.4 (Corollary of Lemma C.5). Denote EN a bound on the difference between the oracle returned by gradient back- propagation and by objective is I-smooth and the stepsizes of TP chosen such that 7 7h7o 1/TL, after K iteralions, get
both applied the whole dataset. Provided that the
(1 2 (c((()) - -mingERa [()))
1
K Σ
min ke(0,...KKII)
+
o
K
k=0
where L(0)
n Eis (Fg(F v.) with F0( the output of the RNN sample (# v.) and E the chosen loss.
Lemma C.5. Let f Rp - R be an L-smooth function. Consider an approzimate gradient descent on f S Ex. After K
with step size 0 S y S 1/L, i.e., ((k+1) ((k) (((()) where [[()()())))))))))))))))))))))[ ( iterations, this method satisfies,
[f 2(f()(0)) - mingeR f(0))
min ke -1
Σ E
K
k=0
&(((())) - ()((())) for all 2 O. By L-smoothness of the objective, the iterations of the approximate gradient descent satisfy, using in (i) that Ly < 1,
Proof. Denote (*))
- B(O)) + I.
f(g(++1)) s {(((0)) V -(
2 -10((+1))8((111
>1 (((4)) )1 ((((())) gg())=++ + In?
= -(((())
[ + g
(1 1 47
L22
-(((()
(
I[I (((((( (k))
2
(1- 4
2
$ F(0())
113
l1g (0
+ 71 L - (1ell
2
2
by
T12
+ 7(1 [n)
$ {((0())
+
2 1lg (k)
(1) (I + (1
2
2
$ {((00)
f(
k) 11 l1g (k)
Summing from k 0 to K - and rearranging the terms, we get
Vf((())) 1 2(f()()) minder f(0))
K-1
Ek-
km
Taking the minimum of (((())) and dividing by K we get the result.
24

---

Published in Transactions on Machine Learning Research (01/2023)
C.3 Target Propagation Gauss-Newton updates
We discuss the interprelation of Targel Propagation (TP) a Gauss-Newton (GN) method which proposed by Bengio (2020) and Meulemans et al. (2020). As already mentioned in Sec. 3, the main similarity between TP and GN is the fact that both TP and GN use the inverse of the gradients or approximations thereof. In this section, we shall discuss this interpretation for feed-forward networks to follow the claims of Meulemans et al. (2020). Namely, we consider here a network defined by T. weights Wi,..., Wt and I. activation functions o1,...,,z which transform an input zo into an output zy by computing (no biases were considered by Meulemans et al. (2020)),
f((=111) = a a((Wax=1) for EE [I....,L}
Denoting 6(x, 0) the output of the network for an input x = zo, with (W....., W2) being the parameters of the network, the objective consists in minimizing the loss between the outpuls of the network and the sample outputs, L.., minimizing Cly, 6(x, 0)) for pairs of input-output samples (zz y).
GN step. Recall first the rationale of GN step for such foed-forward networks with squared-loss, which amount to solving
min Σ 1bb(,, v.l1.
PERP
A GN step amounts linearize the non-linear function & around current set of parameters ((k) and solve the corresponding least-square problems to define the next of parameters, i.e,
g(++1)_ argmin 0
1,, + aodla,, ( - 0(*)) - -vil1
n
g(k)
Σ 0oo(I, ((0) (l.0),, (()))) o(6))T]
Σ aoo(2, g(h) (ole, 21)
To consider TP as an approximate GN method we need the following considerations.
1. Consider the itcration on mini-batch of size 1, s.t.
@(++1) = (A) (Ogb(z,, ((0) (00)),0(1 0(0)) ((4)))) (B0l(E ((0) (0(x;;000) - =1))
Consider that the gradients of the networks are invertible, s.t.
0(++1) = ((k) (0ró(z;, (Bnb(2r,,())) (0(2,, 000)), - Wi) wi)
Consider updating only one of parameters 0: W1 s.t.,
-T (p(a,,00)) of ((++1) = 0f - -0n ( -114))-11)) -1)).
with
dg 0(, olk)) = 0o, fi f0r 00.
so that, provided that all matrices inside the matrix multiplication are invertible, we get -
0g 6x,, (00)
Finally, ignore the last inversion and replace it by the gradient on the parameters 81, then we get an iteration similar to TP, with
g(++11)) 0(h) 0o (0) .......( f C(y,
for L squared loss. Namely, we keep the inversion of the gradients of the intermediate functions.
Our objective here to question whether viewing TP as GN step with the approximations explained above is meaningful or not.
25

---

Published in Transactions on Machine Learning Research (01/2023)
Does the original TP formulation approximate GN? Meulemans et al. (2020) start by considering the original TP formulation, i.e., largets computed = (/x+++1) ) for 1h approximate inverse of fa and with ez- = IL nd dl,, Meulemans et al. (2020, Lemma 1) show then that, provided that we use the exact inverse, th = ft fr
L-1 Ar. = Ut - 2t = -1 II Oz for 0o,l(y, + O(1²).
(Meulemans et al., 2020, Theorem 2) conclude that (i) for mini-batches of size 1, (ii) for a squared loss, (iii) for invertible fi, as 0, TP uses Gauss-Newton optimization with block diagonal approximation to compute the targets in the sense that as + 0,
Ar, ~ - n1 (Se+1 o... ST.) (T.)-((x)))
As the stepsize of any optimization algorithm tends to O, they all are the same, since the update would be 0 in all cases. To make the claim of Meulemans al. (2020) more precise, the constants hidden in O(7²) need to be detailed in order to understand in which regimes of the stepsize the approximation is meaningful.
Assuming the inverses tha to be &s Lipschitz continuous and Ly-smooth (i.e. with L continuous gradients), quick look at the proof of Lemma 1 of Meulemans et al. (2020) shows that
L-1
Us - = -1 II
(c,5s(((z)) 10, + &
s=t
11e/12 _ 6,
8, L 62 + 100.+1 + b, ((,,z))11 for s (t,...,,-111
La x² 1z, Cly, 21).11)
62 < 2
Hence the approximation error is then of the order of ||&:12 I (11 Eg) +0())) that is the approximation error may valid for 17 < (L [(y, xL) xz) (I- ) 1/2 Yet, in practice, TP does not appear use very small stepsizes. Moreover, the similarity of TP with GN could explain its efficiency, then by the reasoning of Meulemans al. (2020), the original TP formulation should be efficient. Yet, the original TP formulation has never been shown to produce satisfying results.
Does TP with the difference target propagation approximate GN? Meulemans et al. (2020) make similar claim for TP with the Difference Target Propagation formula, i.c., dt Iı + ()((+)1) the (2+11). Namely, Meulernass al. (2020, Lemma 3) show that
T-1
Az.
02 F4 7 II
Or, (os))) Oz, ((y, 0(12)
Once again, for the claim to be meaningful beyond infinitesimal stepsizes, the terms in O need to be detailed. If we use linearized version of the difference target propagation formula as presented in (2), namely dt 2t = Ozca Vi (x((1+1)" (1e41 I++1), then we have the equality
Ar.
=
II 0z (x.)) Oz, ((v,+1)
and the idca that TP could be seen an approximate GN method may be pursued in a meaningful way. However the error of approximation of the inverse of the gradients must be taken into account in order understand the validity of the approach.
26

---

Published in Transactions on Machine Learning Research (01/2023)
Propagating the approximation error of the gradient inverses. We compute the approximation error incurred by composing gradients of the inverse instead of inverses of gradients. Formally, the approxi- mation error for one layer can be estimated under the assumption that
(e(f((z(11) = It-1 + e(xx-1),
(16)
with an e-Lipschitz continuous function and the assumption that the minimal singular value a of 0z (x is positive.
The function priori depends on the parameters of the layer; we ignore this dependency and simply consider to be e-Lipschitz continuous for all 0. For function its Lipschitz contimuity constant is = sup,, jupllllas1 [z) Al[2 = sup, 110xe(<)) where || ||-I| denotes the spectral norm. By differentiating
both sides of Eq. (16), we get
0 (I++ I I+0,e(xe-1).
By assuming the minimal singular value of 025e (3 to be positive, we get that ( is invertible and so
= - ([
Hence (24) is o (1 + e) Lipschits-continuouss and
(0=f((((=))) - 0xd1(2)11 S E
(17)
Now for multiple compositions, using Lemma C.3, we get.
(1 al
(0=fi ".. (0.fL( d (211(21) 0. S ah.
Therefore the accumulation error diverges with the length L of the network as soon as 2 1
Testing the hypothesis that TP could be interpreted as using GN updates directions. Here we come back to the sctting of RNNs presented in the paper. In this case the length of the compositions of layers is and according to the previous discussion, the error of approximation of the product of the inverse of the gradients by the product of the gradients of the approximate inverses could easily diverge 7 grows (long sequences). Nevertheless, by using analytical formulas for the inverses, we can ensure that the approximation error is zero, which would correspond then to the ideal setting where TP uses GN update directions for the hidden states.
Formally, in the context of RNNs, a Gauss-Newton update direction for the hidden states given as (ignoring the inverse of the output function)
II (0nfet1,0(ha))) TT Ore(v.go(h+)) s=++1
If no regularization is used in the definition of the regularized inverse, i.c., if we use
fo. (WiJ. " (a-1() - Wanke- - bn),
which requires the inverse of Wss to be well defined, we would get
0f (h.) (h)
The updates of TP using the formula (2) would then be exactly the ones of a GN on the hidden states presented by Meulemans et al. (2020), i.e.,
- h4 =
II (0nft++.0(hh))- Ogl(y, go( -=t+1
So by considering our implementation without regularization, we can test whether the interpretation of TP as an approximatc GN method is meaningful in terms of optimization convergence. As shown in Fig. 7, it appears that regularizing the inverses is necessary to obtain convergence, hence the interpretation of TP as GN may not be sufficient to explain why TP can converge.
27

---

Published in Transactions on Machine Learning Research (01/2023)
BP
TP
7
7h
7
Temporal order problem length 60
10
10 10-2
10-1
10
Temporal order problem length 120
10
10-2
10
1
Adding problem
10-3
10-1
10-1
MNIST pixel by pixel
10-6
10-1
10-1
MINIT pixel by pixel permuted
10-4
10
10-1
CIFAR
10-3 3
10-2
10-2
10
PashionMNIST with GRU
10-2
10-1
10-2
1
Penn Treebank with RNN
10-2
10-3
10-1
1
Table 1: Hyper-parameters chosen for Fig. and
D Experimental Details
D.1 Initialization and Hyper- Paramerrs
Initialization and data generation. In all experiments, the weights of the RNN are initialized as random orthogonal matrices, and the biases are initialized as as presented by Le et al. (2015) and Manchev & Spratling (2020). For all experiments, the data was normalized, as done by Manchev & Spratling (2020). We kept setting as similar as possible as the one of Manchev & Spratling (2020) to be able compare target propagation with regularized parameterized inversos.
peperrparameters. In the synthetic tasks, for BP we used a momentum of 0.9 with Nesterov accelerated gradient scheme as done by Manchev & Spratling (2020). Otherwise, we did not use any momentum for the experiment on MNIST pixel by pixel presented the main paper. The learning rates of BP and the parameters of TP were found by a grid-search on a log1o basis and are presented Table We did not add regularization term in the training of the RNNs.
For Fig. 7, we used batch sizes of size 512 and performed grid search for the stepsizes of BP and for the stepsizes of TP while keeping the same regularization and stepsize to the parameters found for the length 784.
Software. We used Python 3.8 and PyTorch 1.6. The RNN was coded using the cuDNN implementation available in PyTorch that highly optimized for computing forward forward-backwaat backward passes on RNNs for gradient back-propagation.
Hardware. All experiments were performed on GPUs using Nvidia GeForce GTX 1080 Ti (12G memory). Each experiment only used one gpu at a time (clock speed 1.5 Ghz).
Time evaluation. On our GPU, we observed that for the MNIST pixel by pixel experiment, 200 iterations (each itcration considering samples) were taking approximately 60s for BP and 800s for TP. Note that with larger batch-sizes the cost of the regularized inversion would be amortized by the fact that more samples treated simultaneously. We kept the setting of Manchev & Spratling (2020) for of comparison.
28

---

Published in Transactions on Machine Learning Research (01/2023)
TP
BP
DTP-PI
8000
8000
Train Loss
1000
1000
0000
6000
2000
50
4000
1000
2000
01
2000
2000
0
0
Iterations
x 10
Iterations
x 10
Iterations
10
Iterations
x11
Iterations
x 10
100
100
100
100
100
75
75
75
75
73
Accuracy
50
25
25
25
25
25
Iterations
x 10
Iterations
x10"
Iterations
x
Iterations
x10*
Iterations
x10
Comparison of our implementation of TP denoted TP, against TP with a Difference Target Propa- gation formula and Parameterized Inverses, denoted DTP-PI and gradient Pack-Propagation, denoted BP. From Left to right: Temporal order task with T = 60, Temporal order task with T = 120, Adding problem with T = 30, MNIST pixel by pixel and MNIST pixel by pixel with permuted images.
Fig. 11:
D.2 Additional Experiments
Comparison of TP with regularized or parameterized inverses. In Fig. 11, we evaluate the perfor- mance of our implementation of TP using regularized inverses and lincarized propagation as opposed to the implementation of Manchev & Spratling (2020) using the difference largel propagation formula and parame- terized inverses all datasets studied by Manchev & Spratling (2020) using the reported hyper-parameters chosen by Manchev & Spratling (2020). We observe that our implementation generally outperforms the one of Manchev & Spratling (2020) except for the addition task where they perform on on-par. As observed in Fig. 5, the difference of performance can be explained by the use of regularized inverses while the linearized formulation used in our implementation has the advantage to easily adapt TP for other architectures.
Visualization of norms of targets. On Fig. 12, we observe that the norm of the displacements X 1y 8 h4 defined in Sec. 3 propagated by TP do not necessarily enjoy more stable norms than the gradients de/ah, propagated by BP as explained in Appendix A for varying along the length of the sequence. This phenomenon is in contrast with the better performance of TP over BP for this task, namely, predicting images from MNIST with RNNs and suggests interesting avenues for future research.
Gradient norms and spectral radius analyses. On Fig. 13, we present, as done by Manchev & Spratling (2020, Figure 3) for TP with parameterized inverses, the evolution of (i) the norm of the ora- cle directions computed by either TP or BP for all transition parameters Oh, (fi) the spectral radius of the transition matrix Whh, on the MNIST pixel by pixel experiment. We observe that TP provides smaller oracle direction norms which allowed for larger stepsizes with slightly less variance of the gradients. On the other hand the spectral radius of the transition matrix appears to grow for both oracles with TP having larger growth at the start.
TP vs BP in terms of time. To account for the additional cost of inversion for cach mini-batch, we consider the convergence of the algorithms in time rather than in iterations. We found that, on average, iteration of BP takes approximately 13 times less time than one iteration of TP our implementation (note that BP benefits from highly optimized implementations for GPU machines, and TP could potentially also benefit from the same optimized implementations). Therefore we ran BP for 13 times more iterations than TP and multiplied the number of iterations by the approximate time needed for each iteration for all
29

---

Published in Transactions on Machine Learning Research (01/2023)
TP
BP
0-
400
10
400
800
800
1200
10
1200
Iterations
1600
Iterations
1600
0.15
2000
10
2000
0.10
10
2800
2800
3200
3200
0.05
10
3600
3600
4000
10-12 12
4000
756
672
630
588
546
504
462
420
378
336
294
252
210
168
126
84
42
756
672
630
588
546
5504
462
420
378
336
294
252
210
108
1226
84
42
Sequence Depth
Sequence Depth
Fig. 12: Norms of displacements or gradients along the sequence pixels analyzed by an RNN to predict
MNIST digits.
BP
TP
BP
TP
BP
100
Oracle Dir. Norm
Spectral Radius
1.06
10
75
Accuracy
1.04
50
10-1
/
1.02
25
1.00
0
5000
10000
5000
10000
Iterations
Iterations
Time in x 10
Fig. 13: Norms of oracle directions and spectral radií along iterations. Fig. 14: MNIST pixel by pixel in time.
algorithms. In Fig. 14, we observe that in time too, TP performs better than BP, which stays stuck at an accuracy approximately 22%.
30


Reviews:
[{"review_id": "vqfTWX0lQH", "reviewer": "Reviewer_Kkxk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper can be summarized as follows:\n\n- The manuscript is well-written except a few minor problems in the statements. A clear review of the TP and BP was also given in the supplementary material.\n\n- In the theoretical analyses, the stability of the proposed TP, and its relation to the BP was explored.\n\n- The proposed method can be implemented easily with the current libraries with autograd.\n\n- In the initial analyses, the TP performs on par with or outperforms the BP.\n\nOne of the main weaknesses of the paper is novelty and claims. Indeed, the paper proposes the method for training RNNs. Therefore, either the proposed method should be extended for training other models (e.g. CNNs), or the title and the claims of the paper should be revised accordingly.\n\nAnother major issue is the limited experimental analyses. In the initial results, the proposed method outperforms BP or perform on par in some tasks. However, the benefits of TP are not clear in these analyses. First, ablation study exploring the theoretical results should be given. Then, TP should be employed on other tasks where RNNs have been used as vanilla models, such as speech or text recognition or analysis.\n\nFinally, in the theoretical analyses, the relationship between TP and BP has been analyzed. However, how this helps to resolve other claims such as solving vanishing/exploding gradients and improvement of learning rates, are not addressed. These claims should be further theoretically/experimentally analyzed or revised.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new approach to target propagation (TP) to train RNNs. For this purpose, they employ regularized inversions of network layers. The proposed approach can be implemented using differentiable programming frameworks. In the theoretical analyses, the relationship between TP and back-propagation (BP) was explored. In the experimental analyses, they examine the proposed TP for several   sequence modelling tasks using RNNs. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "First, the claims of the paper should be revised. The paper propose a new approach for TP for particularly RNNs. For a more general claim, TP should be employed for other DNNs as well. Otherwise, claims should be revised.\n\nNext, according to the revised claims, additional ablation study, additional experiments with other tasks where RNNs have been used successfully, and more detailed comparison with related work should be provided.\n\nIn the theoretical analyses, how the proposed TP approach helps convergence of models and resolving vanishing/exploding gradients, and its relation with Gauss-Newton methods, should be elucidated.", "broader_impact_concerns": "A Broader Impact Statement was not provided. Ethical implications and its broader impact should be discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MJZFHQ7lle", "reviewer": "Reviewer_9aYY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method's primary advantages over the previous target propagation are the computational efficiency and estimation accuracy of the inversion map, which are attributed to regularized analytic inversion. This method involves the computation of inverse matrix, but computation only needs to be done once per sequence, and its accuracy is also guaranteed to some extent. On the other hand, the existing methods require the estimation of the inverse at each time step, and there is no guarantee for accuracy. This improvement (especially the computational cost) over the existing target propagation would be nice.\n\nOn the other hand, the novelty and contribution seem to be low, as summarized below.\n\n- The computational efficiency of the proposed method and back-propagation should be evaluated regarding the complexity to achieve the same accuracy. If there is a performance difference between each iteration of these methods, then a comparison of per-iteration costs in Section 3 is not very interesting. \n- The aim of Lemma 3.1, which bounds the difference between target propagation and back-propagation, is unclear. If this gap should be small, then I am not sure why the proposed method is used. \nIn Appendix, this lemma is used to discuss the convergence of the proposed method, but the obtained bound seems vacuous because the difference between these two steps, which should be smaller than a given threshold $\\epsilon$, is not properly evaluated. \n- Given [Bengio (2020), Meulemans et al. (2020, 2021)], the relationship between the proposed method and the Gauss-Newton method seems obvious.\n- The advantages over existing target propagation methods are the selling point of this paper, but the experiments in the main text are conducted for comparing only with back-propagation. Moreover, the empirical result is not so convincing because the performance is not evaluated for the NLP tasks, which are the standard application of RNNs. \n\n\n**Minor comments** \n- On page 6: the inequality $\\mathbb{E}_{z\\sim \\mathcal{N}(0,\\sigma^2I)}[a^{-1}...] \\leq \\ell_a ...$ seems wrong. It would probably be a typo.\n- The summation regarding the time-step seems missing in the first result in Lemma 3.1. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a variant of the target propagation for training recurrent neural networks. The method builds on the work [Lee et al. (2015)], [Bengio (2020)], and so on. The key differences from the previous methods are to utilize (i) linear approximation eq. (2) of the inversion instead of the difference target propagation and (ii) (regularized) analytic inversion instead of optimizing the parameterized one. \nThis paper shows the competitive computational efficiency with the back-propagation, shows several aspects of the method (e.g., connection with the noise injection, linearized inversion, and Gauss-Newton method), and estimates the gap between the proposed method and the back-propagation (Lemma 3.1). Moreover, the better performance than the back-propagation is empirically verified on synthetic datasets and image datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- It would be nice if the authors could show the convergence more rigorously. If the comparable convergence guarantee with the back-propagation can be guaranteed, the use of this method can also be justified.\n- Can you conduct experiments on some NLP datasets? Moreover, the proposed method should be empirically compared with the existing target propagation methods on the other datasets other than that in Appendix.\n", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7kaJzZ7Yfa", "reviewer": "Reviewer_wpQC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well-written, and I am happy to recommend this paper for publication after major revision. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the paper under review, the authors present a simple version of target propagation based on regularized inversions of neural network layers that sheds light on the relevance of target propagation from an optimization viewpoint, and the proposed target propagation using the analytical formulation, and the proposed target propagation is simple to implement using differentiable programming. The authors verified the efficiency of target propagation in training recurrent neural networks.  Moreover, the authors consider the regimes in which the computational complexity of target propagation can outperform backpropagation. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Below are some details comments.\n\nFrom Figure 1, I cannot see the computational advantages of target propagation over gradient back-propagation. It seems to me that for target propagation, we also need to store the intermediate states in the forward propagation. Moreover, I wonder about the numerical stability of computing the gradient of the inverse of the function.\n\n\nThe experiments are rather simple. I would suggest authors validate them on a few more recurrent neural network benchmarks, e.g., TIMIT and character- and word-level Penn treebank tasks.\n\n\nI would suggest the authors do some visualization of overcoming vanishing and exploding gradient issues using target propagation. Perhaps the authors can mimic what did in Figure 2 in the paper https://proceedings.neurips.cc/paper/2020/file/149ef6419512be56a93169cd5e6fa8fd-Paper.pdf\n\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The article is very well written and in particular it provides a detailed introduction to the subject of target propagation. The reviewers had some concerns about the motivation and correctness of some of the statements, several of which could be clarified during the discussion. Following the rebuttal and discussion, reviewers found that the submission improved in important ways. Particularly, they acknowledged the empirical evidence in support of the proposed method, although it was also mentioned that further convergence analysis (with concerns about the significance and novelty of Lemma C.5 as well as Corollary 4), as well as experiments on different tasks would have made this a stronger contribution. 

Overall, reviewers lean accept. I find that the article addresses interesting topics and provides adequate evidence in support of their main claims. Hence I am recommending accept. In consideration of the above remarks, I still encourage a minor revision. 


 
Decision: Accept (poster)