Year: 2021

Abstract:
Target Propagation (TP) algorithms compute targets instead of gradients along neural networks and propagate them backward in a way that is similar yet different than gradient back-propagation (BP). The idea was first presented as a perturbative alternative to back-propagation that may improve gradient evaluation accuracy when training multi-layer neural networks (Le Cun et al., 1989). However, TP may have remained more of a template algorithm with many variations than a well-identified algorithm. Revisiting insights of Le Cun et al. (1989) and more recently of Lee et al. (2015), we present a simple version of target propagation based on a regularized inversion of network layers, easily implementable in a differentiable programming framework. We compare its computational complexity to the one of BP and delineate the regimes in which TP can be attractive compared to BP. We show how our TP can be used to train recurrent neural networks with long sequences on various sequence modeling problems. The experimental results underscore the importance of regularization in TP in practice.

heading: 1 INTRODUCTION
text: Target propagation algorithms can be seen as perturbative learning alternatives to the gradient backpropagation algorithm, where virtual targets are propagated backward instead of gradients (Le Cun, 1986; Le Cun et al., 1989; Rohwer, 1990; Mirowski & LeCun, 2009; Bengio, 2014; Goodfellow et al., 2016). A high-level summary is presented in Fig. 1: while gradient back-propagation considers storing intermediate gradients in a forward pass, target propagation algorithms proceed by computing and storing approximate inverses. The approximate inverses are then passed on backward along the graph of computations to finally yield a weight update for stochastic learning.
Target propagation aims to take advantage of the availability of approximate inverses to compute better descent directions for the objective at hand. Bengio et al. (2013); Bengio (2020) argued that the approach could be relevant for problems involving multiple compositions such as the training of Recurrent Neural Networks (RNNs), which generally suffer from the phenomenon of exploding or vanishing gradients (Hochreiter, 1998; Bengio et al., 1994; ?). Recently, empirical results indeed showed the potential advantages of target propagation over classical gradient back-propagation for training RNNs on several tasks (Manchev & Spratling, 2020). However, these recent investigations remain built on multiple approximations, which hinder the analysis of the core idea of TP, i.e., using layer inverses.
On the theoretical side, difference target propagation, a modern variant of target propagation, was related to an approximate Gauss-Newton method, suggesting interesting venues to explain the benefits of target propagation (Bengio, 2020; Meulemans et al., 2020). Previous works have considered approximating inverses by adding multiple reverse layers (Manchev & Spratling, 2020; Meulemans et al., 2020; Bengio, 2020). However, it is unclear whether such reverse layers actually learn layer inverses during the training process. Even if they were, the additional cost of computational complexity of learning approximate inverses should be carefully accounted for.
In this work, we propose a simple target propagation approach, revisiting the original insights of Le Cun et al. (1989) on the critical importance of the good conditioning of layer inverses. We define regularized inverses through a variational formulation and we obtain approximate inverses via these regularized inverses. In this spirit, we can also interpret the difference target propagation formula (Lee et al., 2015) as a finite difference approximation of a linearized regularized inverse. We propose a smoother formula that can directly be integrated into a differentiable programming framework.
G adien Back-P o aga ion Ta ge P o aga ion
We detail the computational complexity of the proposed target propagation and compare it to the one of gradient back-propagation, showing that the additional cost of computing inverses can be effectively amortized for very long sequences. Following the benchmark of Manchev & Spratling (2020), we observe that the proposed target propagation can perform better than classical gradientbased methods on several tasks involving RNNs.
Related work. Many variations of back-propagation algorithms have been explored; see Werbos (1994); Goodfellow et al. (2016) for an extensive bibliography. Closer to target propagation, penalized formulations of the training problem have been considered to decouple the optimization of the weights in a distributed way or using an ADMM approach (Carreira-Perpinan & Wang, 2014; Taylor et al., 2016; Gotmare et al., 2018). Rather than modifying the backward operations in the layers, one can also modify the weight updates for deep forward networks by using a regularized inverse (Frerix et al., 2018). Wiseman et al. (2017) recast target propagation as an ADMM-like algorithm for language modeling and reported disappointing experimental results. Recently, in a careful experimental benchmark evaluation, Manchev & Spratling (2020) explored further target propagation to train RNNs, mapping a sequence to a single final output, in an attempt to understand the benefits of target propagation to capture long-range dependencies, and obtained promising experimental results. Another line of research has considered synthetic gradients that approximate gradients using an additional layer instead of using back-propagated gradients (Jaderberg et al., 2017; Czarnecki et al., 2017) to speed up the training of deep neural networks. Recently, Ahmad et al. (2020); Dalm et al. (2021) considered using analytical inverses to implement target propagation and blend it with what they called a gradient-adjusted incremental formula. Yet, an additional orthogonality penalty is critical for their approach to work. Recently, Meulemans et al. (2020) considered using as many reverse layers as forwarding operations. We focus here on the optimization gains of using target propagation that cannot be obtained by adding a prohibitive number of reverse layers. Finally, we do not discuss the biological plausibility of TP since we are unable to comment on this. We refer the interested reader to, e.g., (Bengio, 2020).
Notations. For f : Rp ⇥ Rq ! Rd, we denote @xf(x, y) = @f j(x, y)/@xi i,j 2 Rd⇥p.

heading: 2 TARGET PROPAGATION WITH LINEARIZED REGULARIZED INVERSES
text: While target propagation was initially developed for multi-layer neural networks, we focus on its implementation for recurrent neural networks, as we shall follow the benchmark of Manchev & Spratling (2020) in the experiments. Recurrent Neural Networks (RNNs) are also a canonical family of neural networks in which interesting phenomena arise in back-propagation algorithms.
Problem setting. A simple RNN parameterized by ✓ = (Whh,Wxh, bh,Why, by) maps a sequence of inputs x1:⌧ = (x1, . . . , x⌧ ) to an output ŷ = g✓(x1:⌧ ) by computing hidden states ht 2 Rp corresponding to the inputs xt.
Formally, the output ŷ and the hidden states ht are computed as an output operation following transition operations defined as
ŷ = c✓(h⌧ ) := s(Whyh⌧ + by),
ht = f✓,t(ht 1) := a(Wxhxt +Whhht 1 + bh) for t 2 {1, . . . , ⌧},
where s is, e.g., the soft-max function for classification tasks, a is a non-linear operation such as the hyperbolic tangent function, and the initial hidden state is generally fixed as h0 = 0. Given samples of sequence-output pairs (x1:⌧ , y), the RNN is trained to minimize the error `(y, g✓(x1:⌧ )) of predicting ŷ = g✓(x1:⌧ ) instead of y.
As one considers longer sequences, RNNs face the challenge of exploding/vanishing gradients @g✓(x1:⌧ )/@ht (Bengio & Frasconi, 1995); see Appendix A for more discussion. We acknowledge that specific parameterization-based strategies have been proposed to address this issue of exploding/vanishing gradients, such as orthonormal parameterizations of the weights (Arjovsky et al., 2016; Helfrich et al., 2018; Lezcano-Casado & Martınez-Rubio, 2019). The focus here is to simplify and understand target propagation as a backpropagation-type algorithm using RNNs as a workbench. Indeed, training RNNs is an optimization problem involving multiple compositions for which approximate inverses can easily be available. The framework could also be potentially applied to, e.g., time-series or control models (Roulet et al., 2019).
Given the parameters Whh,Wxh, bh of the transition operations, we can get approximate inverses of f✓,t(ht 1) for all t 2 {1, . . . , ⌧}, that yield optimization surrogates that can be better performing than the ones corresponding to regular gradients. We present below a simple version of target propagation based on regularized inverses and inverse linearizations.
Back-propagating targets. The idea of target propagation is to compute virtual targets vt for each layer t = ⌧, . . . , 1 such that if the layers were able to match their corresponding target at time t, i.e., f✓,t(ht 1) ⇡ vt, the objective would decrease. The final target v⌧ is computed as a gradient step on the loss w.r.t. h⌧ . The targets are then back-propagated using an approximate inverse1 f 1✓,t of f✓,t at each time step.
Formally, consider an RNN that computed ⌧ states h1, . . . , h⌧ from a sequence x1, . . . , x⌧ with associated output y. For a given stepsize h > 0, we propose to back-propagate targets by computing
v⌧ = h⌧ h@h`(y, c✓(h⌧ )), (1)
vt 1 = ht 1 + @hf 1 ✓,t (ht) > (vt ht), for t 2 {⌧, . . . , 1}. (2)
The update rule (2) blends two ideas: i) regularized inversion; ii) linear approximation. We shall describe below that our update (2) allows us to interpret the “magic formula” of difference target propagation in Eq. 15 of Lee et al. (2015) as 0th-order finite difference approximation, while ours is a 1st-order linear approximation. We shall also show that (2) puts in practice an insight from Bengio (2020) suggesting to use the inverse of the gradients in the spirit of a Gauss-Newton method.
Once all targets are computed, the parameters of the transition operations are updated such that the outputs of f✓,t at each time step move closer to the given target. Formally, the update consists of a gradient step with stepsize ✓ on the squared error between the targets and the current outputs, i.e., for ✓h 2 {Whh,Wxh, bh},
✓ next h = ✓h ✓
⌧X
t=1
@✓hkf✓,t(ht 1) vtk22/2. (3)
As for the parameters ✓y = (Why, by) of the output operation, they are updated by a simple gradient step on the loss with a stepsize ✓.

heading: 2.1 REGULARIZED INVERSION
text: To explore further the original idea of Le Cun et al. (1989), we consider using the variational definition of the inverse,
f 1 ✓,t (vt) = argmin vt 12Rp kf✓,t(vt 1) vtk22 = argmin vt 12Rp ka(Wxhxt +Whhvt 1 + bh) vtk22. (4)
As long as vt belongs to the image f✓,t(Rp) of f✓,t, this definition recovers exactly the inverse of vt by f✓,t. More generally, if vt 62 f✓,t(Rp), Eq. (4) computes the best approximation of the
1In the following, to ease the presentation, we abuse notations and denote approximate inverses by f 1✓,t .
inverse in the sense of the Euclidean projection. When one considers an activation function a and ✓h = (Whh,Wxh, bh), the solution of (4) can easily be computed.
Formally, for the sigmoid, the hyperbolic tangent or the ReLU, their inverse can be obtained analytically for any vt 2 a(Rp). So for vt 2 a(Rp) and Whh full rank, we get
f 1 ✓,t (vt) = (W > hhWhh) 1 W > hh(a 1(vt) Wxhxt bh).
If vt 62 a(Rp), the minimizer of (4) is obtained by first projecting vt onto a(Rp), before inverting the linear operation. To account for non-invertible matrices Whh, we also add a regularization in the computation of the inverse. Overall we consider approximating the inverse of the layer by a regularized inverse of the form
f 1 ✓,t (vt) = (W > hhWhh + r I) 1 W > hh(a 1(⇡(vt)) Wxhxt bh),
with r > 0 and ⇡ a projection onto a(Rp).
Regularized inversion vs. parameterized inversion. Bengio (2014); Manchev & Spratling (2020) parameterize the inverse as a reverse layer such that
f 1 ✓,t (vt) = ✓0,t(vt) := a(Wxhxt + V vt + c),
and learn the parameters ✓0 = (V, c) for this reverse layer to approximate the inverse of the forward computations. The parameterized layer needs to be learned to get a good approximation which involves numerically solving an optimization problem for each layer. These optimization problems come with a computational cost that can be better controlled by using regularized inversions presented earlier.
However, the approach based on parameterized inverses may lack theoretical grounding, as pointed out by Bengio (2020), as we do not know how close the learned inverse is to the actual inverse throughout the training process. In contrast, the regularized inversion (4) is less ad hoc and clearly defined and, as we shall show in the experiments, leads to competitive performance on real datasets.
In any case, the analytic formulation of the inverse gives simple insights on an approach with parameterized inverses. Namely, the analytical formula suggests parameterizing the reverse layer s.t. (i) the reverse activation is defined as the inverse of the activation and not any activation, (ii) the layer uses a non-linear operation followed by a linear one instead of the usual scheme, i.e., a linear operation followed by a non-linear one.

heading: 2.2 LINEARIZED INVERSION
text: Earlier instances of target propagation used direct inverses of the network layers such that the target propagation update formula would read vt 1 = f 1✓,t (vt) in (2). Yet, we are unaware of a successful implementation of TP using directly the inverses. To circumvent this issue, Lee et al. (2015) proposed the difference target propagation formula that back-propagates the targets as
vt 1 = ht 1 + f 1 ✓,t (vt) f 1 ✓,t (ht).
If the inverses were exact, the difference target propagation formula would reduce to vt 1 = f 1 ✓,t (vt). Lee et al. (2015) introduced the difference target propagation formula to mitigate the approximation error of the inverses by parameterized layers. In practice, difference-type target propagation appears to be the only known successful implementation of target propagation we are aware of. The difference target propagation formula can naturally be interpreted as an approximation of the linearization used in (2), as
f 1 ✓,t (vt) f 1 ✓,t (ht) = @hf 1 ✓,t (ht) > (vt ht) +O(kvt htk22),
where @hf 1✓,t (ht) > denotes the Jacobian of the inverse of the layer at ht.
We show in Appendix D that the first-order approximation we propose (2) leads to slightly better training curves than the finite-difference approximation. Moreover, our interpretation illuminates the “mystery” of this formula, which appeared to be critical to the success of target propagation.

heading: 3 GRADIENT BACK-PROPAGATION VERSUS TARGET PROPAGATION
text: Graph of computations. Gradient back-propagation and target propagation both compute a descent direction for the objective at hand. The difference lies in the oracles computed and stored in the forward pass, while the graph of computations remains the same. To clarify this view, we reformulate target propagation in terms of displacements t := vt ht such that Eq. (1), (2) and (3) read
⌧ = h@h`(y, c✓(h⌧ )), t 1 = @hf 1✓,t (ht) > t, for t 2 {⌧, . . . , 1},
d✓h = ⌧X
t=1
@✓hf✓,t(ht 1) t, ✓ next h = ✓h + hd✓h .
Target propagation amounts then to computing a descent direction d✓h for the parameters ✓h with a graph of computations, illustrated in Fig. 2, analogous to that of gradient-back-propagation illustrated in Appendix A. The difference lies in the use of the Jacobian of the inverse
@hf 1 ✓,t (ht) > instead of @hf✓,t(ht 1).
The implementation of TP with the formula (2) can be done in a differentiable programming framework, where, rather than computing the gradient of the layer, one evaluates the inverse and keep the Jacobian of the inverse. With the precise graph of computation of TP and BP, we can compare their computational complexity explicitly and bound the difference of the directions they output.
Arithmetic complexity. Clearly, the space complexities of gradient back-propagation (BP) and our implementation of target propagation (TP) are the same since the Jacobians of the inverse, and the original gradients have the same size. In terms of time complexity, TP appears at first glance to introduce an important overhead since it requires the computation of some inverses. However, a close inspection of the formula of the regularized inverse reveals that a matrix inversion needs to be computed only once for all time steps. Therefore the cost of the inversion may be amortized if the length of the sequence is particularly long.
Formally, the time complexity of the forward-backward pass of gradient back-propagation is essentially driven by matrix-vector products, i.e.,
TBP = ⌧X
t=1
2
4T (f✓,t) + T (@hf✓,t) + T (@✓hf✓,t)| {z } Forward + T (@hf✓,t(ht 1)) + T (@✓f✓,t(ht 1))| {z } Backward
3
5
⇡ ⌧(dp+ p2 + pq) + ⌧(p2 + pq), where d is the dimension of the input xt, q is the dimension of the parameters ✓h, for a function f we denote by T (f) the time complexity to evaluate f and we consider e.g. @✓f✓,t(ht 1)) as the linear function ! @✓f✓,t(ht 1)) .
On the other hand, the time complexity of target propagation is
TTP = ⌧X
t=1
2
64T (f✓,t)+T (f 1✓,t )+T (@✓hf✓,t)+T (@hf 1 ✓,t )| {z }
Forward
+ T (@hf 1✓,t )(ht) >)+T (@✓f✓,t(ht 1))
| {z } Backward
3
75
+ P(f 1✓,t ),
where P(f 1✓,t ) is the cost of encoding the inverse, which, in our case, amounts to the cost of encoding g✓ : z ! (W>hhWhh + r I) 1W>hh, such that our regularized inverse can be computed as f 1 ✓,t (vt) = g✓(a 1(vt) Wxhxt+bh). Encoding g comes at the cost of inverting one matrix of size p. Therefore, the time-complexity of target propagation can be estimated as
TTP ⇡ p3 + ⌧(dp+ p2 + pq) + ⌧(p2 + pq) ⇡ TBP if ⌧ p, i.e., for long sequences whose length is larger than the dimension of the hidden states, the cost of TP with regularized inverses is approximately the same as the cost of BP. If a parameterized inverse was used rather than a regularized inverse, the cost of encoding the inverse would correspond to the cost of updating the reverse layers by, e.g., a stochastic gradient descent. This update has a cost similar to BP. However, it is unclear whether these updates get us close to the actual inverses.
Bounding the difference between target propagation and gradient back-propagation. As the computational graphs of BP and TP are the same, we can bound the difference between the oracles returned by both methods. First, note that the updates of the parameters of the output functions are the same since, in TP, gradients steps of the loss are used to update these parameters. The difference between TP and BP lies in the updates with respect to the parameters of the transition operations. For BP, the updates are computed by chain rule as
@✓h` (y, g✓(x1:⌧ )) = ⌧X
t=1
@✓hf✓,t(ht 1) @h⌧
@ht @h`(y, c✓(h⌧ )),
where the term @h⌧/@ht decomposes along the time steps as @h⌧/@ht = Q⌧
s=t+1 @hf✓,s(hs 1). The direction computed by TP has the same structure, namely it can be decomposed for h = 1 as
d✓ = ⌧X
t=1
@✓hf✓,t(ht 1) @̂h⌧
@̂ht
@h`(y, c✓(h⌧ )),
where @̂h⌧/@̂ht = Q⌧ s=t+1 @hf 1 ✓,s (hs) >. We can then bound the difference between the directions given by BP or TP as, for any matrix norm k · k as formally stated in the following lemma.
Lemma 3.1. The difference between the oracle returned by gradient back-propagation @✓h` (y, g✓(x1:⌧ )) and the oracle returned by target propagation can be bounded as
k@✓h` (y, g✓(x1:⌧ )) d✓k  c sup t=1,...,⌧
k@hf✓,t(ht 1) @hf 1✓,t (ht) >k,
where c = P⌧
t=1 Pt 1 s=0 a s b t 1 s
with a = supt=1,...⌧ k@hf✓,t(ht 1)k, b = supt=1,...⌧ k@hf 1✓,t (ht) >k.
For regularized inverses, we have, denoting ut = Wxhxt +Whhht 1 + bh,
k@hf✓,t(ht 1) @hf 1✓,t (ht) >k  kW>hhk ⇣ kra(ut) ra(ut) 1k + k I (W>hhWhh + r I) 1kkra(ut) 1k ⌘ .
For the two oracles to be close, we then need the preactivation ut = Wxhxt + Whhht 1 + bh to lie in the region of the activation function that is close to being linear s.t. ra(ut) ⇡ I. We also need (W>hhWhh + r I)
1 to be close to the identity which can be the case if, e.g., r = 0 and the weight matrices Wh were orthonormal. By initializing the weight matrices as orthonormal matrices, the differences between the two oracles can be closer. However, in the long term, target propagation appears to give better oracles, as shown in the experiments below.
Target propagation as a Gauss-Newton method? Recently target propagation has been interpreted as an approximate Gauss-Newton method, by considering that the difference target propagation formula approximates the linearization of the inverse, which itself is a priori equal to the inverse of the gradients (Bengio, 2020; Meulemans et al., 2020; 2021). Namely, provided that f 1 ✓,t (f✓,t(ht 1)) ⇡ ht 1 such that @hf✓,t(ht 1)@hf 1 ✓,t (ht) ⇡ I, we have
@hf 1 ✓,t (ht) ⇡ (@hf✓,t(ht 1)) 1 .
By composing the inverses of the gradients, we get an update similar to the one of Gauss-Newton (GN) method. Namely, recall that if n invertible functions f1, . . . , fn were composed to solve a least square problem of the form kfn . . . f1(x) yk22, a Gauss-Newton update would take the form x(k+1) = x(k) @x0f1(x0) >@x1f2(x1) > . . . @xn 1f(xn 1) >(xn y) where xt is defined iteratively as x0 = x(k), xt+1 = ft(xt). In other words, GN and TP share the idea of composing the inverse of gradients. However, numerous differences remain: (i) in e.g. RNNs, the gradients w.r.t. to the weights are defined as a sum of the composition of the gradients and the inverse of this sum is a priori not the sum of the inverses, (ii) if the real rationale of GN was used, the gradients w.r.t. the loss should also be inverted, and the gradients w.r.t. to the weights should also be inverted which is not the case in the usual implementation of TP Lee et al. (2015); Bengio (2020). Even if TP was approximating GN, it is unclear whether GN updates are adapted to stochastic problems.

heading: 4 EXPERIMENTS
text: In the following, we compare our simple target propagation approach, which we shall refer to as TP, to gradient Back-Propagation referred to as BP. We follow the experimental benchmark of Manchev & Spratling (2020) to which we add results on RNNs on CIFAR and GRUs on FashionMNIST. Additional experiments, details on the initialization and the hyper-parameter selection can be found in Appendix D.
Data. We consider two synthetic datasets generated to present training difficulties for RNNs and several real datasets consisting of scanning images pixel by pixel to classify them (Hochreiter & Schmidhuber, 1997; Le et al., 2015; Manchev & Spratling, 2020).
Temporal order problem. A sequence of length T is generated using a set of randomly chosen symbols {a, b, c, d}. Two additional symbols X and Y are added at positions t1 2 [T/10, 2T/10] and t2 2 [4T/10, 5T/10]. The network must predict the correct order of appearance of X and Y out of four possible choices {XX,XY, Y X, Y Y }.
Adding problem. The input consists of two sequences: one is made of randomly chosen numbers from [0, 1], and the other one is a binary sequence full of zeros except at positions t1 2 [1, T/10] and t2 2 [T/10, T/2]. The second position acts as a marker for the time steps t1 and t2. The goal of the network is to output the mean of the two random numbers of the first sequence (Xt1 +Xt2)/2.
Image classification pixel by pixel. The inputs are images of (i) grayscale handwritten digits given in the database MNIST (LeCun & Cortes, 1998), (ii) colored objects from the database CIFAR10 (Krizhevsky, 2009) or (iii) grayscale images of clothes from the database FashionMNIST (Xiao et al., 2017). The images are scanned pixel by pixel and channel by channel for CIFAR10, and fed to a sequential network such as a simple RNN or a GRU network (Cho et al., 2014). The inputs are then sequences of 28 ⇥ 28 = 784 pixels for MNIST or FashionMNIST and 32 ⇥ 32 ⇥ 3 = 3072 pixels for CIFAR with a very long-range dependency problem. We also consider permuting the images of MNIST by a fixed permutation before feeding them into the network, which gives potentially longer dependencies in the sequential data.
Model. In both synthetic settings, we consider randomly generated mini-batches of size 20, a simple RNN with hidden states of dimension 100, and hyperbolic tangent activation. For the temporal order problem, the last layer uses a soft-max function on top of a linear operation, and the loss is the cross-entropy. For the adding problem, the last layer is linear, the loss is the mean-squared error, and a sample is considered to be accurately predicted if the mean squared error is less than 0.04 as done by (Manchev & Spratling, 2020).
For the classification of images with sequential networks, we consider mini-batches of size 16 and a cross-entropy loss. For MNIST and CIFAR, we consider a simple RNN with hidden states of dimension 100, hyperbolic tangent activation, and a softmax output. For FashionMNIST, we consider a GRU network and adapted our implementation of target propagation to that case while using hidden states of dimension 100 and a softmax output.
Target propagation can tackle long sequences better than gradient back-propagation. In Fig. 3, we observe that TP performs better than BP on the temporal ordering problem: it is able to reach 100% accuracy in fewer iterations than BP for sequences of length 60 and, for sequences of length 120, it is still able to reach 100% accuracy in fewer than 40 000 iterations while BP is not. On the other hand, for the adding problem, TP performs less well than BP. The contrast in performance between the two synthetic tasks was also observed by (Manchev & Spratling, 2020) using difference target propagation with parameterized inverses. The main difference between these tasks is the different nature of the outputs, which are binary for the temporal problem and continuous for the adding problem.
In Fig. 4, we observe that TP generally performs better than BP for image classification tasks. For the MNIST dataset, it reaches around 74% accuracy after 4 ·104 iterations. This phenomenon is also
Region of parameters ith con ergence
(6a) Conv. w.r.t. stepsize & regularization
4 14 49 196 392 784 Length
16
32
64
128
256
512
W id
th
BP
TP
(6b) Perf. vs width & length.
observed with permuted images, where the optimization appears smoother, and TP obtains around 86% accuracy after 4 · 104 iterations and is still faster than BP. On the CIFAR dataset, no algorithms appear to reach a significant accuracy, though TP is still faster. On the FashionMNIST dataset, where a GRU network is used, our implementation of TP performs on par with BP, which shows that our approach can be generalized to more complex networks than a simple RNN.
Target propagation requires a non-zero regularization term. As mentioned in Sec. 3, by using an analytical formula to compute the inverse of the layers, we can question the interpretation of TP as a Gauss-Newton method, which would amount to TP without regularization. To understand the effect of the regularization term, we computed the area under the training loss curve of TP for 400 iterations on a log10 grid of varying step-sizes ✓ and regularizations r for a fixed h = 10 3. The results are presented in Fig. 6a, where the smaller the area, the brighter the point and the absence of dots in the grid mean that the algorithm diverged. Fig. 6a shows that without regularization we were not able to obtain convergence of the algorithm. Simply using the gradients of the inverse as in a Gauss-Newton method may not directly work for RNNs. Additional modifications of the method could be added to make target propagation closer to Gauss-Newton, such as inverting the layers with respect to their parameters as proposed by Bengio (2020). For now, the regularization appears to successfully handle the rationale of target propagation.
Target propagation is adapted for long sequences. In Fig. 6b, we compare the performance of BP and TP in terms of accuracy after 400 iterations on the MNIST problem for various widths determined by the size of the hidden states and various lengths determined by the size of the inputs (i.e., we feed the RNN with k pixels at a time, which gives a length 784/k). Fig 6b shows that TP is generally appropriate for long sequences, while BP remains more efficient for short sequences. TP can then be seen as an interesting alternative for dynamical problems which involve many discretization steps as in RNNs and related architectures.
CONCLUSION
We proposed a simple target propagation approach grounded in two important computational components, regularized inversion, and linearized propagation. The proposed approach also sheds light on previous insights and successful rules for target propagation. We will publicly release our code to facilitate the reproduction of the results. We have used target propagation within a stochastic gradient outer loop to train neural networks for a fair comparison to stochastic gradient using gradient backpropagation. Developing adaptive stochastic gradient algorithms in the spirit of Adam that lead to boosts in performance when using target propagation instead of gradient backpropagation is an interesting avenue for future work. Continuous counterparts of target propagation in a neural ODE spirit is also an interesting avenue for future work.

Review summaries:
The authors study a variant of target propagation in which targets are computed by solving a sequence of minimization problems. Instead of resorting to iterative methods the authors propose to use an analytical solution. The algorithm is investigated as a recurrent neural network learning algorithm in a number of experiments. A relationship between the proposed descent direction and the loss function gradient is developed.